Post Syndicated from Kevin DeJong original https://aws.amazon.com/blogs/devops/introducing-aws-cloudformation-hooks-invoked-via-aws-cloud-control-api-ccapi/

Today we are announcing the integration of AWS CloudFormation Hooks with AWS Cloud Control API (CCAPI). This integration enables the use of hooks to validate the configuration of resources being provisioned through CCAPI. In this blog post, we will explore the integration between CloudFormation Hooks and CCAPI by configuring an existing hook to work with CCAPI and then test that hook using the AWS CLI and Terraform.

Understanding CloudFormation Hooks

CloudFormation Hooks integrate seamlessly with your CloudFormation and CCAPI requests to perform validation of your resource configuration during resource create and update operations. You can create hooks using AWS Lambda, AWS CloudFormation Guard rules, or using code and the CloudFormation Command Line Interface (CFN-CLI). A hook can be triggered on change sets, entire stack templates, or by each resource and it will return back any discovered misconfiguration information. Hooks can be configured to warn or fail on the operation allowing you to prevent any misconfigured resources from being deployed in your account. Some key benefits of using CloudFormation Hooks with CCAPI include:

• Enforcing security best practices
• Applying organizational policies to resource deployments
• Optimizing resource configurations for cost and performance
• Standardize validation across different infrastructure as code solutions like CloudFormation, Terraform (Terraform AWS Cloud Control Provider), and Pulumi (AWS Cloud Control)

Prerequisites

For this post we are going to use the new AWS CloudFormation Guard (Guard) hook AWS::Hooks::GuardHook. Guard is an open-source policy-as-code tool that allows you to validate your infrastructure configurations against company policy guidelines. It provides a domain-specific language (DSL) for writing rules to check both required and prohibited resource configurations. The new AWS::Hooks::GuardHook allows you to use the Guard DSL inside of a hook so you can easily implement your organizations guidelines. The result is you can use the same Guard rules in our local development environment, continuous integration and continuous deployment pipelines, and at deployment time (using hooks). To learn more about AWS::Hooks::GuardHook you can look at the blog.

This is what the configuration of the current Guard hook looks like.

{
    "CloudFormationConfiguration": {
        "HookConfiguration": {
            "HookInvocationStatus": "ENABLED",
            "FailureMode": "FAIL",
            "TargetOperations": [
                "RESOURCE"
            ],
            "TargetFilters": {
                "Actions": [
                    "CREATE",
                    "UPDATE"
                ]
            },
            "Properties": {
                "ruleLocation": {
                    "uri": "s3://<my-guard-hook-config-bucket>/s3-guard.zip"
                },
                "logBucket": "<my-guard-hook-logging-bucket>",
            }
        }
    }
}

This hook has been configured to log the Guard validation report to an Amazon Simple Storage Service (S3) bucket. Additionally, this hook is configured to use a rule from the AWS CloudFormation Guard registry. This rule will validate that an S3 bucket is using versioning. This hook is configured with an alias named My::Hooks::Guard.

Here is the rule for reference. This rule will validate that the property VersioningConfiguration is provided and that its value is Enabled.

let s3_buckets_versioning_enabled = Resources.*[ Type == 'AWS::S3::Bucket' ]
rule S3_BUCKET_VERSIONING_ENABLED when %s3_buckets_versioning_enabled !empty {
  %s3_buckets_versioning_enabled.Properties.VersioningConfiguration exists
  %s3_buckets_versioning_enabled.Properties.VersioningConfiguration.Status == 'Enabled'
  <<
    Guard Rule Set: ABS-CCIGv2-Standard
    Controls: section4b-design-and-secure-the-cloud-14-standard-workloads,section4b-design-and-secure-the-cloud-15-standard-workloads    
    Violation: S3 Bucket Versioning must be enabled.
    Fix: Set the S3 Bucket property VersioningConfiguration.Status to 'Enabled' .
  >>
}

Configuring the hook to work with CCAPI

This announcement adds a new hook target that can easily be configured on your existing or new hooks. To configure the hook to work with CCAPI you will edit the configuration to include a new TargetOperations value of CLOUD_CONTROL. This hook is only enabled to execute on CREATE and UPDATE operations. Additionally HookInvocationStatus is ENABLED which will execute the hook and FailureMode will tell the hook to FAIL the operation if the resource is not compliant.

{
    "CloudFormationConfiguration": {
        "HookConfiguration": {
            "HookInvocationStatus": "ENABLED",
            "FailureMode": "FAIL",
            "TargetOperations": [
                "RESOURCE",
                "CLOUD_CONTROL"
            ],
            "TargetFilters": {
                "Actions": [
                    "CREATE",
                    "UPDATE"
                ]
            },
            "Properties": {
                "ruleLocation": {
                    "uri": "s3://<my-guard-hook-confg-bocket>/s3-guard.zip"
                },
                "logBucket": "<my-guard-hook-logging-bocket>",
            }
        }
    }
}

By using TargetOperations of ["RESOURCE", "CLOUD_CONTROL"] the Guard rules will work the same across CloudFormation resource operations and CCAPI operations.

Testing the hook using AWS CLI

Test your hook using the AWS CLI which allows us to create, update, delete, and list resources.

1. Start by creating a S3 bucket using CCAPI. In this example you are providing no properties for creating the S3 bucket. Run the command aws cloudcontrol create-resource --type-name AWS::S3::Bucket --desired-state {}
   Response:

   {
       "ProgressEvent": {
           "TypeName": "AWS::S3::Bucket",
           "RequestToken": "2c7b6f5e-4083-4ef8-9a23-5c81472540b1",
           "Operation": "CREATE",
           "OperationStatus": "IN_PROGRESS",
           "EventTime": "2024-11-05T09:41:38.072000-08:00"
       }
   }

2. Get the request status by using the RequestToken from the response above. Run the command aws cloudcontrol get-resource-request-status --request-token 2c7b6f5e-4083-4ef8-9a23-5c81472540b1

   Response:

   {
       "ProgressEvent": {
           "TypeName": "AWS::S3::Bucket",
           "RequestToken": "2c7b6f5e-4083-4ef8-9a23-5c81472540b1",
           "HooksRequestToken": "4a193a00-4c76-41fe-87b8-75b838f00bbe",
           "Operation": "CREATE",
           "OperationStatus": "FAILED",
           "EventTime": "2024-11-05T09:41:40.785000-08:00",
           "StatusMessage": "Request [4a193a00-4c76-41fe-87b8-75b838f00bbe] failed \ndue to the following failed invocations: [My::Hooks::Guard]"
       },
       "HooksProgressEvent": [
           {
               "HookTypeName": "My::Hooks::Guard",
               "HookTypeVersionId": "00000006",
               "HookTypeArn": "arn:aws:cloudformation:eu-central-1:123456789012:type/hook/My-Hooks-Guard/00000001/aws-hooks/AWS-Hooks-GuardHook/00000001.00000005",
               "InvocationPoint": "PRE_PROVISION",
               "HookStatus": "HOOK_COMPLETE_FAILED",
               "HookEventTime": "2024-11-05T09:41:38.978000-08:00",
               "HookStatusMessage": "Template failed validation, the following rule(s) failed: S3_BUCKET_VERSIONING_ENABLED. Full output was written to s3://<my-guard-hook-logging-bocket>/cfn-guard-validate-report/AWS--S3--Bucket-4a193a00-4c76-41fe-87b8-75b838f00bbe-RESOURCE-AWS--S3--Bucket-CREATE-PRE_PROVISION/1730828427591.json",
               "FailureMode": "FAIL"
           }
       ]
   }

   In the response you will see all hooks that were executed and their response in relation to the request. This response shows that the hook My::Hooks::Guard failed because of the rule S3_BUCKET_VERSIONING_ENABLED . You are also provided a s3 location for where the full Guard output is stored.

3. You can get the Guard results file by using the following command. Replace <path-from-previous-output> with the path provided in the previous output. Run the command aws s3 cp s3://<path-from-previous-output> -

   Response:

   [
       {
           "name": "STDIN",
           "metadata": {},
           "status": "FAIL",
           "not_compliant": [
               {
                   "Rule": {
                       "name": "S3_BUCKET_VERSIONING_ENABLED",
                       "metadata": {},
                       "messages": {
                           "custom_message": null,
                           "error_message": null
                       },
                       "checks": [
                           {
   ...

   We truncated the output as it can be very verbose.

Testing the hook using Terraform

The Terraform AWS Cloud Control Provider allows you to manage AWS resources using CCAPI and Terraform. By leveraging this provider you get the benefit of using hooks to validate the configuration of Terraform provisioned resources.

1. Create a new Terraform configuration file named main.tf with the following content:

   terraform {
       required_providers {
           awscc = {
               source  = "hashicorp/awscc"
               version = "~> 1.20"
           }
       }
   }
   
   resource "awscc_s3_bucket" "example" {}

2. Run the following commands to initialize Terraform and create an execution plan. Run the command terraform init followed by terraform plan.
3. Apply the configuration by running. Run the command terraform apply

   Response:

   ...
   awscc_s3_bucket.example: Creating...
   ╷
   │ Error: AWS SDK Go Service Operation Incomplete
   │ 
   │   with awscc_s3_bucket.example,
   │   on main.tf line 14, in resource "awscc_s3_bucket" "example":
   │   14: resource "awscc_s3_bucket" "example" {
   │ 
   │ Waiting for Cloud Control API service CreateResource operation completion returned: waiter state transitioned to FAILED. StatusMessage: Request [d417b05b-9eff-46ef-b164-08c76aec1801] failed 
   │ due to the following failed invocations: [My::Hooks::Guard]. ErrorCode: 
   ╵

   In this response you can see that the hook My::Hooks::Guard failed and the request token is d417b05b-9eff-46ef-b164-08c76aec1801

4. You can get details on the hook invocation by running the command aws cloudformation list-hook-results --hook-target TargetType=CLOUD_CONTROL,TargetId=d417b05b-9eff-46ef-b164-08c76aec1801Response:

   {
       "TargetType": "CLOUD_CONTROL",
       "TargetId": "d417b05b-9eff-46ef-b164-08c76aec1801",
       "HookResults": [
           {
               "InvocationPoint": "PRE_PROVISION",
               "FailureMode": "FAIL",
               "TypeName": "My::Hooks::Guard",
               "TypeVersionId": "00000006",
               "Status": "HOOK_COMPLETE_FAILED",
               "HookStatusReason": "Template failed validation, the following rule(s) failed: S3_BUCKET_VERSIONING_ENABLED. Full output was written to s3://my-company-guard-hooks-eu-central-1/cfn-guard-validate-report/AWS--S3--Bucket-d417b05b-9eff-46ef-b164-08c76aec1801-RESOURCE-AWS--S3--Bucket-CREATE-PRE_PROVISION/1730829108790.json"
           }
       ]
   }

   As with the AWS CLI you now know what rule failed and additional you have the S3 bucket location for the Guard log file.

5. You can get the Guard results file by running the command aws s3 cp s3://<path-from-previous-output> -. Replace <path-from-previous-output> with the path provided in the previous output.Response:

   [
       {
           "name": "STDIN",
           "metadata": {},
           "status": "FAIL",
           "not_compliant": [
               {
                   "Rule": {
                       "name": "S3_BUCKET_VERSIONING_ENABLED",
                       "metadata": {},
                       "messages": {
                           "custom_message": null,
                           "error_message": null
                       },
                       "checks": [
                           {
   ...

   We truncated the output as it can be very verbose.

6. Let’s correct our S3 bucket configuration in main.tf

   terraform {
       required_providers {
           awscc = {
               source  = "hashicorp/awscc"
               version = "~> 1.20"
           }
       }
   }
   
   resource "awscc_s3_bucket" "example" {
       versioning_configuration = {
           status = "Enabled"
       }
   }

7. Try the deployment again by running terraform applyResponse:

   ...
   awscc_s3_bucket.example: Creating...
   awscc_s3_bucket.example: Still creating... [10s elapsed]
   awscc_s3_bucket.example: Creation complete after 20s [id=rjuzykvh6oum2jeuq42xsxets-evilftb6rutb]
   Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

   Conclusion

   CloudFormation Hooks provide a powerful way to enforce best practices and compliance for your AWS resources. By leveraging CloudFormation Hooks and the Cloud Control API you can create consistent validation of your resources before deployment across many of your infrastructure as code solutions.

Post Syndicated from Kevin DeJong original https://aws.amazon.com/blogs/devops/aws-cloudformation-linter-v1/

Introduction

The CloudFormation Linter, cfn-lint, is a powerful tool designed to enhance the development process of AWS CloudFormation templates. It serves as a static analysis tool that checks CloudFormation templates for potential errors and best practices, ensuring that your infrastructure as code adheres to AWS best practices and standards. With its comprehensive rule set and customizable configuration options, cfn-lint provides developers with valuable insights into their CloudFormation templates, helping to streamline the deployment process, improve code quality, and optimize AWS resource utilization.

What’s Changing?

With cfn-lint v1, we are introducing a set of major enhancements that involve breaking changes. This upgrade is particularly significant as it converts from using the CloudFormation spec to using CloudFormation registry resource provider schemas. This change is aimed at improving the overall performance, stability, and compatibility of cfn-lint, ensuring a more seamless and efficient experience for our users.

Key Features of cfn-lint v1

1. CloudFormation Registry Resource Provider Schemas: The migration to registry schemas brings a more robust and standardized approach to validating CloudFormation templates, offering improved accuracy in linting. We use additional data sources like the AWS pricing API and botocore (the foundation to the AWS CLI and AWS SDK for Python (Boto3)) to improve the schemas and increase the accuracy of our validation. We extend the schemas with additional keywords and logic to extend validation from the schemas.
2. Rule Simplification: for this upgrade, we rewrote over 100 rules. Where possible, we rewrote rules to leverage JSON schema validation, which allows us to use common logic across rules. The result is that we now return more common error messages across our rules.
3. Region Support: cfn-lint supports validation of resource types across regions. v1 expands this validation to check resource properties across all unique schemas for the resource type.

Transition Guidelines

To facilitate a seamless transition, we advise following these steps:

Review Templates

While we aim to preserve backward compatibility, we recommend reviewing your CloudFormation templates to ensure they align with the latest version. This step helps preempt any potential issues in your pipeline or deployment processes. If necessary, you can enforce pinning to cfn-lint v0 by running pip install --upgrade "cfn-lint<1"

Handling cfn-lint configurations

Throughout the process of rewriting rules, we’ve restructured some of the logic. Consequently, if you’ve been ignoring a specific rule, it’s possible that the logic associated with it has shifted to a new rule. As you transition to v1, you may need to adjust your template ignore rules configuration accordingly. Here is a subset of some of the changes with a focus on some of the more significant changes.

• In v0, rule E3002 validated valid resource property names but it also validated object and array type checks. In v1 all type checks are now in E3012.
• In v0, rule E3017 validated that when a property had a certain value other properties may be required. This validation has been rewritten into individual rules. This should allow more flexibility in ignoring and configuring rules.
• In v0, rule E2522 validated when at least one of a list of properties is required. That logic has been moved to rule E3015.
• In v0, rule E2523 validated when only one property from a list is required. That logic has been moved to rule E3014.

Adapting extensions to cfn-lint

If you’ve extended cfn-lint with custom rules or utilized it as a library, be aware that there have been some API changes. It’s advisable to thoroughly test your rules and packages to ensure consistency as you upgrade to v1.

Upgrade to cfn-lint v1

Upon the release of the new version, we highly recommend upgrading to cfn-lint v1 to capitalize on its enriched features and improvements. You can upgrade using pip by running pip install --upgrade cfn-lint.

Stay Updated

Keep yourself informed by monitoring our communication channels for announcements, release notes, and any additional information pertinent to cfn-lint v1. You can follow us on Discord. cfn-lint is an open source solution so you can submit issues on GitHub or follow our v1 discussion on GitHub.

Dependencies

cfn-lint v1 uses Python optional dependencies to reduce the amount of dependencies we install for standard usage. If you want to leverage features like graph, or output formats junit and sarif, you will have to change your install commands.

• pip install cfn-lint[graph] – will include pydot to create graphs of resource dependencies using --build-graph
• pip install cfn-lint[junit] – will include the packages to output JUnit using --output junit
• pip install cfn-lint[sarif] – will include the packages to output SARIF using --output sarif

cfn-lint v0 support

We will continue to update and support cfn-lint v0 until early 2025. This includes regular releases to new CloudFormation spec files. We will only add new features into v1.

Thank You for Your Continued Support

We appreciate your continued trust and support as we work to enhance cfn-lint. Our team is committed to providing you with the best possible experience, and we believe that cfn-lint v1 will elevate your CloudFormation template development process.

If you have any questions or concerns, please don’t hesitate to reach out on our GitHub page.