Tag Archives: Advanced (300)

Code signing using AWS Certificate Manager Private CA and AWS Key Management Service asymmetric keys

Post Syndicated from Ram Ramani original https://aws.amazon.com/blogs/security/code-signing-aws-certificate-manager-private-ca-aws-key-management-service-asymmetric-keys/

In this post, we show you how to combine the asymmetric signing feature of the AWS Key Management Service (AWS KMS) and code-signing certificates from the AWS Certificate Manager (ACM) Private Certificate Authority (PCA) service to digitally sign any binary data blob and then verify its identity and integrity. AWS KMS makes it easy for you to create and manage cryptographic keys and control their use across a wide range of AWS services and with your applications running on AWS. ACM PCA provides you a highly available private certificate authority (CA) service without the upfront investment and ongoing maintenance costs of operating your own private CA. CA administrators can use ACM PCA to create a complete CA hierarchy, including online root and subordinate CAs, with no need for external CAs. Using ACM PCA, you can provision, rotate, and revoke certificates that are trusted within your organization.

Traditionally, a person’s signature helps to validate that the person signed an agreement and agreed to the terms. Signatures are a big part of our lives, from our driver’s licenses to our home mortgage documents. When a signature is requested, the person or entity requesting the signature needs to verify the validity of the signature and the integrity of the message being signed.

As the internet and cryptography research evolved, technologists found ways to carry the usefulness of signatures from the analog world to the digital world. In the digital world, public and private key cryptography and X.509 certificates can help with digital signing, verifying message integrity, and verifying signature authenticity. In simple terms, an entity—which could be a person, an organization, a device, or a server—can digitally sign a piece of data, and another entity can validate the authenticity of the signature and validate the integrity of the signed data. The data that’s being signed could be a document, a software package, or any other binary data blob.

To learn more about AWS KMS asymmetric keys and ACM PCA, see Digital signing with the new asymmetric keys feature of AWS KMS and How to host and manage an entire private certificate infrastructure in AWS.

We provide Java code snippets for each part of the process in the following steps. In addition, the complete Java code with the maven build configuration file pom.xml are available for download from this GitHub project. The steps below illustrate the different processes that are involved and the associated Java code snippet. However, you need to use the GitHub project to be able to build and run the Java code successfully.

Let’s take a look at the steps.

1. Create an asymmetric key pair

For digital signing, you need a code-signing certificate and an asymmetric key pair. In this step, you create an asymmetric key pair using AWS KMS. The below code snippet in the main method within the file Runner.java is used to create the asymmetric key pair within KMS in your AWS account. An asymmetric KMS key with the alias CodeSigningCMK is created.


AsymmetricCMK codeSigningCMK = AsymmetricCMK.builder()
                .withAlias(CMK_ALIAS)
                .getOrCreate();

2. Create a code-signing certificate

To create a code-signing certificate, you need a private CA hierarchy, which you create within the ACM PCA service. This uses a simple CA hierarchy of one root CA and one subordinate CA under the root because the recommendation is that you should not use the root CA directly for signing code-signing certificates. The certificate authorities are needed to create the code-signing certificate. The common name for the root CA certificate is root CA, and the common name for the subordinate CA certificate is subordinate CA. The following code snippet in the main method within the file Runner.java is used to create the private CA hierarchy.


PrivateCA rootPrivateCA = PrivateCA.builder()
                .withCommonName(ROOT_COMMON_NAME)
                .withType(CertificateAuthorityType.ROOT)
                .getOrCreate();

PrivateCA subordinatePrivateCA = PrivateCA.builder()
        .withIssuer(rootPrivateCA)
        .withCommonName(SUBORDINATE_COMMON_NAME)
        .withType(CertificateAuthorityType.SUBORDINATE)
        .getOrCreate();

3. Create a certificate signing request

In this step, you create a certificate signing request (CSR) for the code-signing certificate. The following code snippet in the main method within the file Runner.java is used to create the CSR. The END_ENTITY_COMMON_NAME refers to the common name parameter of the code signing certificate.


String codeSigningCSR = codeSigningCMK.generateCSR(END_ENTITY_COMMON_NAME);

4. Sign the CSR

In this step, the code-signing CSR is signed by the subordinate CA that was generated in step 2 to create the code-signing certificate.


GetCertificateResult codeSigningCertificate = subordinatePrivateCA.issueCodeSigningCertificate(codeSigningCSR);

Note: The code-signing certificate that’s generated contains the public key of the asymmetric key pair generated in step 1.

5. Create the custom signed object

The data to be signed is a simple string: “the data I want signed”. Its binary representation is hashed and digitally signed by the asymmetric KMS private key created in step 1, and a custom signed object that contains the signature and the code-signing certificate is created.

The below code snippet in the main method within the file Runner.java is used to create the custom signed object.


CustomCodeSigningObject customCodeSigningObject = CustomCodeSigningObject.builder()
                .withAsymmetricCMK(codeSigningCMK)
                .withDataBlob(TBS_DATA.getBytes(StandardCharsets.UTF_8))
                .withCertificate(codeSigningCertificate.getCertificate())
                .build();

6. Verify the signature

The custom signed object is verified for integrity, and the root CA certificate is used to verify the chain of trust to confirm non-repudiation of the identity that produced the digital signature.

The below code snippet in the main method within the file Runner.java is used for signature verification:


String rootCACertificate = rootPrivateCA.getCertificate();
 String customCodeSigningObjectCertificateChain = codeSigningCertificate.getCertificate() + "\n" + codeSigningCertificate.getCertificateChain();

 CustomCodeSigningObject.getInstance(customCodeSigningObject.toString())
        .validate(rootCACertificate, customCodeSigningObjectCertificateChain);

During this signature validation process, the validation method shown in the code above retrieves the public key portion of the AWS KMS asymmetric key pair generated in step 1 from the code-signing certificate. This process has the advantage that credentials to access AWS KMS aren’t needed during signature validation. Any entity that has the root CA certificate loaded in its trust store can verify the signature without needing access to the AWS KMS verify API.

Note: The implementation outlined in this post is an example. It doesn’t use a certificate trust store that’s either part of a browser or part of a file system within the resident operating system of a device or a server. The trust store is placed in an instance of a Java class object for the purpose of this post. If you are planning to use this code-signing example in a production system, you must change the implementation to use a trust store on the host. To do so, you can build and distribute a secure trust store that includes the root CA certificate.

Conclusion

In this post, we showed you how a binary data blob can be digitally signed using ACM PCA and AWS KMS and how the signature can be verified using only the root CA certificate. No secret information or credentials are required to verify the signature. You can use this method to build a custom code-signing solution to address your particular use cases. The GitHub repository provides the Java code and the maven pom.xml that you can use to build and try it yourself. The README.md file in the GitHub repository shows the instructions to execute the code.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the AWS Certificate Manager forum or contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Ram Ramani

Ram is a Security Solutions Architect at AWS focusing on data protection. Ram works with customers across different industry verticals to provide them with solutions that help with protecting data at rest and in transit. In prior roles, Ram built ML algorithms for video quality optimization and worked on identity and access management solutions for financial services organizations.

Author

Kyle Schultheiss

Kyle is a Senior Software Engineer on the AWS Cryptography team. He has been working on the ACM Private Certificate Authority service since its inception in 2018. In prior roles, he contributed to other AWS services such as Amazon Virtual Private Cloud, Amazon EC2, and Amazon Route 53.

How to build a CI/CD pipeline for container vulnerability scanning with Trivy and AWS Security Hub

Post Syndicated from Amrish Thakkar original https://aws.amazon.com/blogs/security/how-to-build-ci-cd-pipeline-container-vulnerability-scanning-trivy-and-aws-security-hub/

In this post, I’ll show you how to build a continuous integration and continuous delivery (CI/CD) pipeline using AWS Developer Tools, as well as Aqua Security‘s open source container vulnerability scanner, Trivy. You’ll build two Docker images, one with vulnerabilities and one without, to learn the capabilities of Trivy and how to send all vulnerability information to AWS Security Hub.

If you’re building modern applications, you might be using containers, or have experimented with them. A container is a standard way to package your application’s code, configurations, and dependencies into a single object. In contrast to virtual machines (VMs), containers virtualize the operating system rather than the server. Thus, the images are orders of magnitude smaller, and they start up much more quickly.

Like VMs, containers need to be scanned for vulnerabilities and patched as appropriate. For VMs running on Amazon Elastic Compute Cloud (Amazon EC2), you can use Amazon Inspector, a managed vulnerability assessment service, and then patch your EC2 instances as needed. For containers, vulnerability management is a little different. Instead of patching, you destroy and redeploy the container.

Many container deployments use Docker. Docker uses Dockerfiles to define the commands you use to build the Docker image that forms the basis of your container. Instead of patching in place, you rewrite your Dockerfile to point to more up-to-date base images, dependencies, or both and to rebuild the Docker image. Trivy lets you know which dependencies in the Docker image are vulnerable, and which version of those dependencies are no longer vulnerable, allowing you to quickly understand what to patch to get back to a secure state.

Solution architecture

 

Figure 1: Solution architecture

Figure 1: Solution architecture

Here’s how the solution works, as shown in Figure 1:

  1. Developers push Dockerfiles and other code to AWS CodeCommit.
  2. AWS CodePipeline automatically starts an AWS CodeBuild build that uses a build specification file to install Trivy, build a Docker image, and scan it during runtime.
  3. AWS CodeBuild pushes the build logs in near real-time to an Amazon CloudWatch Logs group.
  4. Trivy scans for all vulnerabilities and sends them to AWS Security Hub, regardless of severity.
  5. If no critical vulnerabilities are found, the Docker images are deemed to have passed the scan and are pushed to Amazon Elastic Container Registry (ECR), so that they can be deployed.

Note: CodePipeline supports different sources, such as Amazon Simple Storage Service (Amazon S3) or GitHub. If you’re comfortable with those services, feel free to substitute them for this walkthrough of the solution.

To quickly deploy the solution, you’ll use an AWS CloudFormation template to deploy all needed services.

Prerequisites

  1. You must have Security Hub enabled in the AWS Region where you deploy this solution. In the AWS Management Console, go to AWS Security Hub, and select Enable Security Hub.
  2. You must have Aqua Security integration enabled in Security Hub in the Region where you deploy this solution. To do so, go to the AWS Security Hub console and, on the left, select Integrations, search for Aqua Security, and then select Accept Findings.

Setting up

For this stage, you’ll deploy the CloudFormation template and do preliminary setup of the CodeCommit repository.

  1. Download the CloudFormation template from GitHub and create a CloudFormation stack. For more information on how to create a CloudFormation stack, see Getting Started with AWS CloudFormation.
  2. After the CloudFormation stack completes, go to the CloudFormation console and select the Resources tab to see the resources created, as shown in Figure 2.

 

Figure 2: CloudFormation output

Figure 2: CloudFormation output

Setting up the CodeCommit repository

CodeCommit repositories need at least one file to initialize their master branch. Without a file, you can’t use a CodeCommit repository as a source for CodePipeline. To create a sample file, do the following.

  1. Go to the CodeCommit console and, on the left, select Repositories, and then select your CodeCommit repository.
  2. Scroll to the bottom of the page, select the Add File dropdown, and then select Create file.
  3. In the Create a file screen, enter readme into the text body, name the file readme.md, enter your name as Author name and your Email address, and then select Commit changes, as shown in Figure 3.

    Figure 3: Creating a file in CodeCommit

    Figure 3: Creating a file in CodeCommit

Simulate a vulnerable image build

For this stage, you’ll create the necessary files and add them to your CodeCommit repository to start an automated container vulnerability scan.

    1. Download the buildspec.yml file from the GitHub repository.

      Note: In the buildspec.yml code, the values prepended with $ will be populated by the CodeBuild environmental variables you created earlier. Also, the command trivy -f json -o results.json –exit-code 1 will fail your build by forcing Trivy to return an exit code 1 upon finding a critical vulnerability. You can add additional severity levels here to force Trivy to fail your builds and ensure vulnerabilities of lower severity are not published to Amazon ECR.

    2. Download the python code file sechub_parser.py from the GitHub repository. This script parses vulnerability details from the JSON file that Trivy generates, maps the information to the AWS Security Finding Format (ASFF), and then imports it to Security Hub.
    3. Next, download the Dockerfile from the GitHub repository. The code clones a GitHub repository maintained by the Trivy team that has purposely vulnerable packages that generate critical vulnerabilities.
    4. Go back to your CodeCommit repository, select the Add file dropdown menu, and then select Upload file.
    5. In the Upload file screen, select Choose file, select the build specification you just created (buildspec.yml), complete the Commit changes to master section by adding the Author name and Email address, and select Commit changes, as shown in Figure 4.

 

Figure 4: Uploading a file to CodeCommit

Figure 4: Uploading a file to CodeCommit

 

  • To upload your Dockerfile and sechub_parser.py script to CodeCommit, repeat steps 4 and 5 for each of these files.
  • Your pipeline will automatically start in response to every new commit to your repository. To check the status, go back to the pipeline status view of your CodePipeline pipeline.
  • When CodeBuild starts, select Details in the Build stage of the CodePipeline, under BuildAction, to go to the Build section on the CodeBuild console. To see a stream of logs as your build progresses, select Tail logs, as shown in Figure 5.

    Figure 5: CodeBuild Tailed Logs

    Figure 5: CodeBuild Tailed Logs

  • After Trivy has finished scanning your image, CodeBuild will fail due to the critical vulnerabilities found, as shown in Figure 6.

    Note: The command specified in the post-build stage will run even if the CodeBuild build fails. This is by design and allows the sechub_parser.py script to run and send findings to Security Hub.

     

    Figure 6: CodeBuild logs failure

    Figure 6: CodeBuild logs failure

 

You’ll now go to Security Hub to further analyze the findings and create saved searches for future use.

Analyze container vulnerabilities in Security Hub

For this stage, you’ll analyze your container vulnerabilities in Security Hub and use the findings view to locate information within the ASFF.

  1. Go to the Security Hub console and select Integrations in the left-hand navigation pane.
  2. Scroll down to the Aqua Security integration card and select See findings, as shown in Figure 7. This filters to only Aqua Security product findings in the Findings view.

    Figure 7: Aqua Security integration card

    Figure 7: Aqua Security integration card

  3. You should now see critical vulnerabilities from your previous scan in the Findings view, as shown in Figure 8. To see more details of a finding, select the Title of any of the vulnerabilities, and you will see the details in the right side of the Findings view.
    Figure 8: Security Hub Findings pane

    Figure 8: Security Hub Findings pane

    Note: Within the Findings view, you can apply quick filters by checking the box next to certain fields, although you won’t do that for the solution in this post.

  4. To open a new tab to a website about the Common Vulnerabilities and Exposures (CVE) for the finding, select the hyperlink within the Remediation section, as shown in Figure 9.
    Figure 9: Remediation information

    Figure 9: Remediation information

    Note: The fields shown in Figure 9 are dynamically populated by Trivy in its native output, and the CVE website can differ greatly from vulnerability to vulnerability.

  5. To see the JSON of the full ASFF, at the top right of the Findings view, select the hyperlink for Finding ID.
  6. To find information mapped from Trivy, such as the CVE title and what the patched version of the vulnerable package is, scroll down to the Other section, as shown in Figure 10.

    Figure 10: ASFF, other

    Figure 10: ASFF, other

This was a brief demonstration of exploring findings with Security Hub. You can use custom actions to define response and remediation actions, such as sending these findings to a ticketing system or aggregating them in a security information event management (SIEM) tool.

Push a non-vulnerable Dockerfile

Now that you’ve seen Trivy perform correctly with a vulnerable image, you’ll fix the vulnerabilities. For this stage, you’ll modify Dockerfile to remove any vulnerable dependencies.

  1. Open a text editor, paste in the code shown below, and save it as Dockerfile. You can overwrite your previous example if desired.
    
    FROM alpine:3.7
    RUN apk add --no-cache mysql-client
    ENTRYPOINT ["mysql"]
    	

  2. Upload the new Dockerfile to CodeCommit, as shown earlier in this post.

Clean up

To avoid incurring additional charges from running these services, disable Security Hub and delete the CloudFormation stack after you’ve finished evaluating this solution. This will delete all resources created during this post. Deleting the CloudFormation stack will not remove the findings in Security Hub. If you don’t disable Security Hub, you can archive those findings and wait 90 days for them to be fully removed from your Security Hub instance.

Conclusion

In this post, you learned how to create a CI/CD Pipeline with CodeCommit, CodeBuild, and CodePipeline for building and scanning Docker images for critical vulnerabilities. You also used Security Hub to aggregate scan findings and take action on your container vulnerabilities.

You now have the skills to create similar CI/CD pipelines with AWS Developer Tools and to perform vulnerability scans as part of your container build process. Using these reports, you can efficiently identify CVEs and work with your developers to use up-to-date libraries and binaries for Docker images. To further build upon this solution, you can change the Trivy commands in your build specification to fail the builds on different severity levels of found vulnerabilities. As a final suggestion, you can use ECR as a source for a downstream CI/CD pipeline responsible for deploying your newly-scanned images to Amazon Elastic Container Service (Amazon ECS).

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the AWS Security Hub forum or contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Amrish Thakkar

Amrish is a Senior Solutions Architect at AWS. He holds the AWS Certified Solutions Architect Professional and AWS Certified DevOps Engineer Professional certifications. Amrish is passionate about DevOps, Microservices, Containerization, and Application Security, and devotes personal time into research and advocacy on these topics. Outside of work, he enjoys spending time with family and watching LOTR trilogy frequently.

How to perform automated incident response in a multi-account environment

Post Syndicated from Vesselin Tzvetkov original https://aws.amazon.com/blogs/security/how-to-perform-automated-incident-response-multi-account-environment/

How quickly you respond to security incidents is key to minimizing their impacts. Automating incident response helps you scale your capabilities, rapidly reduce the scope of compromised resources, and reduce repetitive work by security teams. But when you use automation, you also must manage exceptions to standard response procedures.

In this post, I provide a pattern and ready-made templates for a scalable multi-account setup of an automated incident response process with minimal code base, using native AWS tools. I also explain how to set up exception handling for approved deviations based on resource tags.

Because security response is a broad topic, I provide an overview of some alternative approaches to consider. Incident response is one part of an overarching governance, risk, and compliance (GRC) program that every organization should implement. For more information, see Scaling a governance, risk, and compliance program for the cloud.

Important: Use caution when introducing automation. Carefully test each of the automated responses in a non-production environment, as you should not use untested automated incident response for business-critical applications.

Solution benefits and deliverables

In the solution described below, you use AWS Systems Manager automation to execute most of the incident response steps. In general, you can either write your own or use pre-written runbooks. AWS maintains ready-made operational runbooks (automation documents) so you don’t need to maintain your own code base for them. The AWS runbooks cover many predefined use cases, such as enabling Amazon Simple Storage Service (S3) bucket encryption, opening a Jira ticket, or terminating an Amazon Elastic Compute Cloud (EC2) instance. Every execution for these is well documented and repeatable in AWS Systems Manager. For a few cases where there are no ready-made automation documents, I provide three additional AWS Lambda functions with the required response actions in the templates. The Lambda functions require minimal code, with no external dependencies outside AWS native tools.

You use a central security account to execute the incident response actions in the automation runbooks. You don’t need to do anything in the service accounts where you monitor and respond to incidents. In this post, you learn about and receive patterns for:

  • Responding to an incident based on Amazon GuardDuty alerts or AWS Config findings.
  • Deploying templates for a multi-account setup with a central security account and multiple service accounts. All account resources must be in the same AWS Region and part of the same AWS organization.
  • AWS Systems Manager automation to execute many existing AWS managed automation runbooks (you need access to the AWS Management Console to see all documents at this link). Systems Manager automation is available in these AWS Regions.
  • Prewritten Lambda functions to:
    • Confine permissive (open) security groups to a more-constrained CIDR (classless interdomain routing), such as the VPC (virtual private cloud) range for that security group. This prevents knowable network configuration errors, such as too-open security groups.
    • Isolate a potentially compromised EC2 instance by attaching it to a single, empty security group and removing its existing security groups.
    • Block an AWS Identity and Access Management (IAM) principal (an IAM user or role) by attaching a deny all policy to that principal.
    • Send notifications to the Amazon Simple Notification Service (Amazon SNS) for alerting in addition to (and in concert with) automated response actions.
  • Exception handling when actions should not be executed. I suggest decentralized exception handling based on AWS resource tags.
  • Integrating AWS Security Hub with custom actions for GuardDuty findings. This can be used for manual triggering of remediations that should not receive an automatic response.
  • Custom AWS Config rule for detecting permissive (open) security groups on any port.
  • Mapping the security finding to the response action defined in the CloudWatch Events rule pattern is extendable. I provide an example of how to extend to new findings and responses.

Out-of-scope for this solution

This post only shows you how to get started with security response automation for accounts that are part of the same AWS organization and Region. For more information on developing a comprehensive program for incident response, see How to prepare for & respond to security incidents in your AWS environment.

Understanding the difference between this solution and AWS Config remediation

The AWS Config remediation feature provides remediation of non-compliant resources using AWS Systems Manager within a single AWS account. The solution you learn in this post includes both GuardDuty alerts and AWS Config, a multi-account approach managed from a central security account, rules exception handling based on resource tags, and additional response actions using AWS Lambda functions.

Choosing the right approach for incident response

There are many technical patterns and solutions for responding to security incidents. When considering the right one for your organization, you must consider how much flexibility for response actions you need, what resources are in scope, and the skill level of your security engineers. Also consider dependencies on external code and applications, software licensing costs, and your organization’s experience level with AWS.

If you want the maximum flexibility and extensibility beyond AWS Systems Manager used in this post, I recommend AWS Step Functions as described in the session How to prepare for & respond to security incidents in your AWS environment and in DIY guide to runbooks, incident reports, and incident response. You can create workflows for your runbooks and have full control of all possible actions.

Architecture

 

Figure 1: Architecture diagram

Figure 1: Architecture diagram

The architecture works as follows:

In the service account (the environment for which we want to monitor and respond to incidents):

  1. GuardDuty findings are forwarded to CloudWatch Events.
  2. Changes in the AWS Config compliance status are forwarded to CloudWatch Events.
  3. CloudWatch Events for GuardDuty and AWS Config are forwarded to the central security account, via a CloudWatch Events bus.

In the central security account:

  1. Each event from the service account is mapped to one or more response actions using CloudWatch Events rules. Every rule is an incident response action that is executed on one or more security findings, as defined in the event pattern. If there is an exception rule, the response actions are not executed.

    The list of possible actions that could be taken include:

    1. Trigger a Systems Manager automation document, invoked by the Lambda function StratSsmAutomation within the security account.
    2. Isolate an EC2 instance by attaching an empty security group to it and removing any prior security groups, invoked by the Lambda function IsolateEc2. This assumes the incident response role in the target service account.
    3. Block the IAM principal by attaching a deny all policy, invoked by the Lambda function BlockPrincipal, by assuming the incident response role in the target service account.
    4. Confine security group to safe CIDR, invoked by the Lambda function ConfineSecurityGroup, by assuming the incident response role in the target service account.
    5. Send the finding to an SNS topic for processing outside this solution; for example, by manual evaluation or simply for information.
  2. Invoke AWS Systems Manager within the security account, with a target of the service account and the same AWS Region.
  3. The Systems Manager automation document is executed from the security account against the resources in the service account; for example, EC2, S3, or IAM resources.
  4. The response actions are executed directly from the security account to the service account. This is done by assuming an IAM role, for example, to isolate a potentially compromised EC2 instance.
  5. Manually trigger security responses using Security Hub custom actions. This can be suitable for manual handling of complex findings that need investigation before action.

Response actions decision

Your organization wants to articulate information security policy decisions and then create a list of corresponding automated security responses. Factors to consider include the classification of the resources and the technical complexity of the automation required. Start with common, less complex cases to get immediate gains, and increase complexity as you gain experience. Many stakeholders in your organization, such as business, IT operations, information security, and risk and compliance, should be involved in deciding which incident response actions should be automated. You need executive support for the political will to execute policies.

Creating exceptions to automated responses

There may be cases in which it’s unwise to take an automated response. For example, you might not want an automated response to incidents involving a core production database server that is critical to business operations. Instead, you’d want to use human judgment calls before responding. Or perhaps you know there are alarms that you don’t need for certain resources, like alerting for an open security group when you intentionally use it as a public web server. To address these exceptions, there is a carve-out. If the AWS resource has a tag with the name SecurityException, a response action isn’t executed. The tag name is defined during installation.

Table 1 provides an example of the responses implemented in this solution template. You might decide on different actions and different priorities for your organization. A current list of GuardDuty findings can be found at Active Finding Types and for AWS Config at AWS Config Managed Rules.

NSourceFindingDescriptionResponse
1GuardDuty Backdoor:EC2/Spambot
Backdoor:EC2/C&CActivity.B!DNS,
Backdoor:EC2/DenialOfService.Tcp,
Backdoor:EC2/DenialOfService.Udp,
Backdoor:EC2/DenialOfService.Dns,
Backdoor:EC2/DenialOfService.UdpOnTcpPorts,
Backdoor:EC2/DenialOfService.UnusualProtocol,
Trojan:EC2/BlackholeTraffic,
Trojan:EC2/DropPoint,
Trojan:EC2/BlackholeTraffic!DNS,
Trojan:EC2/DriveBySourceTraffic!DNS,
Trojan:EC2/DropPoint!DNS,
Trojan:EC2/DGADomainRequest.B,
Trojan:EC2/DGADomainRequest.C!DNS,
Trojan:EC2/DNSDataExfiltration,
Trojan:EC2/PhishingDomainRequest!DNS
See Backdoor Finding Types and
 
Trojan Finding Types
Isolate EC2 with empty security group.
 
Archive the GuardDuty finding.
 
Send SNS notification.
2GuardDutyUnauthorizedAccess:IAMUser/InstanceCredentialExfiltration,
UnauthorizedAccess:IAMUser/TorIPCaller,
UnauthorizedAccess:IAMUser/MaliciousIPCaller.Custom,
UnauthorizedAccess:IAMUser/ConsoleLoginSuccess.B,
UnauthorizedAccess:IAMUser/MaliciousIPCaller
See Unauthorized Finding TypesBlock IAM principal by attaching deny all policy.
 
Archive the GuardDuty finding.
 
Send SNS notification.
3AWS ConfigS3_BUCKET_SERVER_SIDE_ENCRYPTION_ENABLEDSee the documentation for s3-bucket-server-side-encryption-enabledEnable server-side encryption with Amazon S3-Managed keys (SSE-S3) with SSM document (AWS-EnableS3BucketEncryption).
 
Send SNS notification.
4AWS ConfigS3_BUCKET_PUBLIC_READ_PROHIBITEDSee the documentation for s3-bucket-public-read-prohibitedDisable S3 PublicRead and PublicWrite with SSM document (AWS-DisableS3BucketPublicReadWrite).
 
Send SNS notification.
5AWS ConfigS3_BUCKET_PUBLIC_WRITE_PROHIBITEDSee the documentation for s3-bucket-public-write-prohibitedDisable S3 PublicRead and PublicWrite with SSM document (AWS-DisableS3BucketPublicReadWrite).
 
Send SNS notification.
6AWS ConfigSECURITY_GROUP_OPEN_PROHIBITEDSee template, custom configuration.Confine security group to safe CIDR 172.31. 0.0/16
 
Send SNS notification.
7AWS ConfigENCRYPTED_VOLUMESSee the documentation for encrypted-volumesSend SNS notification.
8AWS ConfigRDS_STORAGE_ENCRYPTEDSee the documentation for rds-storage-encryptedSend SNS notification.

Installation

  1. In the security account, launch the template by selecting Launch Stack.
     
    Select this image to open a link that starts building the CloudFormation stack
    Additionally, you can find the latest code on GitHub, where you can also contribute to the sample code.
  2. Provide the following parameters for the security account (see Figure 2):
    • S3 bucket with sources: This bucket contains all sources, such as the Lambda function and templates. If you’re not customizing the sources, you can leave the default text.
    • Prefix for S3 bucket with sources: Prefix for all objects. If you’re not customizing the sources, you can leave the default.
    • Security IR-Role names: This is the role assumed for the response actions by the Lambda functions in the security account. The role is created by the stack launched in the service account.
    • Security exception tag: This defines the tag name for security exceptions. Resources marked with this tag are not automatically changed in response to a security finding. For example, you could add an exception tag for a valid open security group for a public website.
    • Organization ID: This is your AWS organization ID used to authorize forwarding of CloudWatch Events to the security account. Your accounts must be members of the same AWS organization.
    • Allowed network range IPv4: This CIDRv4 range is used to confine all open security groups that are not tagged for exception.
    • Allowed network range IPv6: This CIDRv6 range is used to confine all open security groups that are not tagged for exception.
    • Isolate EC2 findings: This is a list of all GuardDuty findings that should lead to an EC2 instance being isolated. Comma delimited.
    • Block principal finding: This is a list of all GuardDuty findings that should lead to blocking this role or user by attaching a deny all policy. Comma delimited.
    Figure 2: Stack launch in security account

    Figure 2: Stack launch in security account

  3. In each service account, launch the template by selecting Launch Stack.
     
    Select this image to open a link that starts building the CloudFormation stack

    Additionally, you can find the latest code on GitHub, where you can also contribute to the sample code.

  4. Provide the following parameters for each service account:
    • S3 bucket with sources: This bucket contains all sources, such as Lambda functions and templates. If you’re not customizing the sources, you can leave the default text.
    • Prefix for S3 bucket with sources: Prefix for all objects. If you’re not customizing the sources, you can leave the default text.
    • IR-Security role: This is the role that is created and used by the security account to execute response actions.
    • Security account ID: The CloudWatch Events are forwarded to this central security account.
    • Enable AWS Config: Define whether you want this stack to enable AWS Config. If you have already enabled AWS Config, then leave this value false.
    • Create SNS topic: Provide the name of an SNS topic only if you enable AWS Config and want to stream the configuration change to SNS (optional). Otherwise, leave this field blank.
    • SNS topic name: This is the name of the SNS topic to be created only if enabling AWS Config. The default text is Config.
    • Create S3 bucket for AWS Config: If you enable AWS Config, the template creates an S3 bucket for AWS Config.
    • Bucket name for AWS Config: The name of the S3 bucket created for AWS Config. The default text is config-bucket-{AccountId}.
    • Enable GuardDuty: If you have not already enabled GuardDuty in the service account, then you can do it here.

Testing

After you have deployed both stacks, you can test your environment by following these example steps in one of your service accounts. Before you test, you can subscribe to the SNS topic with prefix Security_Alerts_[Your_Stack] to be notified of a security event.

  1. Create and open security group 0.0.0.0/0 without creating an exception tag. After several minutes, the security group will be confined to the safe CIDR that you defined in your stack.
  2. Create an S3 bucket without enabling encryption. After several minutes, the default encryption AES-256 will be set on the bucket.
  3. For GuardDuty blocking of IAM principal, you can define a list of malicious IPs under the Threat List in the GuardDuty panel. Create a test role or user. When you execute an API call from this IP with the created test role or user, a GuardDuty finding is generated that triggers blocking of the IAM role or user.
  4. You can deploy Amazon GuardDuty tester and generate findings such as Trojan:EC2/DNSDataExfiltration or CryptoCurrency:EC2/BitcoinTool.B!DN. The GuardDuty findings trigger isolation of an EC2 instance by removing all current security groups and attaching an empty one. This new empty group can then be configured for forensic access later.

Exceptions for response actions

If the resource has the tag name SecurityException, a response action is not executed. The tag name is a parameter of the CloudFormation stack in the security account and can be customized at installation. The value of the tag is not validated, but it is good practice for the value to refer to an approval document such as a Jira issue. In this way, you can build an auditing chain of the approval. For example:


Tag name:  SecurityException
Tag value: Jira-1234

Make sure that the security tag can only be set or modified by an appropriate role. You can do this in two ways. The first way is to attach a deny statement to all policies that do not have privileges to assign this tag. An example policy statement to deny setting, removing, and editing of this tag for IAM, EC2, and S3 services is shown below. This policy does not prohibit working with the resources, such as starting or stopping an EC2 instance with this tag. See Controlling access based on tag keys for more information.


{
        {
            "Sid": "S3-Sec-Tag",
            "Effect": "Deny",
            "A	ction": [
                "s3:PutBucketTagging",
                "s3:PutObjectTagging"
            ],
            "Resource": "*",
            "Condition": {
                "ForAnyValue:StringLikeIfExists": {
                    "aws:TagKeys": "TAG-NAME-SecurityException"
                }
            }
        },
        {
            "Sid": "EC2-VPC-Sec-Tag",
            "Effect": "Deny",
            "Action": "*",
            "Resource": "*",
            "Condition": {
                "ForAnyValue:StringLikeIfExists": {
                    "aws:TagKeys": "TAG-NAME-SecurityException"
                }
            }
        }
    ]
}

In the above policy, you must modify the TAG-NAME-SecurityException to match your own tag name.

The second way to restrict the attachment of this tag is to use Tag policies to manage tags across multiple AWS accounts.

Centralized versus decentralized exception management

Attaching security tags is a decentralized approach in which you don’t need a centralized database to record remediation exceptions. A centralized exception database requires that you know each individual resource ID to set exceptions, and this is not always possible. Amazon EC2 Auto Scaling is a good example where you might not know the EC2 instance ID in advance. On an up-scaling event, the instance ID can’t be known in advance and preapproved. Furthermore, a central database must be kept in sync with the lifecycle of the resources, like with an instance down-scaling event. Hence, using tags on resources is a decentralized approach. If needed, the automatic scaling launch configuration can propagate the security exception tag; see Tag your auto scaled EC2 instances.

You can manage the IAM policies and roles for tagging either centrally or within an AWS CodePipeline. In this way, you can implement a centralized enforcement point for security exceptions but decentralize storing of the resource tags to the resources themselves.

Using AWS resource groups, you can always find all resources that have the Security tag for inventory and auditing proposes. For more information, see Build queries and groups in AWS resource groups.

Monitoring for security alerts

You can subscribe to the SNS topic with prefix Security_Alerts_[Your_Stack] to be notified of a security event or an execution that has failed.

Every execution is triggered from CloudWatch Events, and each of these events generates CloudWatch metrics. Under CloudWatch rules, you can see the rules for event forwarding from service accounts, or for triggering the responses in the central security account. For each CloudWatch rule, you can view the metrics by checking the box next to the rule, as shown in Figure 3, and then select Show metrics for the rule.
 

Figure 3: Metrics for Forwarding GuardDuty Events in CloudWatch

Figure 3: Metrics for Forwarding GuardDuty Events in CloudWatch

Creating new rules and customization

The mapping of findings to responses is defined in the CloudWatch Events rules in the security account. If you define new responses, you only update the central security account. You don’t adjust the settings of service accounts because they only forward events to the security account.

Here’s an example: Your team decides that a new SSM action – Terminate EC2 instance – should be triggered on the GuardDuty finding Backdoor:EC2/C&CActivity.B!DNS. In the security account, go to CloudWatch in the AWS Management Console. Create a new rule as described in the following steps (and shown in Figure 4):
 

Figure 4: CloudWatch rule creation

Figure 4: CloudWatch rule creation

  1. Go to CloudWatch Events and create a new rule. In the Event Pattern, under Build custom event pattern, define the following JSON:
    
    {
      "detail-type": [
        "GuardDuty Finding"
      ],
      "source": [
        "aws.guardduty"
      ],
      "detail": {
        "type": [
          "Backdoor:EC2/C&CActivity.B!DNS"
        ]
      }
    }
    

  2. Set the target for the rule as the Lambda function for execution of SSM, StratSsmAutomation. Use Input transformer to customize the parameters in invocation of the Lambda function.

    For the Input paths map:

    
    {
       "resourceId":"$.detail.resource.instanceDetails.instanceId",
       "account":"$.account"
    } 
    

    The expression extracts the instanceId from the event as resourceId, and the Account ID as account.

    For the field Input template, enter the following:

    
    {
       "account":<account>,
       "resourseType":"ec2:instance",
       "resourceId":<resourceId>,
       "AutomationDocumentName":"AWS-TerminateEC2Instance",
       "AutomationParameters":{
          "InstanceId":[
          <resourceId>   
          ]
       }
    }
    

You pass the name of the SSM automation document that you want to execute as an input parameter. In this case, the document is AWS-TerminateEC2Instance and the document input parameters are a JSON structure named AutomationParameters.

You have now created a new response action that is ready to test.

Troubleshooting

The automation execution that takes place in the security account is documented in AWS Systems Manager under Automation Execution. You can also refer to the AWS Systems Manager automation documentation.

If Lambda function execution fails, then a CloudWatch alarm is triggered, and notification is sent to an SNS topic with prefix Security_Alerts_[Your_Stack]. Lambda function execution in the security account is documented in the CloudWatch log group for the Lambda function. You can use the logs to understand whether the Lambda function was executed successfully.

Security Hub integration

AWS Security Hub aggregates alarms from GuardDuty and other providers. To get Security Hub to aggregate alarms, you must first activate Security Hub before deploying this solution. Then, invite your environment accounts to the security account as master. For more information, see Master and member accounts in AWS Security Hub.

The integration with Security Hub is for manual triggering and is not part of the automated response itself. This can be useful if you have findings that require security investigation before triggering the action.

From the Security Hub console, select the finding and use the Actions drop down menu in the upper right corner to trigger a response, as shown in Figure 5. A CloudWatch event invokes the automated response, such as a Lambda function to isolate an EC2 instance. The Lambda function fetches the GuardDuty finding from the service account and executes one of the following: Isolate EC2 Instance, Block Principal, or Send SNS Notification.

Only one resource can be selected for execution. If you select multiple resources, the response is not executed, and the Lambda log message is shown in CloudWatch Logs.
 

Figure 5: Security Hub integration

Figure 5: Security Hub integration

Cost of the solution

The cost of the solution depends on the events generated in your AWS account and chosen AWS Region. Costs might be as little as several U.S. dollars per month and per account. You can model your costs by understanding GuardDuty pricing, automation pricing in AWS system manager, and AWS Config pricing for six AWS Config rules. The costs for AWS Lambda and Amazon CloudWatch are expected to be minimal and might be included in your free tier. You can optionally use Security Hub; see AWS Security Hub pricing.

Summary

In this post, you learned how to deploy an automated incident response framework using AWS native features. You can easily extend this framework to meet your future needs. If you would like to extend it further, contact AWS professional services or an AWS partner. If you have technical questions, please use the Amazon GuardDuty or AWS Config forums. Remember, this solution is only an introduction to automated security response and is not a comprehensive solution.

If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Vesselin Tzvetkov

Vesselin is senior security consultant at AWS Professional Services and is passionate about security architecture and engineering innovative solutions. Outside of technology, he likes classical music, philosophy, and sports. He holds a Ph.D. in security from TU-Darmstadt and a M.S. in electrical engineering from Bochum University in Germany.

How to create SAML providers with AWS CloudFormation

Post Syndicated from Rich McDonough original https://aws.amazon.com/blogs/security/how-to-create-saml-providers-with-aws-cloudformation/

As organizations grow, they often experience an inflection point where it becomes impractical to manually manage separate user accounts in disparate systems. Managing multiple AWS accounts is no exception. Many large organizations have dozens or even hundreds of AWS accounts spread across multiple business units.

AWS provides many solutions that can orchestrate a person’s identity across multiple accounts. AWS Identity and Access Management (IAM), SAML, and OpenID can all help. The solution I describe in this post uses these features and services and can scale to thousands of AWS accounts. It provides a repeatable and automated means for deploying a unified identity management structure across all of your AWS environments. It does this while extending existing identity management into AWS without needing to change your current sources of user information.

About multi-account management

This approach uses AWS CloudFormation StackSets to deploy an identity provider and AWS IAM roles into multiple accounts. Roles may be tailored for your business needs and mapped to administrators, power users, or highly specialized roles that perform domain-specific tasks within your environment. StackSets are a highly scalable tool for multi-account orchestration and enforcement of policies across an organization.

Before deploying this solution across your organization, I recommend that you deploy the stack below into a single AWS account for testing purposes. Then, extend the solution using StackSets across your organization once tested. For more information about AWS CloudFormation Stacks and StackSets, see our documentation.

About SAML and SAML providers

Security Assertion Markup Language 2.0 (SAML) is an open standard for exchanging identity and security information with applications and other external systems. For example, if your organization uses Microsoft Active Directory, you can federate your user accounts to other organizations by way of SAML assertions. Actions that users take in the external organization can then be mapped to the identity of the original user and executed with a role that has predetermined privileges.

If you’re new to SAML, don’t worry! Not only is it ubiquitous, it’s also well documented. Many websites use it to drive their security and infrastructure, and you may have already used it without knowing it. For more information about SAML and AWS, I suggest you start with Enabling SAML for Your AWS Resources.

Properly configured SAML federation provides core security requirements for authentication and authorization, while maintaining a single source of truth for who your users are. It also enables important constraints such as multi-factor authentication and centralized access management. Within AWS IAM, you setup a SAML trust by configuring your identity provider with information about AWS and the AWS IAM roles that you want your federated users to use. SAML is secured using public key authentication.

This solution is a companion to the blog post AWS Federated Authentication with Active Directory Federation Services (AD FS), but it can be applied to any SAML provider.

Solution overview

To orchestrate SAML providers across multiple accounts, you will need the following:

  • A basic understanding of federated users.
  • An identity provider that supports SAML, such as Active Directory Federation Services (AD FS), or Shibboleth.
  • An Amazon Simple Storage Service (Amazon S3) bucket to host the provider’s federation metadata file.
  • An AWS account that can create StackSets in other accounts. Usually, this is an AWS Organizations master account, with StackSets provisioned into member accounts. See the AWS Organizations User Guide for more details.

Note: You can adapt this solution to retrieve federation metadata from a HTTPS endpoint, such as those published by publicly hosted identity providers, but this solution does not currently offer that.

At this time, CloudFormation does not directly support provisioning a SAML provider. However, you can accomplish this by using a custom resource Lambda function. The sample code provided in this post takes this one step further and deploys your federation metadata from a secure Amazon S3 bucket, as some organizations have requirements about how this metadata is stored.
 

Figure 1: Architecture diagram

Figure 1: Architecture diagram

Here’s what the solution architecture looks like, as shown in Figure 1:

  1. An AWS CloudFormation template is created within an AWS account. Most commonly this is your master account within AWS Organizations, but it can be a standalone account as well.
  2. The AWS CloudFormation template is deployed to other AWS accounts within your organization using AWS CloudFormation StackSets.
  3. The AWS CloudFormation StackSet creates the following resources:
    1. A new read-only role within AWS IAM that uses the new identity provider as the principal in the IAM role’s trust policy. This is a sample that I use to demonstrate how you can deploy IAM Roles that are bound to a single, dedicated identity provider.
    2. An execution role for a custom resource AWS Lambda function.
    3. The AWS Lambda function, which creates the identity provider within the destination account.
  4. The new AWS Lambda function is executed, pulling your federation metadata from the master account’s S3 Bucket and creating the new Identity Provider.

Step 1: Upload your federation metadata to a restricted S3 bucket

To ensure that your SAML provider’s federation metadata is exposed only to accounts within your AWS organization, you can upload the metadata to an Amazon S3 bucket and then restrict access to accounts within your organization. The bucket policy follows this template:


{
    "Version": "2012-10-17",
    "Id": "Policy1545178796950",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<replace with bucket name>/*",
            "Condition": {
                "StringEquals": {
                    "aws:PrincipalOrgID": "<replace with organization ID>"
                }
            }
        }
    ]
}

Once this Amazon S3 bucket policy is in place, upload your federation metadata document to the bucket and note the object URL.

If your organization can share your federation metadata publicly, then there’s no need for this bucket policy. Simply upload your file to the S3 bucket, and make the bucket publicly accessible.

Step 2: Launch the CloudFormation template

To create the SAML provider within AWS IAM, this solution uses a custom resource Lambda function, as CloudFormation does not currently offer the ability to create the configuration directly. In this case, I show you how to use the AWS API to create a SAML provider in each of your member accounts, pulling the federation metadata from an S3 bucket owned by the organization master account.

The actual CloudFormation template follows. A sample read-only role has been included, but I encourage you to replace this with IAM roles that make sense for your environment. Save this template to your local workstation, or to your code repository for tracking infrastructure changes (if you need a source code management system, we recommend AWS CodeCommit).


---
AWSTemplateFormatVersion: 2010-09-09
Description: Create SAML provider

Parameters:
  FederationName:
    Type: String
    Description: Name of SAML provider being created in IAM
  FederationBucket:
    Type: String
    Description: Bucket containing federation metadata
  FederationFile:
    Type: String
    Description: Name of file containing the federation metadata

Resources:
  FederatedReadOnlyRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: Federated-ReadOnly
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Action: sts:AssumeRoleWithSAML
            Principal:
              Federated: !Sub arn:aws:iam::${AWS::AccountId}:saml-provider/${FederationName}
            Condition:
              StringEquals:
                SAML:aud: https://signin.aws.amazon.com/saml
      Path: '/'
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/ReadOnlyAccess

  SAMLProviderCustomResourceLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - sts:AssumeRole
      Path: "/"
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                Resource: 'arn:aws:logs:*:*:log-group:/aws/lambda/*-SAMLProviderCustomResourceLambda-*:*'
              - Effect: Allow
                Action:
                  - iam:CreateSAMLProvider
                  - iam:DeleteSAMLProvider
                Resource: !Sub arn:aws:iam::${AWS::AccountId}:saml-provider/${FederationName}
              - Effect: Allow
                Action:
                  - iam:ListSAMLProviders
                Resource: '*'
              - Effect: Allow
                Action:
                  - s3:GetObject
                Resource: !Sub 'arn:aws:s3:::${FederationBucket}/*'

  CustomResource:
    Type: Custom::CustomResource
    DependsOn:
      - SAMLProviderCustomResourceLambda
      - SAMLProviderCustomResourceLambdaExecutionRole
    Properties:
      ServiceToken: !GetAtt SAMLProviderCustomResourceLambda.Arn

  SAMLProviderCustomResourceLambda:
    Type: "AWS::Lambda::Function"
    Properties:
      Handler: index.lambda_handler
      Role: !GetAtt SAMLProviderCustomResourceLambdaExecutionRole.Arn
      Runtime: python3.7
      Timeout: 300
      Environment:
        Variables:
          FEDERATION_NAME: !Ref FederationName
          FEDERATION_BUCKET: !Ref FederationBucket
          FEDERATION_FILE: !Ref FederationFile
      Code:
        ZipFile: |
          import boto3, json, os, urllib.request, ssl, time, traceback


          BUCKET = os.getenv('FEDERATION_BUCKET')
          FILE = os.getenv('FEDERATION_FILE')
          NAME = os.getenv('FEDERATION_NAME')


          class SAMLProvider(object):
              def __init__(self):
                  self.iam_client = boto3.client('iam')
                  self.existing_providers = []
                  self._list_saml_providers()
                  self.s3 = boto3.resource('s3')

              def get_federation_metadata(self):
                  try:
                      self.s3.Bucket(BUCKET).download_file(FILE, '/tmp/' + FILE)
                      handle = open('/tmp/' + FILE)
                      data = handle.read()
                      handle.close()
                      os.remove('/tmp/' + FILE)
                      return data
                  except:
                      traceback.print_exc()
                      raise

              def _list_saml_providers(self):
                  providers = []
                  response = self.iam_client.list_saml_providers()
                  for provider in response['SAMLProviderList']:
                      self.existing_providers.append(provider['Arn'])

              def add_saml_provider(self, name):
                  for arn in self.existing_providers:
                      if arn.split('/')[1] == name:
                          print(name + ' already exists as a provider')
                          return False
                  response = self.iam_client.create_saml_provider(SAMLMetadataDocument=self.get_federation_metadata(), Name=name)
                  print('Create response: ' + str(response))
                  return True

              def delete_saml_provider(self, name):
                  for arn in self.existing_providers:
                      if arn.split('/')[1] == name:
                          response = self.iam_client.delete_saml_provider(SAMLProviderArn=arn)
                          print('Delete response: ' + str(response))

          def send_response(event, context, response_status, response_data):
              response_body = json.dumps({
                  'Status': response_status,
                  'Reason': 'See the details in CloudWatch Log Stream: ' + context.log_stream_name,
                  'PhysicalResourceId': context.log_stream_name,
                  'StackId': event['StackId'],
                  'RequestId': event['RequestId'],
                  'LogicalResourceId': event['LogicalResourceId'],
                  'Data': response_data
              })
              print('ResponseURL: %s', event['ResponseURL'])
              print('ResponseBody: %s', response_body)
              try:
                  opener = urllib.request.build_opener(urllib.request.HTTPHandler)
                  request = urllib.request.Request(event['ResponseURL'], data=response_body.encode())
                  request.add_header('Content-Type', '')
                  request.add_header('Content-Length', len(response_body))
                  request.get_method = lambda: 'PUT'
                  response = opener.open(request)
                  print("Status code: %s", response.getcode())
                  print("Status message: %s", response.msg)
              except:
                  traceback.print_exc()


          def lambda_handler(event, context):
              print(event)
              print(context)
              saml = SAMLProvider()
              try:
                  if event['RequestType'] == 'Create':
                      saml.add_saml_provider(NAME)
                      send_response(event, context, 'SUCCESS', {"Message": "Resource creation successful!"})
                  if event['RequestType'] == 'Update':
                      saml.delete_saml_provider(NAME)
                      time.sleep(10)
                      saml.add_saml_provider(NAME)
                      send_response(event, context, 'SUCCESS', {"Message": "Resource update successful!"})
                  if event['RequestType'] == 'Delete':
                      saml.delete_saml_provider(NAME)
                      send_response(event, context, 'SUCCESS', {"Message": "Resource deletion successful!"})
              except:
                  send_response(event, context, "FAILED", {"Message": "Exception during processing"})
                  traceback.print_exc()

As you review this template, note that a Python Lambda function is embedded in the template body itself. This is a useful way to package code along with your CloudFormation templates. The limit for the length of code inline within a CloudFormation template is fixed, as detailed in the AWS CloudFormation user guide. Code that exceeds this length must be saved to a separate location within an Amazon S3 bucket and referenced from your template. However, the preceding snippet is short enough to be included all in one CloudFormation template.

Step 3: Create the StackSet

With your SAML provider metadata securely uploaded to your Amazon S3 bucket, it’s time to create the StackSet.

  1. First, navigate to the CloudFormation console and select StackSets, then Create StackSet.
     
    Figure 2: Creating a new StackSet

    Figure 2: Creating a new StackSet

  2. Select Template is ready, then Upload a template file. Select Choose file to choose the location of the CloudFormation template, then select Next.
     
    Figure 3: Specifying the template details

    Figure 3: Specifying the template details

  3. Enter a value for StackSet name. Then provide the following configuration parameters and select Next:
    1. FederationName: The name of the SAML provider. This is the name of the provider as you see it in IAM once provisioned, and it appears in the ARN of the SAML provider.
    2. FederationFile: The file name within S3.
    3. FederationBucket: The name of the S3 bucket.

     

    Figure 4: Specifying the StackSet parameters

    Figure 4: Specifying the StackSet parameters

  4. In the Account numbers field, enter the list of accounts into which you wish to provision this stack. Select a Region from the drop-down menu, then select any concurrent deployment options that you need. For most customers, the concurrent and failure tolerance options generally remain unmodified.

    Important: When you enter your list of accounts, you must use the 12-digit account number for each destination, without spaces, and separated by commas.

    Because IAM is a global service, there is no need to deploy this to multiple regions. The template must be deployed once per AWS account, although it is managed by a CloudFormation Stack that exists in a single Region.
     

    Figure 5: Selecting your deployment options

    Figure 5: Selecting your deployment options

  5. Depending on your organization’s security posture, you may have chosen to use self-managed permissions. If that is the case, then you must specify an AWS IAM role that is permitted to execute StackSet deployments, as well as an execution role in the destination accounts. For more info, visit the StackSets’ Grant Self-Managed Permissions documentation page.
  6. Review your selections, accept the IAM role creation acknowledgment, and click Submit to create your stacks. Once completed, ensure that the new identity provider is present in each account by visiting the AWS IAM console.

Summary

Creating a SAML provider across multiple AWS accounts enables enterprises to extend their existing authentication infrastructure into the cloud.

You can extend the solution provided here to deploy different federation metadata to AWS accounts based on specific criteria—for example, if your development and staging accounts have a separate identity provider. You can also add simple logic to the Lambda function that I’ve included in the CloudFormation template. Additionally, you might need to create any number of predefined roles within your AWS organization—using the StackSets approach enables a zero-touch means for creating these roles from a central location.

Organizations looking for even stronger controls over multiple accounts should implement this solution with AWS Organizations. Combined with federated access and centrally managed IAM roles, Organizations provides Service Control Policies and a common framework for deploying CloudFormation Stacks across multiple business units.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Rich McDonough

Rich McDonough is a Solutions Architect for Amazon Web Services based in Toronto. His primary focus is on Management and Governance, helping customers scale their use of AWS safely and securely. Before joining AWS two years ago, he specialized in helping migrate customers into the cloud. Rich loves helping customers learn about AWS CloudFormation, AWS Config, and AWS Control Tower.

Introducing the Well-Architected Framework for Machine Learning

Post Syndicated from Shelbee Eigenbrode original https://aws.amazon.com/blogs/architecture/introducing-the-well-architected-framework-for-machine-learning/

We have published a new whitepaper, Machine Learning Lens, to help you design your machine learning (ML) workloads following cloud best practices. This whitepaper gives you an overview of the iterative phases of ML and introduces you to the ML and artificial intelligence (AI) services available on AWS using scenarios and reference architectures.

How often are you asking yourself “Am I doing this right?” when building and running applications in the cloud? You are not alone. To help you answer this question, we released the AWS Well-Architected Framework in 2015. The Framework is a formal approach for comparing your workload against AWS best practices and getting guidance on how to improve it.

We added “lenses” in 2017 that extended the Framework beyond its general perspective to provide guidance for specific technology domains, such as the Serverless Applications Lens, High Performance Computing (HPC) Lens, and IoT (Internet of Things) Lens. The new Machine Learning Lens further extends that guidance to include best practices specific to machine learning workloads.

A typical question we often hear is: “Should I use both the Lens whitepaper and the AWS Well-Architected Framework whitepaper when reviewing a workload?” Yes! We purposely exclude topics covered by the Framework in the Lens whitepapers. To fully evaluate your workload, use the Framework along with any applicable Lens whitepapers.

Applying the Machine Learning Lens

In the Machine Learning Lens, we focus on how to design, deploy, and architect your machine learning workloads in the AWS Cloud. The whitepaper starts by describing the general design principles for ML workloads. We then discuss the design principles for each of the five pillars of the Framework—operational excellence, security, reliability, performance efficiency, and cost optimization—as they relate to ML workloads.

Although the Lens is written to provide guidance across all pillars, each pillar is designed to be consumable by itself. This design results in some intended redundancy across the pillars. The following figure shows the structure of the whitepaper.

ML Lens-Well Architected (1)

The primary components of the AWS Well-Architected Framework are Pillars, Design Principles, Questions, and Best Practices. These components are illustrated in the following figure and are outlined in the AWS Well-Architected Framework whitepaper.

WA-Failure Management (2)

The Machine Learning Lens follows this pattern, with Design Principles, Questions, and Best Practices tailored for machine learning workloads. Each pillar has a set of questions, mapped to the design principles, which drives best practices for ML workloads.

ML Lens-Well Architected (2)

To review your ML workloads, start by answering the questions in each pillar. Identify opportunities for improvement as well as critical items for remediation. Then make a prioritized plan to address the improvement opportunities and remediation items.

We recommend that you regularly evaluate your workloads. Use the Lens throughout the lifecycle of your ML workload—not just at the end when you are about to go into production. When used during the design and implementation phase, the Machine Learning Lens can help you identify potential issues early, so that you have more time to address them.

Available now

The Machine Learning Lens is available now. Start using it today to review your existing workloads or to guide you in building new ones. Use the Lens to ensure that your ML workloads are architected with operational excellence, security, reliability, performance efficiency, and cost optimization in mind.

Easily control the naming of individual IAM role sessions

Post Syndicated from Derrick Oigiagbe original https://aws.amazon.com/blogs/security/easily-control-naming-individual-iam-role-sessions/

AWS Identity and Access Management (IAM) now has a new sts:RoleSessionName condition element for the AWS Security Token Service (AWS STS), that makes it easy for AWS account administrators to control the naming of individual IAM role sessions. IAM roles help you grant access to AWS services and resources by using dynamically generated short-term credentials. Each instantiation of an IAM role, and the associated set of short-term credentials, is known as an IAM role session. Each IAM role session is uniquely identified by a role session name. You can now use the new condition to control how IAM principals and applications name their role sessions when they assume an IAM role, and rely on the role session name to easily track their actions when viewing AWS CloudTrail logs.

How do you name a role session?

There are different ways to name a role session, and it depends on the method used to assume the IAM role. In some cases, AWS sets the role session on your behalf. For example, for Amazon Elastic Compute Cloud (Amazon EC2) instance profiles, AWS sets the role session name to the instance profile ID. When you use the AssumeRolewithSAML API to assume an IAM role, AWS sets the role session name value to the attribute provided by the identity provider, which your administrator defined. In other cases, you provide the role session name when assuming the IAM role. For example, when assuming an IAM role with APIs such as AssumeRole or AssumeRoleWithWebIdentity, the role session name is a required input parameter that you set when making the API request.

What is a condition element?

A condition is an optional IAM policy element. You can use a condition to specify the circumstances under which the IAM policy grants or denies permissions. A condition includes a condition key, operator, and value for the condition.

There are two types of conditions: service-specific conditions and global conditions. Service-specific conditions are specific to certain actions in an AWS service. For example, specific EC2 actions support the ec2:InstanceType condition. All AWS service actions support global conditions.

Now that I’ve explained the meaning of a role session name and the meaning of a condition element in an IAM policy, let me introduce the new condition, sts:RoleSessionName.

sts:RoleSessionName condition

The sts:RoleSessionName is a service-specific condition that you use with the AssumeRole API action, in an IAM policy to control what is set as the role session name. You can use any string operator, such as StringLike, when using this condition.

Condition KeyDescriptionOperator(s)Value
sts:RoleSessionNameUniquely identifies a session when IAM principals, federated identities, and applications assume the same IAM role.All string operatorsString of upper-case and lower-case alphanumeric characters with no spaces. It can include underscores or any of the following characters: =,[email protected]IAM policy element variables can be set as values.

In this post, I will walk you through two examples of how to use the sts:RoleSessionName condition. In the first example, you will learn how to require IAM users to set their aws:username as their role session name when they assume an IAM role in your AWS account. In the second example, you will learn how to require IAM principals to choose from a pre-selected set of role session names when they assume an IAM role in your AWS account.

The examples shared in this post describe a scenario in which you have pricing data that is stored in an Amazon DynamoDB database in your AWS account, and you want to share the pricing data with members from your marketing department, who are in a different AWS account. In addition, you want to use your AWS CloudTrail logs to track the activities of members from the marketing department whenever they access the pricing data. This post will show you how to achieve this by doing the following:

  1. Dedicate an IAM role in your AWS account for the marketing department.
  2. Define the role trust policy for the IAM role, to specify who can assume the IAM role.
  3. Use the new sts:RoleSessionName condition in the role trust policy to define the allowed role session name values for the dedicated IAM role.

When members from the marketing department attempt to assume the IAM role in your AWS account, AWS will verify that their role session name does not conflict with the IAM role trust policy, before authorizing the assume-role action. The new sts:RoleSessionName condition gives you control of the role session name. With this control, when you view the AWS CloudTrail logs, you can now rely on the role session name for any of the following information:

  • To identify the IAM principal or application that assumed an IAM role.
  • The reason why the IAM principal or application assumed an IAM role.
  • To track the actions performed by the IAM principal or application with the assumed IAM role.

Example 1 – Require IAM users to set their aws:username as their role session name when they assume an IAM role in your AWS account

When an IAM user assumes an IAM role in your AWS account, you can require them to set their aws:username as the role session name. With this requirement, you can rely on the role session name to identify the IAM user who performed an action with the IAM role.

This example continues the scenario of sharing pricing data with members of the marketing department within your organization, who are in a different AWS account. John is a member of the marketing department, he is an IAM user in the marketing AWS account and his aws:username is john_s. For John to access the pricing data in your AWS account, you first create a dedicated IAM role for the marketing department, called marketing. John will assume the marketing IAM role to access the pricing information in your AWS account.

Next, you establish a two-way trust between the marketing AWS account and your AWS account. The administrator of the marketing AWS account will need to grant John sts:AssumeRole permission with an IAM policy, so that John can assume the marketing IAM role in your AWS account. The following is a sample policy to grant John assume-role permission. Be sure to replace <AccountNumber> with your account number.


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AssumeRole",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::<AccountNumber>:role/marketing"
        }
    ]
}

You then create a role trust policy for the marketing IAM role, which permits members of the marketing department to assume the IAM role. The following is a sample policy to create a role trust policy for the marketing IAM role. Be sure to replace <AccountNumber> with your account number.


{
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::<AccountNumber>:root"
          },
          "Action": "sts:AssumeRole",
          "Condition": {
            "StringLike": {
              "sts:RoleSessionName": "${aws:username}"
            }
          }
        }
      ]
    }

In the role trust policy above, you use the sts:RoleSessionName condition to ensure that members of the marketing department set their aws:username as their role session name when they assume the marketing IAM role. If John attempts to assume the marketing IAM role and does not set his role session name to john_s, then AWS will not authorize the request. When John sets the role session name to his aws:username, AWS will permit him to assume the marketing IAM role. The following is a sample CLI command to assume an IAM role. Replace <AccountNumber> with your account number.


aws sts assume-role --role-arn arn:aws:iam::<AccountNumber>:role/marketing --role-session-name john_s

In the AWS CLI command above, John assumes the marketing IAM role and sets the role session name to john_s. John then calls the get-caller-identity API to verify that he assumed the marketing IAM role. The following is confirmation that John successfully assumed the marketing IAM role.


{
    "UserId": " AIDACKCEVSQ6C2EXAMPLE:john_s",
    "Account": "<AccountNumber>",
    "Arn": "arn:aws:sts::<AccountNumber>:assumed-role/marketing/john_s"
}

AWS CloudTrail captures any action that John performs with the marketing IAM role, and you can easily identify John’s sessions in your AWS CloudTrail logs by searching for any Amazon Resource Name (ARN) with John’s aws:username (which is john_s) as the role session name. The following is an example of AWS CloudTrail event details that shows the role session name. Replace <AccountNumber> with your account number.



"assumedRoleUser": {
            "assumedRoleId": "AIDACKCEVSQ6C2EXAMPLE:john_s",
            "arn": "arn:aws:sts::<AccountNumber>:assumed-role/marketing/john_s"
        }

Example 2 – Require IAM principals to choose from a pre-selected set of role session names when they assume an IAM role in your AWS account

You can also define the acceptable role session names that an IAM principal or application can use when they assume an IAM role in your AWS account. With this requirement, you ensure that IAM principals and applications that assume IAM roles in your AWS account use a pre-approved role session name that you can easily understand.

Expanding on the previous example, in the following scenario, you have a new AWS account with an Amazon DynamoDB database that stores competitive analysis data. You do not want members of the marketing department to have direct access to this new AWS account. You will achieve this by requesting your marketing partners to first assume the marketing IAM role in your other AWS account with pricing information, and from that AWS account, assume the Analyst IAM role in the new AWS account to access the competitive analysis data. Also, you want your marketing partners to select from a pre-defined set of role session names: “marketing-campaign”, “product-development” and “other”, which will identify their reason for accessing the competitive analysis data.

First, you establish a two-way trust. You grant the marketing IAM role sts:AssumeRole permission with an IAM policy. The following is a sample policy to grant the marketing IAM role assume-role permission. Be sure to replace <AccountNumber> with your account number.


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AssumeRole",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::<AccountNumber>:role/Analyst"
        }
    ]
}

Next, you create a role trust policy for the Analyst IAM role. In the role trust policy, you set the marketing IAM role as the Principal, to restrict who can access the Analyst IAM role. Then you use the sts:RoleSessionName condition to define the acceptable role session names: marketing-campaign, product-development and other. The following is a role trust policy to limit the list of acceptable role session names. Replace <AccountNumber> with your account number.


        {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
             "AWS": " arn:aws:iam::<AccountNumber>:role/marketing"          
             },
          "Action": "sts:AssumeRole",
          "Condition": {
            "StringLike": {
              "sts:RoleSessionName": [
                "marketing-campaign",
                "product-development",
                "other"
              ]
            }
          }
        }
      ]
    }

If John from the marketing department wants to access the competitive analysis data, and he has assumed the marketing IAM role as shown in example #1, then he can assume the Analyst IAM role in the new AWS account by using the marketing IAM role. For AWS to authorize the assume-role request, when he assumes the Analyst IAM role, he must set the role session name to one of the pre-defined values. The following is a sample CLI command to assume the Analyst IAM role. Replace <AccountNumber> with your account number.


aws sts assume-role --role-arn arn:aws:iam::<AccountNumber>:role/Analyst --role-session-name marketing-campaign

In the CLI command above, John assumes the Analyst IAM role, using the marketing IAM role. He also sets the role session name to marketing-campaign, which is an allowed role session name. John then calls the get-caller-identity API to verify that he successfully assumed the Analyst IAM role. The following log results show the marketing IAM role successfully assumed the Analyst IAM role with the role session name as marketing-campaign.


{
    "UserId": " AIDACKCEVSQ6C2EXAMPLE:marketing-campaign",
    "Account": "<AccountNumber>",
    "Arn": "arn:aws:sts::<AccountNumber>:assumed-role/Analyst/marketing-campaign"
}

AWS CloudTrail captures any action performed with the Analyst IAM role. By viewing the role session names in your AWS CloudTrail logs, you can easily identify the reasons why your marketing partners accessed the competitive analysis data.


"assumedRoleUser": {
            "assumedRoleId": "AIDACKCEVSQ6C2EXAMPLE:marketing-campaign",
            "arn": "arn:aws:sts::<AccountNumber>:assumed-role/Analyst/marketing-campaign"
        }

Conclusion

In this post, I showed how AWS account administrators can use the sts:RoleSessionName condition to control what IAM principal names their session when they assume an IAM role. This control gives you increased confidence to rely on the role session name, when viewing AWS CloudTrail logs, to identify who performed an action with an IAM role, or get additional context for why an IAM principal assumed an IAM role.

For more information about the sts:RoleSessionName condition, and for policy examples, see Available Keys for AWS STS in the AWS IAM User Guide.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the Amazon Identity and Access Management forum.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Derrick Oigiagbe

Derrick is a Senior Product Manager for Identity and Access Management service at AWS. Prior to his career at Amazon, he received his MBA from the Carnegie Mellon’s Tepper School of Business. Derrick spent his early career as a technology consultant for Summa Technologies (recently acquired by CGI). In his spare time, Derrick enjoys playing soccer and travelling.

BBVA: Helping Global Remote Working with Amazon AppStream 2.0

Post Syndicated from Joe Luis Prieto original https://aws.amazon.com/blogs/architecture/bbva-helping-global-remote-working-with-amazon-appstream-2-0/

This post was co-written with Javier Jose Pecete, Cloud Security Architect at BBVA, and Javier Sanz Enjuto, Head of Platform Protection – Security Architecture at BBVA.

Introduction

Speed and elasticity are key when you are faced with unexpected scenarios such as a massive employee workforce working from home or running more workloads on the public cloud if data centers face staffing reductions. AWS customers can instantly benefit from implementing a fully managed turnkey solution to help cope with these scenarios.

Companies not only need to use technology as the foundation to maintain business continuity and adjust their business model for the future, but they also must work to help their employees adapt to new situations.

About BBVA

BBVA is a customer-centric, global financial services group present in more than 30 countries throughout the world, has more than 126,000 employees, and serves more than 78 million customers.

Founded in 1857, BBVA is a leader in the Spanish market as well as the largest financial institution in Mexico. It has leading franchises in South America and the Sun Belt region of the United States and is the leading shareholder in Turkey’s Garanti BBVA.

The challenge

BBVA implemented a global remote working plan that protects customers and employees alike, including a significant reduction of the number of employees working in its branch offices. It also ensured continued and uninterrupted operations for both consumer and business customers by strengthening digital access to its full suite of services.

Following the company’s policies and adhering to new rules announced by national authorities in the last weeks, more than 86,000 employees from across BBVA’s international network of offices and its central service functions now work remotely.

BBVA, subject to a set of highly regulated requirements, and was looking for a global architecture to accommodate remote work. The solution needed to be fast to implement, adaptable to scale out gradually in the various countries in which it operates, and able to meet its operational, security, and regulatory requirements.

The architecture

BBVA selected Amazon AppStream 2.0 for particular use cases of applications that, due to their sensitivity, are not exposed to the internet (such as financial, employee, and security applications). Having had previous experience with the service, BBVA chose AppStream 2.0 to accommodate the remote work experience.

AppStream 2.0 is a fully managed application streaming service that provides users with instant access to their desktop applications from anywhere, regardless of what device they are using.

AppStream 2.0 works with IT environments, can be managed through the AWS SDK or console, automatically scales globally on demand, and is fully managed on AWS. This means there is no hardware or software to deploy, patch, or upgrade.

AppStream 2.0 can be managed through the AWS SDK (1)

  1. The streamed video and user inputs are sent over HTTPS and are SSL-encrypted between the AppStream 2.0 instance executing your applications, and your end users.
  2. Security groups are used to control network access to the customer VPC.
  3. AppStream 2.0 streaming instance access to the internet is through the customer VPC.

AppStream 2.0 fleets are created by use case to apply security restrictions depending on data sensitivity. Setting clipboard, file transfer, or print to local device options, the fleets control the data movement to and from employees’ AppStream 2.0 streaming sessions.

BBVA relies on a proprietary service called Heimdal to authenticate employees through the corporate identity provider. Heimdal calls the AppStream 2.0 API CreateStreamingURL operation to create a temporary URL to start a streaming session for the specified user, and tries to abstract the user from the service using:

  • FleetName to connect the most appropriate fleet based on the user’s location (BBVA has fleets deployed in Europe and America to improve the user’s experience.)
  • ApplicationId to launch the proper application without having to use an intermediate portal
  • SessionContext in situations where, for instance, the authentication service generates a token and needs to be forwarded to a browser application and injected as a session cookie

BBVA uses AWS Transit Gateway to build a hub-and-spoke network topology (2)

To simplify its overall network architecture, BBVA uses AWS Transit Gateway to build a hub-and-spoke network topology with full control over network routing and security.

There are situations where the application streamed in AppStream 2.0 needs to connect:

  1. On-premises, using AWS Direct Connect plus VPN providing an IPsec-encrypted private connection
  2. To the Internet through an outbound VPC proxy with domain whitelisting and content filtering to control the information and threats in the navigation of the employee

AppStream 2.0 activity is logged into a centralized repository support by Amazon S3 for detecting unusual behavior patterns and by regulatory requirements.

Conclusion

BBVA built a global solution reducing implementation time by 90% compared to on-premises projects, and is meeting its operational and security requirements. As well, the solution is helping with the company’s top concern: protecting the health and safety of its employees.

Deploying a ASP.NET Core web application to Amazon ECS using an Azure DevOps pipeline

Post Syndicated from John Formento original https://aws.amazon.com/blogs/devops/deploying-a-asp-net-core-web-application-to-amazon-ecs-using-an-azure-devops-pipeline/

For .NET developers, leveraging Team Foundation Server (TFS) has been the cornerstone for CI/CD over the years. As more and more .NET developers start to deploy onto AWS, they have been asking questions about using the same tools to deploy to the AWS cloud. By configuring a pipeline in Azure DevOps to deploy to the AWS cloud, you can easily use familiar Microsoft development tools to build great applications.

Solution overview

This blog post demonstrates how to create a simple Azure DevOps project, repository, and pipeline to deploy an ASP.NET Core web application to Amazon ECS using Azure DevOps. The following screenshot shows a high-level architecture diagram of the pipeline:

 

Solution Architecture Diagram

In this example, you perform the following steps:

  1. Create an Azure DevOps Project, clone project repo, and push ASP.NET Core web application.
  2. Create a pipeline in Azure DevOps
  3. Build an Amazon ECS Cluster, Task and Service.
  4. Kick-off deployment of the ASP.Net Core web application using the newly create Azure DevOps pipeline.

 

Prerequisites

Ensure you have the following prerequisites set up:

  • An Amazon ECR repository
  • An IAM user with permissions for Amazon ECR and Amazon ECS (the user will need an access key and secret access key)

 

Create an Azure DevOps Project, clone project repo, and push ASP.NET Core web application

Follow these steps to deploy a .NET Core app onto your Amazon ECS cluster using the Azure DevOps (ADO) repository and pipeline:

 

  1. Login to dev.azure.com and navigate to the marketplace.
  2. Go to Visual Studio, search for “AWS”, and add the AWS Tools for Microsoft Visual Studio Team Services.
  3. Create a project in ADO: Provide a project name and choose Create.
  4. On the Project Summary page, choose Project Settings.
  5. In the Project Settings pane, navigate to the Service Connections page.
  6. Choose Create service connection, select AWS, and choose Next.
  7. Input an Access Key ID and Secret Access Key. (You’ll need an IAM user with permissions for Amazon ECR and Amazon ECS in order to deploy via the Azure DevOps pipeline.) Choose Save.
  8. Choose Repos in the left pane, then Clone in Visual Studio under Clone to your computer.
  9. Create a ASP.NET Core web application in Visual Studio, set the location to locally cloned repository, and check Enable Docker support.
  10. Once you’ve created the new project, perform an initial commit and push to the repository in Azure DevOps.

 

Creating a pipeline in Azure DevOps

Now that you have synced the repository, create a pipeline in Azure DevOps.

  1. Go to the pipeline page within Azure DevOps and choose Create Pipeline.
  2. Choose Use the classic editor.Pipeline configuration with repository
  3. Select Azure Repos Git for the location of your code and select the repository you created earlier.
  4. On the Choose a Template page, select Docker Container and choose Apply.
  5. Remove the Push an image step.
  6. Add an Amazon ECR Push task by choosing the + symbol next to Agent job 1. You can search for “AWS” in the Add tasks pane to filter for all AWS tasks.

 

Now, configure each task:

  1. Choose the Build an image task and ensure that the action is set to Build an image. Additionally, you can modify the Image Name to your standards.Pipeline configuration page Azure DevOps
  2. Choose the Push Image task and provide the following
    • Enter a name under Display Name.
    • Select the AWS Credentials that you created in Service Connections.
    • Select the AWS Region.
    • Provide the source image name, which you can find in the setting for the Build an image task.
    • Enter the name of the repository in Amazon ECR to which the image is pushedPipeline configuration page Azure DevOps
  3. Choose Save and queue.

Build Amazon ECS Cluster, Task, and Service

The goal here is to test up to building the Docker image and ensure it’s pushed to Amazon ECR. Once the Docker image is in Amazon ECR, you can create the Amazon ECS cluster, task definition, and service leveraging the newly created Docker image.

  1. Create an Amazon ECS cluster.
  2. Create an Amazon ECS task definition. When you create the task definition and configure the container, use the Amazon ECR URI for the Docker image that was just pushed to Amazon ECR.
  3. Create an Amazon ECS service.

Go back and edit the pipeline:

  1. Add the last step by choosing the + symbol next to Agent job 1.
  2. Search for “AWS CLI” in the search bar and add the task.
  3. Choose AWS CLI and configure the task.
  4. Enter a name under Display Name, such as Update ECS Service.
  5. Select the AWS Credentials that you created in Service Connections.
  6. Select the AWS Region.
  7. Input the following command, which updates the Amazon ECS service after a new image is pushed to Amazon ECR. Replace <clustername> and <servicename> with your Amazon ECS cluster and service names.
    • Command:ecs
    • Subcommand:update-service
    • Options and parameters: --cluster <clustername> --service <servicename> --force-new-deployment
  8. Now choose the Triggers tab and select Enable continuous integration with the repository you created.
  9. Choose Save and queue.

 

At this point, your build pipeline kicks off and builds a Docker image from the source code in the repository you created, pushes the image to Amazon ECR, and updates the Amazon ECS service with the new image.

You can verify by viewing the build. Choose Pipelines in Azure DevOps, selecting the entry for the latest run, and then the icon under the status column. Once it successfully completes, you can log in to the AWS console and view the updated image in Amazon ECR and the updated service in Amazon ECS.Pipeline status page Azure DevOps

Every time you commit and push your code through Visual Studio, this pipeline kicks off and builds and deploys your application to Amazon ECS.

Cleanup

At the end of this example, once you’ve completed all steps and are finished testing, follow these steps to disable or delete resources to avoid incurring costs:

  1. Go to the Amazon ECS console within the AWS Console.
  2. Navigate to the cluster you created, then choose the Tasks tab.
  3. Choose Stop all to turn off the tasks.

Conclusion

This blog post reviewed how to create a CI/CD pipeline in Azure DevOps to deploy a Docker Image to Amazon ECR and container to Amazon ECS. It provided detailed steps on how to set up a basic CI/CD pipeline, leveraging tools with which .NET developers are familiar and the steps needed to integrate with Amazon ECR and Amazon ECS.

I hope this post was informative and has helped you learn the basics of how to integrate Amazon ECR and Amazon ECS with Azure DevOps to create a robust CI/CD pipeline.

About the Authors

John Formento

 

 

John Formento is a Solution Architect at Amazon Web Services. He helps large enterprises achieve their goals by architecting secure and scalable solutions on the AWS Cloud.

Serving Billions of Ads in Just 100 ms Using Amazon Elasticache for Redis

Post Syndicated from Rodrigo Asensio original https://aws.amazon.com/blogs/architecture/serving-billions-of-ads-with-amazon-elasticache-for-redis/

This post was co-written with Lucas Ceballos, CTO of Smadex

Introduction

Showing ads may seem to be a simple task, but it’s not. Showing the right ad to the right user is an incredibly complex challenge that involves multiple disciplines such as artificial intelligence, data science, and software engineering. Doing it one million times per second with a 100-ms constraint is even harder.

In the ad-tech business, speed and infrastructure costs are the keys to success. The less the final user waits for an ad, the higher the probability of that user clicking on the ad. Doing that while keeping infrastructure costs under control is crucial for business profitability.

About Smadex

Smadex is the leading mobile-first programmatic advertising platform specifically built to deliver best user acquisition performance and complete transparency.

Its state-of-the-art digital signal processing (DSP) technology provides advertisers with the tools they need to achieve their goals and ROI, with measurable results from web forms, post-app install events, store visits, and sales.

Smadex advertising architecture

What does showing ads look like under the hood? At Smadex, our technology works based on the OpenRTB (Real-Time Bidding) protocol.

RTB is a means by which advertising inventory is bought and sold on a per-impression basis, via programmatic instantaneous auction, which is similar to financial markets.

To show ads, we participate in auctions deciding in real time which ad to show and how much to bid trying to optimize the cost of every impression.

High level diagram

  1. The final user browses the publisher’s website or app.
  2. Ad-exchange is called to start a new auction.
  3. Smadex receives the bid request and has to decide which ad to show and how much to offer in just 100 ms (and this is happening one million times per second).
  4. If Smadex won the auction, the ad must be sent and rendered on the publisher’s website or app.
  5. In the end, the user interacts with the ad sending new requests and events to Smadex platform.

Flow of data

As you can see in the previous diagram, showing ads is just one part of the challenge. After the ad is shown, the final user interacts with it in multiple ways, such as clicking it, installing an application, subscribing to a service, etc. This happens during a determined period that we call the “attribution window.” All of those interactions must be tracked and linked to the original bid transaction (using the request_id parameter).

Doing this is complicated: billions of bid transactions must be stored and available so that they can be quickly accessed every time the user interacts with the ad. The longer we store the transactions, the longer we can “wait” for an interaction to take place, and the better for our business and our clients, too.

Detailed diagram

Challenge #1: Cost

The challenge is: What kind of database can store billions of records per day, with at least a 30-day retention capacity (attribution window), be accessed by key-value, and all by spending as little as possible?

The answer is…none! Based on our research, all the available options that met the technical requirements were way out of our budget.

So…how to solve it? Here is when creativity and the combination of different AWS services comes into place.

We started to analyze the time dispersion of the events trying to find some clues. The interesting thing we spotted was that 90% of what we call “post-bid events” (impression, click, install, etc.) happened within one hour after the auction took place.

That means that we can process 90% of post-bid events by storing just one hour of bids.

Under our current workload, in one hour we participate in approximately 3.7 billion auctions generating 100 million bid records of an average 600 bytes each. This adds up to 55 gigabytes per hour, an easier amount of data to process.

Instead of thinking about one single database to store all the bid requests, we decided to split bids into two different categories:

  • Hot Bid: A request that took place within the last hour (small amount and frequently accessed)
  • Cold Bid: A request that took place more than our hour ago (huge amount and infrequently accessed)

Amazon ElastiCache for Redis is the best option to store 55 GB of data in memory, which gives us the ability to query in a key-value way with the lowest possible latency.

Hot Bids flow

Hot Bids flow diagram

  1. Every new bid is a hot bid by definition so it’s going to be stored in the hot bids Redis cluster.
  2. At the moment of the user interaction with the ad, the Smadex tracker component receives an HTTPS notification, including the bid request UUID that originated it.
  3. Based on the date of occurrence extracted from the received UUID, the tracker component can determine if it’s looking for a hot bid or not. If it’s a hot bid, the tracker reads it directly from Redis performing a key-value lookup query.

It’s been easy so far but what to do with the other 29 days and 23 hours we need to store?

Challenge #2: Performance

As we previously mentioned, cold bids are a huge infrequently accessed number of records with only 10% of post-bid events pointing to them. That sounds like a good use case for an inexpensive and slower data store like Amazon S3.

Thanks to the S3 low-cost storage prices combined with the ability to query S3 objects directly using Amazon Athena, we were able to optimize our costs by storing and querying cold bids by implementing a serverless architecture.

Cold Bids Flow

Cold Bids flow diagram

  1. Incoming bids are buffered by Fluentd and flushed to S3 every one minute in JSON format. Every single file flushed to S3 contains all the bids processed by a specific EC2 instance for one minute.
  2. An AWS Lambda function is automatically triggered on every new PutObject event from S3. This function transforms the JSON records to Parquet format and will save it back the S3 bucket, but this time into a specific partition folder based on file creation timestamp.
  3. As seen on the hot bids flow, the tracker component will determine if it’s looking for a hot or a cold bid based on the extracted timestamp of the request UUID. In this case, the cold bid will be retrieved by running an Amazon Athena look-up query leveraging the use of partitions and Parquet format to reduce as much as possible the latency and data that needs to be scanned.

Conclusion

Thanks to this combined approach using different technologies and a variety of AWS services we were able to extend our attribution window from 30 to 90 days while reducing the infrastructure costs by 45%.

 

 

Enhancing automated database continuous integration with AWS CodeBuild and Amazon RDS Database Snapshot

Post Syndicated from bobyeh original https://aws.amazon.com/blogs/devops/enhancing-automated-database-continuous-integration-with-aws-codebuild-and-amazon-rds-database-snapshot/

In major integration merges, it’s sometimes necessary to verify the changes with existing online data. To inspect the changes with a cloned database can give us confidence to deploy to the production database. This post demonstrates how to use AWS CodeBuild and Amazon RDS Database Snapshot to verify your code revisions in both the application layer and the underlying layer, ensuring that your existing data works seamlessly with your revised code.

Making code revisions using continuous integration requires running periodic verification to ensure that your new deliverable works functionally and reliably. It’s easy to focus attention solely on the surface level changes made to the application layer. However, it’s important to remember to inspect the changes made to the underlying data layer too.

From the application layer, users modify the data model for different reasons. Any data model definition change in the application layer maps to a schema change in the database. For those services backed with a relational database (RDBMS), a user might perform data definition language (DDL) operations directly toward a database schema or rely on an object-relational mapping (ORM) library to migrate the schema to fit the application revision. These schema changes (CREATE, DROP, ALTER, TRUNCATE, etc.) can be very critical, especially for those services serving real customers.

Performing proper verification and simulation for these changes mitigates the risk of bringing down services. After the changes are applied, fundamental operation testing (CRUD – CREATE, READ, UPDATE, DELETE) toward data models is mandatory; this leads to data control language (DCL) operations (INSERT, SELECT, UPDATE, DELETE, etc.). After all the necessary steps, a user can move on to the deployment stage.

About this page

  • Time to read:6 minutes
  • Time to complete:30 minutes
  • Cost to complete (estimated):Less than $1 for 1-GB database snapshot and restored instance
  • Learning level:Advanced (300)
  • Services used:AWS CodeBuild, IAM, RDS

Solution overview

This example uses a buildspec file in CodeBuild. Set up a build project that points to a source control repository containing that buildspec file. The CodeBuild runtime environment restores the database server from an RDS snapshot.We restore snapshot to an Amazon Aurora cluster as example through AWS Command Line Interface (AWS CLI). After the database is restored, the build process starts to run your integration process, which is in mock code in the buildspec definition. After the verification stage, CodeBuild drops the restored database.

 

Architecture diagram showing an overview of how we use CodeBuild to restore a database snapshot to verify and validate the new database schema change.

Prerequisites

The following components are required to implement this example:

Walkthrough

Follow these steps to execute the solution.

Prepare your build specification file

Before you begin, prepare your CodeBuild Build Specification file with following information:

  • db-cluster-identifier-prefix
  • db-snapshot-identifier
  • region-ID
  • account-ID
  • vpc-security-group-id

The db-cluster-identifier-prefix creates a temporary database followed by a timestamp. Make sure that this value does not overlap with any other databases. The db-snapshot-identifier points to the snapshot you are calling to run with your application. Region-ID and account-ID describe the account on which you are running. The vpc-security-group-id indicates the security group you use in the CodeBuild environment and temporary database.

YAML
Version: 0.2
phases:
  install:
    runtime-versions:
      python: 3.7
pre_build:
  commands:
    - pip3 install awscli --upgrade --user
    - export DATE=`date +%Y%m%d%H%M`
    - export DBIDENTIFIER=db-cluster-identifier-prefix-$DATE
    - echo $DBIDENTIFIER
    - aws rds restore-db-cluster-from-snapshot --snapshot-identifier arn:aws:rds:region-ID:account-ID:cluster-snapshot:db-snapshot-identifier –vpc-security-group-ids vpc-security-group-id --db-cluster-identifier $DBIDENTIFIER --engine aurora
    - while [ $(aws rds describe-db-cluster-endpoints --db-cluster-identifier $DBNAME | grep -c available) -eq 0 ]; do echo "sleep 60s"; sleep 60; done
    - echo "Temp db ready"
    - export ENDPOINT=$(aws rds describe-db-cluster-endpoints --db-cluster-identifier $DBIDENTIFIER| grep "\"Endpoint\"" | grep -v "\-ro\-" | awk -F '\"' '{print $4}')
    - echo $ENDPOINT
build:
  commands:
    - echo Build started on `date`
    - echo proceed db connection to $ENDPOINT
    - echo proceed db migrate update, DDL proceed here
    - echo proceed application test, CRUD test run here
post_build:
  commands:
    - echo Build completed on `date`
    - echo $DBNAME
    - aws rds delete-db-cluster --db-cluster-identifier $DBIDENTIFIER --skip-final-snapshot &

 

After you finish editing the file, name it buildspec.yml. Save it in the root directory with which you plan to build, then commit the file into your code repository.

  1. Open the CodeBuild console.
  2. Choose Create build project.
  3. In Project Configuration, enter the name and description for the build project.
  4. In Source, select the source provider for your code repository.
  5. In Environment image, choose Managed image, Ubuntu, and the latest runtime version.
  6. Choose the appropriate service role for your project.
  7. In the Additional configuration menu, select the VPC with your Amazon RDS database snapshots, as shown in the following screenshot, and then select Validate VPC Settings. For more information, see Use CodeBuild with Amazon Virtual Private Cloud.
  8. In Security Groups, select the security group needed for the CodeBuild environment to access your temporary database.
  9. In Build Specifications, select Use a buildspec file.

CodeBuild Project Additional Configuration - VPC

Grant permission for the build project

Follow these steps to grant permission.

  1. Navigate to the AWS Management Console Policies.
  2. Choose Create a policy and select the JSON tab.To give CodeBuild access to the Amazon RDS resource in the pre_build stage, you must grant RestoreDBClusterFromSnapshot and DeleteDBCluster. Follow the least privilege guideline and limit the DeleteDBCluster action point to “arn:aws:rds:*:*:cluster: db-cluster-identifier-*”.
  3. Copy the following code and paste it into your policy:
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "VisualEditor0",
          "Effect": "Allow",
          "Action": "rds:RestoreDBClusterFromSnapshot",
          "Resource": "*"
        },
        {
          "Sid": "VisualEditor1",
          "Effect": "Allow",
          "Action": "rds:DeleteDBCluster*",
          "Resource": "arn:aws:rds:*:*:cluster:db-cluster-identifier-*"
        }
      ]
    }
  4. Choose Review Policy.
  5. Enter a Name and Description for this policy, then choose Create Policy.
  6. After the policy is ready, attach it to your CodeBuild service role, as shown in the following screenshot.

Attach created policy to IAM role

Use database snapshot restore to launch the build process

  1. Navigate back to CodeBuild and locate the project you just created.
  2. Give an appropriate timeout setting and make sure that you set it to the correct branch for your repository.
  3. Choose Start Build.
  4. Open the Build Log to view the database cluster from your snapshot in the pre_build stage, as shown in the following screenshot.CodeBuild ProjectBuild Log - pre_build stage
  5. In the build stage, use $ENDPOINT to point your application to this temporary database, as shown in the following screenshot.CodeBuild Project Build Log - build stage
  6. In the post_build, delete the cluster, as shown in the following screenshot.CodeBuild Project Build log - post build stage

Test your database schema change

After you set up this pipeline, you can begin to test your database schema change within your application code. This example defines several steps in the Build Specifications file to migrate the schema and run with the latest application code. In this example, you can verify that all the modifications fit from the application to the database.

YAML
build:
  commands:
    - echo Build started on `date`
    - echo proceed db connection to $ENDPOINT
    # run a script to apply your latest schema change
    - echo proceed db migrate update
    # start the latest code, and run your own testing
    - echo proceed application test

After validation

After we validated the database schema change in the above steps, a suitable strategy for deployment to production should be utilized that would align with the criteria to satisfy the business goals.

Cleaning up

To avoid incurring future charges, delete the resources as following steps:

  1. Open the CodeBuild console
  2. Click the project you created for this test.
  3. Click the delete build project and input delete to confirm deletion.

Conclusion

In this post, you created a mechanism to set up a temporary database and limit access into the build runtime. The temporary database stands alone and isolated. This mechanism can be applied to secure the permission control for the database snapshot, or not to break any existing environment. The database engine applies to all available RDS options, including Amazon Aurora, PostgreSQL, MySQL, MariaDB, Oracle Database, and SQL Server. This provides options, without impacting any existing environments, for critical events triggered by major changes in the production database schema, or data format changes required by business decisions.

 

Cost Optimize your Jenkins CI/CD pipelines using EC2 Spot Instances

Post Syndicated from Ben Peven original https://aws.amazon.com/blogs/compute/cost-optimize-your-jenkins-ci-cd-pipelines-using-ec2-spot-instances/

Author: Rajesh Kesaraju, Sr. Specialist Solution Architect, EC2 Spot Instances

In this blog post, I go over using Amazon EC2 Spot Instances on continuous integration and continuous deployment (CI/CD) workloads, via the popular open-source automation server Jenkins. I also break down the steps required to adopt Spot Instances into your CI/CD pipelines for cost optimization purposes. In this blog, I explain how to configure your Jenkins environment to achieve significant cost savings by using Spot Instances with the EC2 Fleet Jenkins plugin.

Overview of EC2 Spot, CI/CD, and Jenkins

AWS offers multiple purchasing models for its EC2 instances. This particular blog post focuses on Amazon EC2 Spot Instances, which lets you take advantage of unused EC2 capacity in the AWS Cloud at a steep discount.

You can use Spot Instances for various stateless, fault-tolerant, or flexible applications such as big data, containerized workloads, CI/CD, web servers, high performance computing (HPC), and other test and development workloads.

CI/CD pipelines are familiar to many readers via a popular piece of open-source software called Jenkins. Jenkins’ automation of development, testing and deployment scenarios, courtesy of more than 2000 plugins, plays a key role in many organizations’ software development and delivery ecosystems. Jenkins accelerates software development through multiple stages, including building and documenting, packaging and analytics, staging and deploying, etc.

Lyft began using EC2 Spot Instances for their Jenkins CI pipelines, and discovered they could save up to 90 percent compared to their previous non-Spot EC2 implementations. They moved their entire CI/CD pipeline to EC2 Spot Instances by modifying just four lines of their deployment code.

In this blog, I walk through how to configure your Jenkins environment to achieve significant cost savings by using Spot Instances with the EC2 Fleet Jenkins plugin.

Solution Overview

For the following tutorial, you need both an AWS account and Jenkins downloaded and installed on your system.

This blog post uses Spot Instances. If your Jenkins server runs on On-Demand Instances, you can easily switch to Spot Instances with EC2-Fleet Plugin. Now, let’s look at how this plugin can be configured to make your Jenkins elastically scale up/down depending on pending jobs, and save significantly on compute costs.

Solution

Create a new EC2 key pair

To access the SSH interfaces of your Jenkins instances, you must have an EC2 key pair. Please follow below steps to create a new EC2 key pair.

1. Log in to your AWS Account;
2. Switch to your preferred Region;
3. Provision a new EC2 key pair:

    1. Go to the EC2 console and click on the key pairs option from the left frame.
    2. Click on the Create key pair button;
    3. Provide key pair name and click on the Create button;
    4. Your web browser should download a .pem file – keep this file as it will be required to access the EC2 instances that you create in this workshop. If you’re using a Windows system, convert the .pem file to a PuTTY .ppk file. If you’re not sure how to do this, instructions are available here.

Create an AWS IAM User for EC2-Fleet Plugin

To control Spot Instances from the EC2-Fleet plugin, you first create an IAM user (with programmatic access) in your AWS account. Then configure an IAM policy for AWS permissions. Use the following code to achieve this step.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:*",
                "autoscaling:*",
                "iam:ListInstanceProfiles",
                "iam:ListRoles",
                "iam:PassRole"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

These permissions allow you to configure the plugin, allow programmatic access to AWS resources to create and terminate Spot Instances, and control Auto Scaling Group (ASG) parameters.

In the IAM dashboard, click User and select the Jenkins User you created. Next, click Create access key, and save the Access key ID and Secret access key for use in next steps.

IAM > Users > Jenkins User > Security Credentials > Create Access Key

Create Access Key for Jenkins user fig. 1

Create Access Key for Jenkins user fig. 2

Create an Auto Scaling Group

Auto Scaling groups help you configure your Jenkins EC2-Fleet plugin to control Jenkins build agents and scale up or down depending on the job queue. It also replaces instances that were terminated due to demand spike in specific Spot Instance pools.

Documentation on how to create Auto Scaling Group is here.

Set your ASG to diversify your Spot Fleet across multiple Spot pools to increase your chances of getting a Spot Instance for your Jenkins jobs, and set an allocation strategy. I used “capacity-optimized” as the allocation strategy in the following example.

The “capacity-optimized” option allocates Spot Instances from the deepest pools of available spare capacity, which lowers the chance of interruptions. Alternatively, you may choose the “lowest-price” allocation strategy if you have builds that finish quicker, and the cost of re-processing of failed jobs due to interruption isn’t that significant. Learn more about allocation strategies in this blog.

The Jenkins EC2-Fleet plugin overrides and controls ASG’s capacity configuration. So, start with one instance for now. I also cover a scenario that starts with 0 instances to further minimize costs.

A sample ASG configuration looks like the following image. Notice there are 6 instance types across 3 Availability Zones, this means EC2 Spot capacity is provided from 18 (6×3) Spot Instance pools! This configuration increases the likelihood of getting Spot Instances from the deepest Spot pools at a steep discount.

An example ASG configuration Image

Install and Configure an EC2-Fleet plugin in Jenkins

Install the latest version of EC2-Fleet plugin in Jenkins. This plugin launches Spot Instances using ASG or EC2 Spot Fleet where you can run your build jobs. In this blog, I launch EC2 Spot Instances using ASG.

After installing you see it in the plugin manager. This blog uses current version 2.0.0.

Go to Manage Jenkins > Plugin Manager then install EC2 Fleet Jenkins Plugin

Jenkins Server EC2-Fleet Plugin

In the first part of this solution, you created an AWS user. Now, configure this user in your Jenkins Amazon EC2 Fleet configuration section.

Navigate to Manage Jenkins -> Configure Clouds -> Add a New Cloud -> Amazon EC2 Fleet.

Configure Amazon EC2 Fleet plugin as Cloud Setting

Create a name for your EC2-Fleet plugin configuration. I use Amazon EC2 Spot Fleet. Then configure your AWS Credentials.

Configure ASG in Jenkins EC2-fleet plugin

1. Change the Kind to AWS Credentials;
2. Change the Scope to System (Jenkins and nodes only) – you don’t want your builds to have access to these credentials!
3. At the ID (optional) field. Enter this if need to access this using scripts
4. Provide Access key ID and Secret access key fields, saved when you created Jenkins user before, then click Add.
5. Once you are done adding credentials, select the corresponding AWS Region to your ASG.
6. EC2 Fleet dropdown automatically populates the ASG that you created earlier.
7. Once the ASG is selected, check your configuration. The following image shows this test:

Testing Jenkins EC2 Fleet Plugin configuration

Configure Launcher

1. Change the Kind to SSH Username with private key;
2. Change the Scope to System (Jenkins and nodes only) – you also don’t want your builds to have access to these credentials;
3. Enter ec2-user as the Username.
4. Select the Enter directly button for the Private Key. Open the .pem file that you downloaded previously, and copy the contents of the file to the Key field including the BEGIN RSA PRIVATE KEY and END RSA PRIVATE KEY fields.
5. Verify your launcher looks like as below, and click on the Add button

Configuring Launcher and providing Jenkins credentials

Once your credentials are added, you move on to complete rest of the Launcher configuration.

1. Select the ec2-user option from the Credentials drop-down.
2. Select the “Non verifying Verification Strategy” option from the Host Key Verification Strategy drop-down. Select this option because Spot Instances have a random SSH host fingerprint.

Configuring Jenkins launch agents by ec2-user via SSH

3. Mark the Connect Private check box to ensure that your Jenkins Master always communicates with the Agents via their internal VPC IP addresses (in real-world scenarios, your build      agents would likely not be publicly addressable).
4. Change the Label field to spot-agents.
5. Set the Max Idle Minutes Before Scaledown. In this example, I used AWS launched per-second billing in 2017, so there’s no need to keep a build agent running for too much longer than it’s required.
6. Change the Maximum Cluster Size depending on your need. For example, I set the Maximum Cluster Size to 2.

After saving these configurations, your screen should look similar to the following image.

Configuring Launcher with cluster scaling parameters

Configure Number of Executors

Determine the number of executors based on your build requirements, such as how many builds on average can be executed concurrently on each machine based on machine’s vCPU and RAM allocations.

If you cram many executors into one machine, each build average execution may increase, which slows down the pipeline.

Once you determine optimum executors per machine, any additional pending jobs get executed on scaled out machines by auto scaling.

Some Important aspects about Cluster size settings

Jenkins EC2-Fleet agent settings override ASG settings. So, “Minimum Cluster Size” and “Maximum Cluster Size” values mentioned here override ASG’s settings dynamically.

If you set minimum cluster size as 0, then when there are no pending jobs there won’t be any idle servers after Max Idle Minutes before shut down minutes are met. In this scenario, when there is a new build request, it takes roughly two to five minutes for new EC2 Spot Instances to start processing after boot strapping and installing necessary Jenkins agents.

If your jobs are time-insensitive, this strategy maximizes savings, as you eliminate spending money on idle instances.

Alternatively, you may set “Minimum Cluster Size” as 1 or 2 so you have running instances all the time if you have a need to process several builds/tests a day and/or builds taking very long time and occur on a daily basis.

In this blog the idea is to cost optimize CI/CD pipelines, to avoid idle instances.

Configure Jenkins build Jobs to utilize EC2 Spot Instances

Finally, configure your build job to check “Restrict where this project can be run” and enter “spot-agents” as the label expression. When builds are initiated, they are executed against Spot Instance.

Configuring Jenkins builds to utilize EC2 Spot Instances with Label Expression

At this point, you are ready to run your builds on Spot Instances and save significantly!

Configuring Jenkins server to run on EC2 Spot Instances

Now you started saving on your build agents, is it possible to save on Jenkins server also using Spot Instances? Yes! Let’s see how this can be done with a few simple techniques.

By running your Jenkins server on Spot, you optimize the compute costs associated with the whole Jenkins CI/CD environment. With Spot Instance diversification strategy and making use of ASG features, you can move your Jenkins server also to EC2 Spot Instance.

There is one slight wrinkle in the above process. Jenkins requires persistent data on a local file system, whereas a Spot Instance cannot be guaranteed to be persistent due to the chance of interruptions. Therefore, to switch your Jenkins server over to Spot, you must first move your Jenkins data to an Amazon Elastic File System (EFS) volume — which your Spot Instance can then access.

Amazon Elastic File System provides a simple, scalable, fully managed elastic NFS file system for use with AWS Cloud services and on-premises resources. More here

Here are the steps to mount EFS volume to an existing Jenkins server, and move its content to Amazon EFS managed store. Then, you can point the Jenkins server to use EFS mount point for its operations and maintain server state. This way when a Spot Instance gets interrupted, server state is not lost, and another Spot Instance can pick up from where the previous server left off.

Here are the steps move data from JENKINS_HOME to Amazon EFS

1. Mount EFS volume:

sudo mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 \
$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone)\
.%FILE-SYSTEM-ID%.efs.<AWS Region>.amazonaws.com:/ (http://efs.<AWS Region>.amazonaws.com/) /mnt

2. Copy existing JENKINS_HOME (/var/lib/Jenkins) content to EFS after shutting down Jenkins server
3. >> sudo chown jenkins:jenkins /mnt
>> sudo cp -rpv /var/lib/jenkins/* /mnt

Once you’ve moved the contents of JENKINS_HOME to Amazon EFS, now it’s safe to run Jenkins Server to EC2 Spot Instance.

Spot Instances can be interrupted by AWS, so you may lose your Jenkins server access momentarily when an interruption occurs.

Since you already externalized the state to Amazon EFS, you don’t lose any previous state. So, when the instance running your Jenkins server gets interrupted, in a matter of a few minutes, ASG replenishes new EC2 Spot Instance to run your Jenkins Server.

To get this into service automatically complete the following steps:

1. Ensure ASG launch instances into target group that is pointed by Application Load Balancer (ALB)
2. If Spot Instance is terminated with the two minute warning, ASG launches a replacement instance. The new Spot Instance will be bootstrapped just as your original Spot Instance was and then mount to EFS as configured in user data of “Launch Template”
3. Configure ASG from Launch Template (Sample UserData section as below)

#!/bin/bash
# Install all pending updates to the system
yum -y update
# Configure YUM to be able to access official Jenkins RPM packages
wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
# Import the Jenkins repository public key
rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
# Configure YUM to be able to access contributed Maven RPM packages
wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
# Update the release version in the Maven repository configuration for this mainline release of Amazon Linux
sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
# Install the Java 8 SDK, Git, Jenkins and Maven
yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel git jenkins apache-maven
# Set the default version of java to run out of the Java 8 SDK path (required by Jenkins)
update-alternatives --set java /usr/lib/jvm/jre-1.8.0-openjdk.x86_64/bin/java
update-alternatives --set javac /usr/lib/jvm/java-1.8.0-openjdk.x86_64/bin/javac
# Mount the Jenkins EFS volume at JENKINS_HOME
mount -t nfs -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 $(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone).${EFSJenkinsHomeVolume}.efs.${AWSRegion}.amazonaws.com:/ /var/lib/jenkins
# Start the Jenkins service
service jenkins start

That’s it, you are now ready to run your Jenkins server on EC2 Spot Instances and save up to 90% compared to On-Demand price.

Conclusion

You are now ready to leverage the power and flexibility of EC2 Spot Instances.

With a few modifications to your deployment you can significantly reduce your compute costs or accelerate throughput by accessing 10x compute for the same cost.

For users getting started on their Amazon EC2 Spot Instances, we are here to help. Please also share any questions in the comments section below.

Here is where to begin in the Amazon EC2 Spot Instance console — and start transforming your Jenkins workloads today.


About the author

Rajesh Kesaraju

Rajesh Kesaraju is a Sr. Specialist SA for EC2 Spot with Amazon AWS. He helps customers to cost optimize their workloads by utilizing EC2 Spot instances in various types of workloads such as Big Data, Containers, HPC, CI/CD, and Sateless Applications Etc.

AWS Online Tech Talks for April 2020

Post Syndicated from Jimmy Cooper original https://aws.amazon.com/blogs/aws/aws-online-tech-talks-for-april-2020/

Join us for live, online presentations led by AWS solutions architects and engineers. AWS Online Tech Talks cover a range of topics and expertise levels, and feature technical deep dives, demonstrations, customer examples, and live Q&A with AWS experts.

Note – All sessions are free and in Pacific Time. Can’t join us live? Access webinar recordings and slides on our On-Demand Portal.

Tech talks this month are:

April 20, 2020 | 9:00 AM – 10:00 AM PT – Save Costs Running Kubernetes Clusters with EC2 Spot Instances – ​Learn how you can lower costs and improve application resiliency by running Kubernetes workloads on Amazon EKS with Spot Instances.​

April 20, 2020 | 11:00 AM – 12:00 PM PT – Hadoop 3.0 and Docker on Amazon EMR 6.0 – A deep dive into what’s new in EMR 6.0 including Apache Hadoop 3.0, Docker containers & Apache Hive performance improvements​.

​April 20, 2020 | 1:00 PM – 2:00 PM PT – Infrastructure as Code on AWS – ​Join this tech talk to learn how to use AWS CloudFormation and AWS CDK to provision and manage infrastructure, deploy code, and automate your software-release processes.

April 21, 2020 | 9:00 AM – 10:00 AM PT – How to Maximize Results with a Cloud Contact Center, Featuring Aberdeen Research – ​Learn how to maximize results with a cloud contact center, featuring Aberdeen Research and Amazon Connect​.

April 21, 2020 | 11:00 AM – 12:00 PM PT – Connecting Microcontrollers to the Cloud for IoT Applications – ​Learn how you can connect microcontrollers to the cloud for IoT applications​.

April 21, 2020 | 1:00 PM – 2:00 PM PT – Reducing Machine Learning Inference Cost for PyTorch Models – ​Join us for a tech talk to learn about deploying your PyTorch models for low latency at low cost.​

April 22, 2020 | 11:00 AM – 12:00 PM PT – Top 10 Security Items to Improve in Your AWS Account – Learn about the top 10 security items to improve in your AWS environment and how you can automate them.​

April 22, 2020 | 1:00 PM – 2:00 PM PT – Building Your First Application with AWS Lambda – ​Learn how to build your first serverless application with AWS Lambda, including basic design patterns and best practices.​

April 23, 2020 | 9:00 AM – 10:00 AM PT – Persistent Storage for Containers with Amazon EFS – ​Learn how to securely store your containers in the cloud with Amazon EFS​.

April 23, 2020 | 11:00 AM – 12:00 PM PT – Build Event Driven Graph Applications with AWS Purpose-Built Databases – ​Learn how to build event driven graph applications using AWS purpose-built database services including Amazon Neptune, Amazon DynamoDB, and Amazon ElastiCache.​

April 23, 2020 | 1:00 PM – 2:00 PM PT – Migrate with AWS – ​Introduction to best practice driven process for migrations to AWS, developed by the experience in helping thousands of enterprises migrate.

April 27, 2020 | 9:00 AM – 10:00 AM PT – Best Practices for Modernizing On-Premise Big Data Workloads Using Amazon EMR – ​Learn about best practices to migrate from on-premises big data (Apache Spark and Hadoop) to Amazon EMR.​

April 27, 2020 | 11:00 AM – 12:00 PM PT – Understanding Game Changes and Player Behavior with Graph Databases – ​Learn how to solve problems with highly connected data in game datasets with Amazon Neptune.

​​April 27, 2020 | 1:00 PM – 2:00 PM PT – Assess, Migrate, and Modernize from Legacy Databases to AWS: Oracle to Amazon Aurora PostgreSQL Migration – ​Punitive licensing and high cost of on-premises legacy databases could hold you back. Join this tech talk to learn how to assess, migrate, and modernize your Oracle workloads over to Amazon Aurora PostgreSQL, using Amazon Database Migration Service (DMS).​

April 28, 2020 | 9:00 AM – 10:00 AM PT – Implementing SAP in the Cloud with AWS Tools and Services – ​This tech talk will help architects and administrators to understand the automation capabilities available that can assist your SAP migration.​

April 28, 2020 | 11:00 AM – 12:00 PM PT – Choosing Events, Queues, Topics, and Streams in Your Serverless Application – ​Learn how to choose between common Lambda event sources like EventBridge, SNS, SQS, and Kinesis Data Streams.​

April 30, 2020 | 9:00 AM – 10:00 AM PT – Inside Amazon DocumentDB: The Makings of a Managed Non-relational Database – Join Rahul Pathak, GM of Emerging Databases and Blockchain at AWS, to learn about the inner workings of Amazon DocumentDB and how it provides better performance, scalability, and availability while reducing operational overhead for managing your own non-relational databases.

Round 2 Hybrid Post-Quantum TLS Benchmarks

Post Syndicated from Alex Weibel original https://aws.amazon.com/blogs/security/round-2-hybrid-post-quantum-tls-benchmarks/

AWS Cryptography has completed benchmarks of Round 2 Versions of the Bit Flipping Key Encapsulation (BIKE) and Supersingular Isogeny Key Encapsulation (SIKE) hybrid post-quantum Transport Layer Security (TLS) Algorithms. Both of these algorithms have been submitted to the National Institute of Standards and Technology (NIST) as part of NIST’s Post-Quantum Cryptography standardization process.

In the first hybrid post-quantum TLS blog, we announced that AWS Key Management Service (KMS) had launched support for hybrid post-quantum TLS 1.2 using Round 1 versions of BIKE and SIKE. In this blog, we are announcing AWS Cryptography’s benchmark results of using Round 2 versions of BIKE and SIKE with hybrid post-quantum TLS 1.2 against an HTTP webservice. Round 2 versions of BIKE and SIKE include performance improvements, parameter tuning, and algorithm updates in response to NIST’s comments on Round 1 versions. I’ll give a refresher on hybrid post-quantum TLS 1.2, go over our Round 2 hybrid post-quantum TLS 1.2 benchmark results, and then describe our benchmarking methodology.

This blog post is intended to inform software developers, AWS customers, and cryptographic researchers about the potential upcoming performance differences between classical and hybrid post-quantum TLS.

Refresher on Hybrid Post-Quantum TLS 1.2

Some of this section is repeated from the previous hybrid post-quantum TLS 1.2 launch announcement for KMS. If you are already familiar with hybrid post-quantum TLS, feel free to skip to the Benchmark Results section.

What is Hybrid Post-Quantum TLS 1.2?

Hybrid post-quantum TLS 1.2 is a proposed extension to the TLS 1.2 Protocol implemented by Amazon’s open source TLS library s2n that provides the security protections of both the classical and post-quantum schemes. It does this by performing two independent key exchanges (one classical and one post-quantum), and then cryptographically combining both keys into a single TLS master secret.

Why is Post-Quantum TLS Important?

Hybrid post-quantum TLS allows connections to remain secure even if one of the key exchanges (either classical or post-quantum) performed during the TLS Handshake is compromised in the future. For example, if a sufficiently large-scale quantum computer were to be built, it could break the current classical public-key cryptography that is used for key exchange in every TLS connection today. Encrypted TLS traffic recorded today could be decrypted in the future with a large-scale quantum computer if post-quantum TLS is not used to protect it.

Round 2 Hybrid Post-Quantum TLS Benchmark Results

Figure 2: Latency in relation to HTTP request count for four key exchange algorithms

Figure 2: Latency in relation to HTTP request count for four key exchange algorithms

Key Exchange AlgorithmServer PQ ImplementationTLS Handshake
+ 1 HTTP Request
TLS Handshake
+ 2 HTTP Requests
TLS Handshake
+ 10 HTTP Requests
TLS Handshake
+ 25 HTTP Requests
ECDHE OnlyN/A10.8 ms15.1 ms52.6 ms124.2 ms
ECDHE + BIKE1‑CCA‑L1‑R2C19.9 ms24.4 ms61.4 ms133.2 ms
ECDHE + SIKE‑P434‑R2C169.6 ms180.3 ms219.1 ms288.1 ms
ECDHE + SIKE‑P434‑R2x86-64
Assembly
20.1 ms24.5 ms62.0 ms133.3 ms

Table 1 shows the time (in milliseconds) that a client and server in the same region take to complete a TCP Handshake, a TLS Handshake, and complete varying numbers of HTTP Requests sent to an HTTP web service running on an i3en.12xlarge host.

Key Exchange AlgorithmClient HelloServer Key ExchangeClient Key ExchangeOtherTLS Handshake Total
ECDHE Only2183387524303061
ECDHE + BIKE1‑CCA‑L1‑R22203288302324308961
ECDHE + SIKE‑P434‑R221467242324303739

Table 2 shows the amount of data (in bytes) used by different messages in the TLS Handshake for each Key Exchange algorithm.

1 HTTP
Request
2 HTTP Requests10 HTTP
Requests
25 HTTP Requests
HTTP Request Bytes8781,7618,82522,070
HTTP Response Bytes6981,3776,80916,994
Total HTTP Bytes15763,13815,63439,064

Table 3 shows the amount of data (in bytes) sent and received through each TLS connection for varying numbers of HTTP requests.

Benchmark Results Analysis

In general, we find that the major trade off between BIKE and SIKE is data usage versus processing time, with BIKE needing to send more bytes but requiring less time processing them, and SIKE making the opposite trade off of needing to send fewer bytes but requiring more time processing them. At the time of integration for our benchmarks, an x86-64 assembly optimized implementation of BIKE1-CCA-L1-R2 was not available in s2n.

Our results show that when only a single HTTP request is sent, completing a BIKE1-CCA-L1-R2 hybrid TLS 1.2 handshake takes approximately 84% more time compared to a non-hybrid TLS connection, and completing an x86-64 assembly optimized SIKE-P434-R2 hybrid TLS 1.2 handshake takes approximately 86% more time than non-hybrid. However, at 25 HTTP Requests per TLS connection, when using the fastest available implementation for both BIKE and SIKE, the increased TLS Handshake latency is amortized, and only 7% more total time is needed for both BIKE and SIKE compared to a classical TLS connection.

Our results also show that BIKE1-CCA-L1-R2 hybrid TLS Handshakes used 5900 more bytes than a classical TLS Handshake, while SIKE-P434-R2 hybrid TLS Handshakes used 678 more bytes than classical TLS.

In the AWS EC2 network, using modern x86-64 CPU’s with the fastest available algorithm implementations, we found that BIKE and SIKE performed similarly, with their maximum latency difference being only 0.6 milliseconds apart, and BIKE being the faster of the two in every benchmark. However when compared to SIKE’s C implementation, which would be used on hosts without the ADX and MULX x86-64 instructions used by SIKE’s assembly implementation, BIKE performed significantly better, seeing a maximum improvement of 157 milliseconds over SIKE.

Hybrid Post-Quantum TLS Benchmark Details and Methodology

Hybrid Post-Quantum TLS Client

Figure 3: Architecture diagram of the AWS SDK Java Client using Java Native Interface (JNI) to communicate with the native AWS Common Runtime (CRT)

Figure 3: Architecture diagram of the AWS SDK Java Client using Java Native Interface (JNI) to communicate with the native AWS Common Runtime (CRT)

Our post-quantum TLS Client is using the aws-crt-dev-preview branch of the AWS SDK Java v2 Client, that has Java Native Interface Bindings to the AWS Common Runtime (AWS CRT) written in C. The AWS Common Runtime uses s2n for TLS negotiation on Linux platforms.

Our client was a single EC2 i3en.6xlarge host, using v0.5.1 of the AWS Common Runtime (AWS CRT) Java Bindings, with commit f3abfaba of s2n and used the x86-64 Assembly implementation for all SIKE-P434-R2 benchmarks.

Hybrid Post-Quantum TLS Server

Our server was a single EC2 i3en.12xlarge host running a REST-ful HTTP web service which used s2n to terminate TLS connections. In order to measure the latency of the SIKE-P434-R2 C implementation on these hosts, we used an s2n compile time flag to build a 2nd version of s2n with SIKE’s x86-64 assembly optimization disabled, and reran our benchmarks with that version.

We chose i3en.12xlarge as our host type because it is optimized for high IO usage, provides high levels of network bandwidth, has a high number of vCPU’s that is typical for many web service endpoints, and has a modern x86-64 CPU with the ADX and MULX instructions necessary to use the high performance Round 2 SIKE x86-64 assembly implementation. Additional TLS Handshake benchmarks performed on other modern types of EC2 hosts, such as the C5 family and M5 family of EC2 instances, also showed similar latency results to those generated on i3en family of EC2 instances.

Post-Quantum Algorithm Implementation Details

The implementations of the post-quantum algorithms used in these benchmarks can be found in the pq-crypto directory of the s2n GitHub Repository. Our Round 2 BIKE implementation uses portable optimized C code, and our Round 2 SIKE implementation uses an optimized implementation in x86-64 assembly when available, and falls back to a portable optimized C implementation otherwise.

Key Exchange Algorithms2n Client Cipher Preferences2n Server Cipher PreferenceNegotiated Cipher
ECDHE OnlyELBSecurityPolicy-TLS-1-1-2017-01KMS-PQ-TLS-1-0-2020-02ECDHE-RSA-AES256-GCM-SHA384
ECDHE + BIKE1‑CCA‑L1‑R2KMS-PQ-TLS-1-0-2020-02KMS-PQ-TLS-1-0-2020-02ECDHE-BIKE-RSA-AES256-GCM-SHA384
ECDHE + SIKE‑P434‑R2PQ-SIKE-TEST-TLS-1-0-2020-02KMS-PQ-TLS-1-0-2020-02ECDHE-SIKE-RSA-AES256-GCM-SHA384

Table 4 shows the Clients and Servers TLS Cipher Config name used in order to negotiate each Key Exchange Algorithm.

Hybrid Post-Quantum TLS Benchmark Methodology

Figure 4: Benchmarking Methodology Client/Server Architecture Diagram

Figure 4: Benchmarking Methodology Client/Server Architecture Diagram

Our Benchmarks were run with a single client host connecting to a single host running a HTTP web service in a different availability zone within the same AWS Region (us-east-1), through a TCP Load Balancer.

We chose to include varying numbers of HTTP requests in our latency benchmarks, rather than TLS Handshakes alone, because customers are unlikely to establish a secure TLS connection and let the connection sit idle performing no work. Customers use TLS connections in order to send and receive data securely, and HTTP web services are one of the most common types of data being secured by TLS. We also chose to place our EC2 server behind a TCP Load Balancer to more closely approximate how an HTTP web service would be deployed in a typical setup.

Latency was measured at the client in Java starting from before a TCP connection was established, until after the final HTTP Response was received, and includes all network transfer time. All connections used RSA Certificate Authentication with a 2048-bit key, and ECDHE Key Exchange used the secp256r1 curve. All latency values listed in Tables 1 above were calculated from the median value (50th percentile) from 60 minutes of continuous single-threaded measurements between the EC2 Client and Server.

More Info

If you’re interested to learn more about post-quantum cryptography check out the following links:

Conclusion

In this blog post, I gave a refresher on hybrid post-quantum TLS, I went over our hybrid post-quantum TLS 1.2 benchmark results, and went over our hybrid post-quantum benchmarking methodology. Our benchmark results found that BIKE and SIKE performed similarly when using s2n’s fastest available implementation on modern CPU’s, but that BIKE performed better than SIKE when both were using their generic C implementation.

If you have feedback about this blog post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Alex Weibel

Alex is a Senior Software Development Engineer on the AWS Crypto Algorithms team. He’s one of the maintainers for Amazon’s TLS Library s2n. Previously, Alex worked on TLS termination and request proxying for S3 and the Elastic Load Balancing Service developing new features for customers. Alex holds a Bachelor of Science degree in Computer Science from the University of Texas at Austin.

Translating documents at enterprise scale with serverless

Post Syndicated from James Beswick original https://aws.amazon.com/blogs/compute/translating-documents-at-enterprise-scale-with-serverless/

For organizations operating in multiple countries, helping customers in different languages is an everyday reality. But in many IT systems, data remains static in a single language, making it difficult or impossible for international customers to use. In this blog post, I show how you can automate language translation at scale to solve a number of common enterprise problems.

Many types of data are good targets for translation. For example, product catalog information, for sharing with a geographically broad customer base. Or customer emails and interactions in multiple languages, translating back to a single language for analytics. Or even resource files for mobile applications, to automate string processing into different languages during the build process.

Building the machine learning models for language translation is extraordinarily complex, so fortunately you have a pay-as-you-go service available in Amazon Translate. This can accurately translate text between 54 languages, and automatically detects the source language.

Developing a scalable translation solution for thousands of documents can be challenging using traditional, server-based architecture. Using a serverless approach, this becomes much easier since you can use storage and compute services that scale for you – Amazon S3 and AWS Lambda:

Integrating S3 with Translate via Lambda.

In this post, I show two solutions that provide an event-based architecture for automated translation. In the first, S3 invokes a Lambda function when objects are stored. It immediately reads the file and requests a translation from Amazon Translate. The second solution explores a more advanced method for scaling to large numbers of documents, queuing the requests and tracking their state. The walkthrough creates resources covered in the AWS Free Tier but you may incur costs for usage.

To set up both example applications, visit the GitHub repo and follow the instructions in the README.md file. Both applications use the AWS Serverless Application Model (SAM) to make it easy to deploy in your AWS account.

Translating in near real time

In the first application, the workflow is straightforward. The source text is sent immediately to Translate for processing, and the result is saved back into the S3 bucket. It provides near real-time translation whenever an object is saved. This uses the following architecture:

Architecture for the first example application.

  1. The source text is saved in the Batching S3 bucket.
  2. The S3 put event invokes the Batching Lambda function. Since Translate has a limit of 5,000 characters per request, it slices the contents of the input into parts small enough for processing.
  3. The resulting parts are saved in the Translation S3 bucket.
  4. The S3 put events invoke the Translation function, which scales up concurrently depending on the number of parts.
  5. Amazon Translate returns the translations back to the Lambda function, which saves the results in the Translation bucket.

The repo’s SAM template allows you to specify a list of target languages, as a space-delimited list of supported language codes. In this case, any text uploaded to S3 is translated into French, Spanish, and Italian:

Parameters:
  TargetLanguage:
    Type: String
    Default: 'fr es it'

Testing the application

  1. Deploy the first application by following the README.md in the GitHub repo. Note the application’s S3 Translation and Batching bucket names shown in the output:Output values after SAM deployment.
  2. The testdata directory contains several sample text files. Change into this directory, then upload coffee.txt to the S3 bucket, replacing your-bucket below with your Translation bucket name:
    cd ./testdata/
    aws s3 cp ./coffee.txt s3://your-bucket
  3. The application invokes the translation workflow, and within a couple of seconds you can list the output files in the translations folder:aws s3 ls s3://your-bucket/translations/

    Translations output.

  4. Create an output directory, then download the translations to your local machine to view the contents:
    mkdir output
    aws s3 cp s3://your-bucket/translations/ ./output/ --recursive
    more ./output/coffee-fr.txt
    more ./output/coffee-es.txt
    more ./output/coffee-it.txt
  5. For the next step, translate several text files containing test data. Copy these to the Translation bucket, replacing your-bucket below with your bucket name:aws s3 cp ./ s3://your-bucket --include "*.txt" --exclude "*/*" --recursive
  6. After a few seconds, list the files in the translations folder to see your translated files:aws s3 ls s3://your-bucket/translations/

    Listing the translated files.

  7. Finally, translate a larger file using the batching process. Copy this file to the Batching S3 bucket (replacing your-bucket with this bucket name):
    cd ../testdata-batching/
    aws s3 cp ./your-filename.txt s3://your-bucket
  8. Since this is a larger file, the batching Lambda function breaks it apart into smaller text files in the Translation bucket. List these files in the terminal, together with their translations:
    aws s3 ls s3://your-bucket
    aws s3 ls s3://your-bucket/translations/

    Listing the output files.

In this example, you can translate a reasonable number of text files for a trivial use-case. However, in an enterprise environment where there could be thousands of files in a single bucket, you need a more robust architecture. The second application introduces a more resilient approach.

Scaling up the translation solution

In an enterprise environment, the application must handle long documents and large quantities of documents. Amazon Translate has service limits in place per account – you can request an increase via an AWS Support Center ticket if needed. However, S3 can ingest a large number of objects quickly, so the application should decouple these two services.

The next example uses Amazon SQS for decoupling, and also introduces Amazon DynamoDB for tracking the status of each translation. The new architecture looks like this:

Decoupled translation architecture.

  1. A downstream process saves text objects in the Batching S3 bucket.
  2. The Batching function breaks these files into smaller parts, saving these in the Translation S3 bucket.
  3. When an object is saved in this bucket, this invokes the Add to Queue function. This writes a message to an SQS queue, and logs the item in a DynamoDB table.
  4. The Translation function receives messages from the SQS queue, and requests translations from the Amazon Translate service.
  5. The function updates the item as completed in the DynamoDB table, and stores the output translation in the Results S3 bucket.

Testing the application

This test uses a much larger text document – the text version of the novel War and Peace, which is over 3 million characters long. It’s recommended that you use a shorter piece of text for the walkthrough, between 20-50 kilobytes, to minimize cost on your AWS bill.

  1. Deploy the second application by following the README.md in the GitHub repo, and note the application’s S3 bucket name and DynamoDB table name.
    The output values from the SAM deployment.
  2. Download your text sample and then upload it the Batching bucket. Replace your-bucket with your bucket name and your-text.txt with your text file name:aws s3 cp ./your-text.txt s3://your-bucket/ 
  3. The batching process creates smaller files in the Translation bucket. After a few seconds, list the files in the Translation bucket (replacing your-bucket with your bucket name):aws s3 ls s3://patterns-translations-v2/ --recursive --summarize

    Listing the translated files.

  4. To see the status of the translations, navigate to the DynamoDB console. Select Tables in the left-side menu and then choose the application’s DynamoDB table. Select the Items tab:Listing items in the DynamoDB table.

    This shows each translation file and a status of Queue or Translated.

  5. As translations complete, these appear in the Results bucket:aws s3 ls s3://patterns-results-v2/ --summarize

    Listing the output translations.

How this works

In the second application, the SQS queue acts as a buffer between the Batching process and the Translation process. The Translation Lambda function fetches messages from the SQS queue when they are available, and submits the source file to Amazon Translate. This throttles the overall speed of processing.

There are configuration settings you can change in the SAM template to vary the speed of throughput:

  • Translator function: this consumes messages from the SQS queue. The BatchSize configured in the SAM template is set to one message per invocation. This is equivalent to processing one source file at a time. You can set a BatchSize value from 1 to 10, so could increase this from the application’s default.
  • Function concurrency: the SAM template sets the Loader function’s concurrency to 1, using the ReservedConcurrentExecutions attribute. In effect, this means Lambda can only invoke 1 function at the same time. As a result, it keeps fetching the next batch from SQS as soon as processing finishes. The concurrency is a multiplier – as this value is increased, the translation throughput increases proportionately, if there are messages available in SQS.
  • Amazon Translate limits: the service limits in place are designed to protect you from higher-than-intended usage. If you need higher soft limits, open an AWS Support Center ticket.

Combining these settings, you have considerable control over the speed of processing. The defaults in the sample application are set at the lowest values possible so you can observe the queueing mechanism.

Conclusion

Automated translation using deep learning enables you to make documents available at scale to an international audience. For organizations operating globally, this can improve your user experience and increase customer access to your company’s products and services.

In this post, I show how you can create a serverless application to process large numbers of files stored in S3. The write operation in the S3 bucket triggers the process, and you use SQS to buffer the workload between S3 and the Amazon Translate service. This solution also uses DynamoDB to help track the state of the translated files.

Use AWS Lambda authorizers with a third-party identity provider to secure Amazon API Gateway REST APIs

Post Syndicated from Bryant Bost original https://aws.amazon.com/blogs/security/use-aws-lambda-authorizers-with-a-third-party-identity-provider-to-secure-amazon-api-gateway-rest-apis/

Note: This post focuses on Amazon API Gateway REST APIs used with OAuth 2.0 and custom AWS Lambda authorizers. API Gateway also offers HTTP APIs, which provide native OAuth 2.0 features. For more information about which is right for your organization, see Choosing Between HTTP APIs and REST APIs.

Amazon API Gateway is a fully managed AWS service that simplifies the process of creating and managing REST APIs at any scale. If you are new to API Gateway, check out Amazon API Gateway Getting Started to get familiar with core concepts and terminology. In this post, I will demonstrate how an organization using a third-party identity provider can use AWS Lambda authorizers to implement a standard token-based authorization scheme for REST APIs that are deployed using API Gateway.

In the context of this post, a third-party identity provider refers to an entity that exists outside of AWS and that creates, manages, and maintains identity information for your organization. This identity provider issues cryptographically signed tokens to users containing information about the user identity and their permissions. In order to use these non-AWS tokens to control access to resources within API Gateway, you will need to define custom authorization code using a Lambda function to “map” token characteristics to API Gateway resources and permissions.

Defining custom authorization code is not the only way to implement authorization in API Gateway and ensure resources can only be accessed by the correct users. In addition to Lambda authorizers, API Gateway offers several “native” options that use existing AWS services to control resource access and do not require any custom code. To learn more about the established practices and authorization mechanisms, see Controlling and Managing Access to a REST API in API Gateway.

Lambda authorizers are a good choice for organizations that use third-party identity providers directly (without federation) to control access to resources in API Gateway, or organizations requiring authorization logic beyond the capabilities offered by “native” authorization mechanisms.

Benefits of using third-party tokens with API Gateway

Using a Lambda authorizer with third-party tokens in API Gateway can provide the following benefits:

  • Integration of third-party identity provider with API Gateway: If your organization has already adopted a third-party identity provider, building a Lambda authorizer allows users to access API Gateway resources by using their third-party credentials without having to configure additional services, such as Amazon Cognito. This can be particularly useful if your organization is using the third-party identity provider for single sign-on (SSO).
  • Minimal impact to client applications: If your organization has an application that is already configured to sign in to a third-party identity provider and issue requests using tokens, then minimal changes will be required to use this solution with API Gateway and a Lambda authorizer. By using credentials from your existing identity provider, you can integrate API Gateway resources into your application in the same manner that non-AWS resources are integrated.
  • Flexibility of authorization logic: Lambda authorizers allow for the additional customization of authorization logic, beyond validation and inspection of tokens.

Solution overview

The following diagram shows the authentication/authorization flow for using third-party tokens in API Gateway:

Figure 1: Example Solution Architecture

Figure 1: Example Solution Architecture

  1. After a successful login, the third-party identity provider issues an access token to a client.
  2. The client issues an HTTP request to API Gateway and includes the access token in the HTTP Authorization header.
  3. The API Gateway resource forwards the token to the Lambda authorizer.
  4. The Lambda authorizer authenticates the token with the third-party identity provider.
  5. The Lambda authorizer executes the authorization logic and creates an identity management policy.
  6. API Gateway evaluates the identity management policy against the API Gateway resource that the user requested and either allows or denies the request. If allowed, API Gateway forwards the user request to the API Gateway resource.

Prerequisites

To build the architecture described in the solution overview, you will need the following:

  • An identity provider: Lambda authorizers can work with any type of identity provider and token format. The post uses a generic OAuth 2.0 identity provider and JSON Web Tokens (JWT).
  • An API Gateway REST API: You will eventually configure this REST API to rely on the Lambda authorizer for access control.
  • A means of retrieving tokens from your identity provider and calling API Gateway resources: This can be a web application, a mobile application, or any application that relies on tokens for accessing API resources.

For the REST API in this example, I use API Gateway with a mock integration. To create this API yourself, you can follow the walkthrough in Create a REST API with a Mock Integration in Amazon API Gateway.

You can use any type of client to retrieve tokens from your identity provider and issue requests to API Gateway, or you can consult the documentation for your identity provider to see if you can retrieve tokens directly and issue requests using a third-party tool such as Postman.

Before you proceed to building the Lambda authorizer, you should be able to retrieve tokens from your identity provider and issue HTTP requests to your API Gateway resource with the token included in the HTTP Authorization header. This post assumes that the identity provider issues OAuth JWT tokens, and the example below shows a raw HTTP request addressed to the mock API Gateway resource with an OAuth JWT access token in the HTTP Authorization header. This request should be sent by the client application that you are using to retrieve your tokens and issue HTTP requests to the mock API Gateway resource.


# Example HTTP Request using a Bearer token\
GET /dev/my-resource/?myParam=myValue HTTP/1.1\
Host: rz8w6b1ik2.execute-api.us-east-1.amazonaws.com\
Authorization: Bearer eyJraWQiOiJ0ekgtb1Z5eEpPSF82UDk3...}

Building a Lambda authorizer

When you configure a Lambda authorizer to serve as the authorization source for an API Gateway resource, the Lambda authorizer is invoked by API Gateway before the resource is called. Check out the Lambda Authorizer Authorization Workflow for more details on how API Gateway invokes and exchanges information with Lambda authorizers. The core functionality of the Lambda authorizer is to generate a well-formed identity management policy that dictates the allowed actions of the user, such as which APIs the user can access. The Lambda authorizer will use information in the third-party token to create the identity management policy based on “permissions mapping” documents that you define — I will discuss these permissions mapping documents in greater detail below.

After the Lambda authorizer generates an identity management policy, the policy is returned to API Gateway and API Gateway uses it to evaluate whether the user is allowed to invoke the requested API. You can optionally configure a setting in API Gateway to automatically cache the identity management policy so that subsequent API invocations with the same token do not invoke the Lambda authorizer, but instead use the identity management policy that was generated on the last invocation.

In this post, you will build your Lambda authorizer to receive an OAuth access token and validate its authenticity with the token issuer, then implement custom authorization logic to use the OAuth scopes present in the token to create an identity management policy that dictates which APIs the user is allowed to access. You will also configure API Gateway to cache the identity management policy that is returned by the Lambda authorizer. These patterns provide the following benefits:

  • Leverage third-party identity management services: Validating the token with the third party allows for consolidated management of services such as token verification, token expiration, and token revocation.
  • Cache to improve performance: Caching the token and identity management policy in API Gateway removes the need to call the Lambda authorizer for each invocation. Caching a policy can improve performance; however, this increased performance comes with addition security considerations. These considerations are discussed below.
  • Limit access with OAuth scopes: Using the scopes present in the access token, along with custom authorization logic, to generate an identity management policy and limit resource access is a familiar OAuth practice and serves as a good example of customizable authentication logic. Refer to Defining Scopes for more information on OAuth scopes and how they are typically used to control resource access.

The Lambda authorizer is invoked with the following object as the event parameter when API Gateway is configured to use a Lambda authorizer with the token event payload; refer to Input to an Amazon API Gateway Lambda Authorizer for more information on the types of payloads that are compatible with Lambda authorizers. Since you are using a token-based authorization scheme, you will use the token event payload. This payload contains the methodArn, which is the Amazon Resource Name (ARN) of the API Gateway resource that the request was addressed to. The payload also contains the authorizationToken, which is the third-party token that the user included with the request.


# Lambda Token Event Payload  
{   
 type: 'TOKEN',  
 methodArn: 'arn:aws:execute-api:us-east-1:2198525...',  
 authorizationToken: 'Bearer eyJraWQiOiJ0ekgt...'  
}

Upon receiving this event, your Lambda authorizer will issue an HTTP POST request to your identity provider to validate the token, and use the scopes present in the third-party token with a permissions mapping document to generate and return an identity management policy that contains the allowed actions of the user within API Gateway. Lambda authorizers can be written in any Lambda-supported language. You can explore some starter code templates on GitHub. The example function in this post uses Node.js 10.x.

The Lambda authorizer code in this post uses a static permissions mapping document. This document is represented by apiPermissions. For a complex or highly dynamic permissions document, this document can be decoupled from the Lambda authorizer and exported to Amazon Simple Storage Service (Amazon S3) or Amazon DynamoDB for simplified management. The static document contains the ARN of the deployed API, the API Gateway stage, the API resource, the HTTP method, and the allowed token scope. The Lambda authorizer then generates an identity management policy by evaluating the scopes present in the third-party token against those present in the document.

The fragment below shows an example permissions mapping. This mapping restricts access by requiring that users issuing HTTP GET requests to the ARN arn:aws:execute-api:us-east-1:219852565112:rz8w6b1ik2 and the my-resource resource in the DEV API Gateway stage are only allowed if they provide a valid token that contains the email scope.


# Example permissions document  
{  
 "arn": "arn:aws:execute-api:us-east-1:219852565112:rz8w6b1ik2",  
 "resource": "my-resource",  
 "stage": "DEV",  
 "httpVerb": "GET",  
 "scope": "email"  
}

The logic to create the identity management policy can be found in the generateIAMPolicy() method of the Lambda function. This method serves as a good general example of the extent of customization possible in Lambda authorizers. While the method in the example relies solely on token scopes, you can also use additional information such as request context, user information, source IP address, user agents, and so on, to generate the returned identity management policy.

Upon invocation, the Lambda authorizer below performs the following procedure:

  1. Receive the token event payload, and isolate the token string (trim “Bearer ” from the token string, if present).
  2. Verify the token with the third-party identity provider.

    Note: This Lambda function does not include this functionality. The method, verifyAccessToken(), will need to be customized based on the identity provider that you are using. This code assumes that the verifyAccessToken() method returns a Promise that resolves to the decoded token in JSON format.

  3. Retrieve the scopes from the decoded token. This code assumes these scopes can be accessed as an array at claims.scp in the decoded token.
  4. Iterate over the scopes present in the token and create identity and access management (IAM) policy statements based on entries in the permissions mapping document that contain the scope in question.
  5. Create a complete, well-formed IAM policy using the generated IAM policy statements. Refer to IAM JSON Policy Elements Reference for more information on programmatically building IAM policies.
  6. Return complete IAM policy to API Gateway.
    
    /*
     * Sample Lambda Authorizer to validate tokens originating from
     * 3rd Party Identity Provider and generate an IAM Policy
     */
    
    const apiPermissions = [
      {
        "arn": "arn:aws:execute-api:us-east-1:219852565112:rz8w6b1ik2", // NOTE: Replace with your API Gateway API ARN
        "resource": "my-resource", // NOTE: Replace with your API Gateway Resource
        "stage": "dev", // NOTE: Replace with your API Gateway Stage
        "httpVerb": "GET",
        "scope": "email"
      }
    ];
    
    var generatePolicyStatement = function (apiName, apiStage, apiVerb, apiResource, action) {
      'use strict';
      // Generate an IAM policy statement
      var statement = {};
      statement.Action = 'execute-api:Invoke';
      statement.Effect = action;
      var methodArn = apiName + "/" + apiStage + "/" + apiVerb + "/" + apiResource + "/";
      statement.Resource = methodArn;
      return statement;
    };
    
    var generatePolicy = function (principalId, policyStatements) {
      'use strict';
      // Generate a fully formed IAM policy
      var authResponse = {};
      authResponse.principalId = principalId;
      var policyDocument = {};
      policyDocument.Version = '2012-10-17';
      policyDocument.Statement = policyStatements;
      authResponse.policyDocument = policyDocument;
      return authResponse;
    };
    
    var verifyAccessToken = function (accessToken) {
      'use strict';
      /*
      * Verify the access token with your Identity Provider here (check if your 
      * Identity Provider provides an SDK).
      *
      * This example assumes this method returns a Promise that resolves to 
      * the decoded token, you may need to modify your code according to how
      * your token is verified and what your Identity Provider returns.
      */
    };
    
    var generateIAMPolicy = function (scopeClaims) {
      'use strict';
      // Declare empty policy statements array
      var policyStatements = [];
      // Iterate over API Permissions
      for ( var i = 0; i  -1 ) {
          // User token has appropriate scope, add API permission to policy statements
          policyStatements.push(generatePolicyStatement(apiPermissions[i].arn, apiPermissions[i].stage, apiPermissions[i].httpVerb,
                                                        apiPermissions[i].resource, "Allow"));
        }
      }
      // Check if no policy statements are generated, if so, create default deny all policy statement
      if (policyStatements.length === 0) {
        var policyStatement = generatePolicyStatement("*", "*", "*", "*", "Deny");
        policyStatements.push(policyStatement);
      }
      return generatePolicy('user', policyStatements);
    };
    
    exports.handler = async function(event, context) {
      // Declare Policy
      var iamPolicy = null;
      // Capture raw token and trim 'Bearer ' string, if present
      var token = event.authorizationToken.replace("Bearer ", "");
      // Validate token
      await verifyAccessToken(token).then(data => {
        // Retrieve token scopes
        var scopeClaims = data.claims.scp;
        // Generate IAM Policy
        iamPolicy = generateIAMPolicy(scopeClaims);
      })
      .catch(err => {
        console.log(err);
        // Generate default deny all policy statement if there is an error
        var policyStatements = [];
        var policyStatement = generatePolicyStatement("*", "*", "*", "*", "Deny");
        policyStatements.push(policyStatement);
        iamPolicy = generatePolicy('user', policyStatements);
      });
      return iamPolicy;
    };  
    

The following is an example of the identity management policy that is returned from your function.


# Example IAM Policy
{
  "principalId": "user",
  "policyDocument": {
    "Version": "2012-10-17",
    "Statement": [
      {
        "Action": "execute-api:Invoke",
        "Effect": "Allow",
        "Resource": "arn:aws:execute-api:us-east-1:219852565112:rz8w6b1ik2/get/DEV/my-resource/"
      }
    ]
  }
}

It is important to note that the Lambda authorizer above is not considering the method or resource that the user is requesting. This is because you want to generate a complete identity management policy that contains all the API permissions for the user, instead of a policy that only contains allow/deny for the requested resource. By generating a complete policy, this policy can be cached by API Gateway and used if the user invokes a different API while the policy is still in the cache. Caching the policy can reduce API latency from the user perspective, as well as the total amount of Lambda invocations; however, it can also increase vulnerability to Replay Attacks and acceptance of expired/revoked tokens.

Shorter cache lifetimes introduce more latency to API calls (that is, the Lambda authorizer must be called more frequently), while longer cache lifetimes introduce the possibility of a token expiring or being revoked by the identity provider, but still being used to return a valid identity management policy. For example, the following scenario is possible when caching tokens in API Gateway:

  • Identity provider stamps access token with an expiration date of 12:30.
  • User calls API Gateway with access token at 12:29.
  • Lambda authorizer generates identity management policy and API Gateway caches the token/policy pair for 5 minutes.
  • User calls API Gateway with same access token at 12:32.
  • API Gateway evaluates access against policy that exists in the cache, despite original token being expired.

Since tokens are not re-validated by the Lambda authorizer or API Gateway once they are placed in the API Gateway cache, long cache lifetimes may also increase susceptibility to Replay Attacks. Longer cache lifetimes and large identity management policies can increase the performance of your application, but must be evaluated against the trade-off of increased exposure to certain security vulnerabilities.

Deploying the Lambda authorizer

To deploy your Lambda authorizer, you first need to create and deploy a Lambda deployment package containing your function code and dependencies (if applicable). Lambda authorizer functions behave the same as other Lambda functions in terms of deployment and packaging. For more information on packaging and deploying a Lambda function, see AWS Lambda Deployment Packages in Node.js. For this example, you should name your Lambda function myLambdaAuth and use a Node.js 10.x runtime environment.

After the function is created, add the Lambda authorizer to API Gateway.

  1. Navigate to API Gateway and in the navigation pane, under APIs, select the API you configured earlier
  2. Under your API name, choose Authorizers, then choose Create New Authorizer.
  3. Under Create Authorizer, do the following:
    1. For Name, enter a name for your Lambda authorizer. In this example, the authorizer is named Lambda-Authorizer-Demo.
    2. For Type, select Lambda
    3. For Lambda Function, select the AWS Region you created your function in, then enter the name of the Lambda function you just created.
    4. Leave Lambda Invoke Role empty.
    5. For Lambda Event Payload choose Token.
    6. For Token Source, enter Authorization.
    7. For Token Validation, enter:
      
      ^(Bearer )[a-zA-Z0-9\-_]+?\.[a-zA-Z0-9\-_]+?\.([a-zA-Z0-9\-_]+)$
      			

      This represents a regular expression for validating that tokens match JWT format (more below).

    8. For Authorization Caching, select Enabled and enter a time to live (TTL) of 1 second.
  4. Select Save.

 

Figure 2: Create a new Lambda authorizer

Figure 2: Create a new Lambda authorizer

This configuration passes the token event payload mentioned above to your Lambda authorizer, and is necessary since you are using tokens (Token Event Payload) for authentication, rather than request parameters (Request Event Payload). For more information, see Use API Gateway Lambda Authorizers.

In this solution, the token source is the Authorization header of the HTTP request. If you know the expected format of your token, you can include a regular expression in the Token Validation field, which automatically rejects any request that does not match the regular expression. Token validations are not mandatory. This example assumes the token is a JWT.


# Regex matching JWT Bearer Tokens  
^(Bearer )[a-zA-Z0-9\-_]+?\.[a-zA-Z0-9\-_]+?\.([a-zA-Z0-9\-_]+)$

Here, you can also configure how long the token/policy pair will be cached in API Gateway. This example enables caching with a TTL of 1 second.

In this solution, you leave the Lambda Invoke Role field empty. This field is used to provide an IAM role that allows API Gateway to execute the Lambda authorizer. If left blank, API Gateway configures a default resource-based policy that allows it to invoke the Lambda authorizer.

The final step is to point your API Gateway resource to your Lambda authorizer. Select the configured API Resource and HTTP method.

  1. Navigate to API Gateway and in the navigation pane, under APIs, select the API you configured earlier.
  2. Select the GET method.

    Figure 3: GET Method Execution

    Figure 3: GET Method Execution

  3. Select Method Request.
  4. Under Settings, edit Authorization and select the authorizer you just configured (in this example, Lambda-Authorizer-Demo).

    Figure 4: Select your API authorizer

    Figure 4: Select your API authorizer

Deploy the API to an API Gateway stage that matches the stage configured in the Lambda authorizer permissions document (apiPermissions variable).

  1. Navigate to API Gateway and in the navigation pane, under APIs, select the API you configured earlier.
  2. Select the / resource of your API.
  3. Select Actions, and under API Actions, select Deploy API.
  4. For Deployment stage, select [New Stage] and for the Stage name, enter dev. Leave Stage description and Deployment description blank.
  5. Select Deploy.

    Figure 5: Deploy your API stage

    Figure 5: Deploy your API stage

Testing the results

With the Lambda authorizer configured as your authorization source, you are now able to access the resource only if you provide a valid token that contains the email scope.

The following example shows how to issue an HTTP request with curl to your API Gateway resource using a valid token that contains the email scope passed in the HTTP Authorization header. Here, you are able to authenticate and receive an appropriate response from API Gateway.


# HTTP Request (including valid token with "email" scope)  
$ curl -X GET \  
> 'https://rz8w6b1ik2.execute-api.us-east-1.amazonaws.com/dev/my-resource/?myParam=myValue' \  
> -H 'Authorization: Bearer eyJraWQiOiJ0ekgtb1Z5eE...'  
  
{  
 "statusCode" : 200,  
 "message" : "Hello from API Gateway!"  
}

The following JSON object represents the decoded JWT payload used in the previous example. The JSON object captures the token scopes in scp, and you can see that the token contained the email scope.

Figure 6: JSON object that contains the email scope

Figure 6: JSON object that contains the email scope

If you provide a token that is expired, is invalid, or that does not contain the email scope, then you are not able to access the resource. The following example shows a request to your API Gateway resource with a valid token that does not contain the email scope. In this example, the Lambda authorizer rejects the request.


# HTTP Request (including token without "email" scope)  
$ curl -X GET \  
> 'https://rz8w6b1ik2.execute-api.us-east-1.amazonaws.com/dev/my-resource/?myParam=myValue' \  
> -H 'Authorization: Bearer eyJraWQiOiJ0ekgtb1Z5eE...'  
  
{  
 "Message" : "User is not authorized to access this resource with an explicit deny"  
}

The following JSON object represents the decoded JWT payload used in the above example; it does not include the email scope.

Figure 7: JSON object that does not contain the email scope

Figure 7: JSON object that does not contain the email scope

If you provide no token, or you provide a token not matching the provided regular expression, then you are immediately rejected by API Gateway without invoking the Lambda authorizer. API Gateway only forwards tokens to the Lambda authorizer that have the HTTP Authorization header and pass the token validation regular expression, if a regular expression was provided. If the request does not pass token validation or does not have an HTTP Authorization header, API Gateway rejects it with a default HTTP 401 response. The following example shows how to issue a request to your API Gateway resource using an invalid token that does match the regular expression you configured on your authorizer. In this example, API Gateway rejects your request automatically without invoking the authorizer.


# HTTP Request (including a token that is not a JWT)  
$ curl -X GET \  
> 'https://rz8w6b1ik2.execute-api.us-east-1.amazonaws.com/dev/my-resource/?myParam=myValue' \  
> -H 'Authorization: Bearer ThisIsNotAJWT'  
  
{  
 "Message" : "Unauthorized"  
}

These examples demonstrate how your Lambda authorizer allows and denies requests based on the token format and the token content.

Conclusion

In this post, you saw how Lambda authorizers can be used with API Gateway to implement a token-based authentication scheme using third-party tokens.

Lambda authorizers can provide a number of benefits:

  • Leverage third-party identity management services directly, without identity federation.
  • Implement custom authorization logic.
  • Cache identity management policies to improve performance of authorization logic (while keeping in mind security implications).
  • Minimally impact existing client applications.

For organizations seeking an alternative to Amazon Cognito User Pools and Amazon Cognito identity pools, Lambda authorizers can provide complete, secure, and flexible authentication and authorization services to resources deployed with Amazon API Gateway. For more information about Lambda authorizers, see API Gateway Lambda Authorizers.

If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Bryant Bost

Bryant Bost is an Application Consultant for AWS Professional Services based out of Washington, DC. As a consultant, he supports customers with architecting, developing, and operating new applications, as well as migrating existing applications to AWS. In addition to web application development, Bryant specializes in serverless and container architectures, and has authored several posts on these topics.

Identifying and resolving security code vulnerabilities using Snyk in AWS CI/CD Pipeline

Post Syndicated from Jay Yeras original https://aws.amazon.com/blogs/devops/identifying-and-resolving-vulnerabilities-in-your-code/

The majority of companies have embraced open-source software (OSS) at an accelerated rate even when building proprietary applications. Some of the obvious benefits for this shift include transparency, cost, flexibility, and a faster time to market. Snyk’s unique combination of developer-first tooling and best in class security depth enables businesses to easily build security into their continuous development process.

Even for teams building proprietary code, use of open-source packages and libraries is a necessity. In reality, a developer’s own code is often a small core within the app, and the rest is open-source software. While relying on third-party elements has obvious benefits, it also presents numerous complexities. Inadvertently introducing vulnerabilities into your codebase through repositories that are maintained in a distributed fashion and with widely varying levels of security expertise can be common, and opens up applications to effective attacks downstream.

There are three common barriers to truly effective open-source security:

  1. The security task remains in the realm of security and compliance, often perpetuating the siloed structure that DevOps strives to eliminate and slowing down release pace.
  2. Current practice may offer automated scanning of repositories, but the remediation advice it provides is manual and often un-actionable.
  3. The data generated often focuses solely on public sources, without unique and timely insights.

Developer-led application security

This blog post demonstrates techniques to improve your application security posture using Snyk tools to seamlessly integrate within the developer workflow using AWS services such as Amazon ECR, AWS Lambda, AWS CodePipeline, and AWS CodeBuild. Snyk is a SaaS offering that organizations use to find, fix, prevent, and monitor open source dependencies. Snyk is a developer-first platform that can be easily integrated into the Software Development Lifecycle (SDLC). The examples presented in this post enable you to actively scan code checked into source code management, container images, and serverless, creating a highly efficient and effective method of managing the risk inherent to open source dependencies.

Prerequisites

The examples provided in this post assume that you already have an AWS account and that your account has the ability to create new IAM roles and scope other IAM permissions. You can use your integrated development environment (IDE) of choice. The examples reference AWS Cloud9 cloud-based IDE. An AWS Quick Start for Cloud9 is available to quickly deploy to either a new or existing Amazon VPC and offers expandable Amazon EBS volume size.

Sample code and AWS CloudFormation templates are available to simplify provisioning the various services you need to configure this integration. You can fork or clone those resources. You also need a working knowledge of git and how to fork or clone within your source provider to complete these tasks.

cd ~/environment && \ 
git clone https://github.com/aws-samples/aws-modernization-with-snyk.git modernization-workshop 
cd modernization-workshop 
git submodule init 
git submodule update

Configure your CI/CD pipeline

The workflow for this example consists of a continuous integration and continuous delivery pipeline leveraging AWS CodeCommit, AWS CodePipeline, AWS CodeBuild, Amazon ECR, and AWS Fargate, as shown in the following screenshot.

CI/CD Pipeline

For simplicity, AWS CloudFormation templates are available in the sample repo for services.yaml, pipeline.yaml, and ecs-fargate.yaml, which deploy all services necessary for this example.

Launch AWS CloudFormation templates

A detailed step-by-step guide can be found in the self-paced workshop, but if you are familiar with AWS CloudFormation, you can launch the templates in three steps. From your Cloud9 IDE terminal, change directory to the location of the sample templates and complete the following three steps.

1) Launch basic services

aws cloudformation create-stack --stack-name WorkshopServices --template-body file://services.yaml \
--capabilities CAPABILITY_NAMED_IAM until [[ `aws cloudformation describe-stacks \
--stack-name "WorkshopServices" --query "Stacks[0].[StackStatus]" \
--output text` == "CREATE_COMPLETE" ]]; do echo "The stack is NOT in a state of CREATE_COMPLETE at `date`"; sleep 30; done &&; echo "The Stack is built at `date` - Please proceed"

2) Launch Fargate:

aws cloudformation create-stack --stack-name WorkshopECS --template-body file://ecs-fargate.yaml \
--capabilities CAPABILITY_NAMED_IAM until [[ `aws cloudformation describe-stacks \ 
--stack-name "WorkshopECS" --query "Stacks[0].[StackStatus]" \ 
--output text` == "CREATE_COMPLETE" ]]; do echo "The stack is NOT in a state of CREATE_COMPLETE at `date`"; sleep 30; done &&; echo "The Stack is built at `date` - Please proceed"

3) From your Cloud9 IDE terminal, change directory to the location of the sample templates and run the following command:

aws cloudformation create-stack --stack-name WorkshopPipeline --template-body file://pipeline.yaml \
--capabilities CAPABILITY_NAMED_IAM until [[ `aws cloudformation describe-stacks \
--stack-name "WorkshopPipeline" --query "Stacks[0].[StackStatus]" \
--output text` == "CREATE_COMPLETE" ]]; do echo "The stack is NOT in a state of CREATE_COMPLETE at `date`"; sleep 30; done &&; echo "The Stack is built at `date` - Please proceed"

Improving your security posture

You need to sign up for a free account with Snyk. You may use your Google, Bitbucket, or Github credentials to sign up. Snyk utilizes these services for authentication and does not store your password. Once signed up, navigate to your name and select Account Settings. Under API Token, choose Show, which will reveal the token to copy, and copy this value. It will be unique for each user.

Save your password to the session manager

Run the following command, replacing abc123 with your unique token. This places the token in the session parameter manager.

aws ssm put-parameter --name "snykAuthToken" --value "abc123" --type SecureString

Set up application scanning

Next, you need to insert testing with Snyk after maven builds the application. The simplest method is to insert commands to download, authorize, and run the Snyk commands after maven has built the application/dependency tree.

The sample Dockerfile contains an environment variable from a value passed to the docker build command, which contains the token for Snyk. By using an environment variable, Snyk automatically detects the token when used.

#~~~~~~~SNYK Variable~~~~~~~~~~~~ 
# Declare Snyktoken as a build-arg ARG snyk_auth_token
# Set the SNYK_TOKEN environment variable ENV
SNYK_TOKEN=${snyk_auth_token}
#~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Download Snyk, and run a test, looking for medium to high severity issues. If the build succeeds, post the results to Snyk for monitoring and reporting. If a new vulnerability is found, you are notified.

# package the application
RUN mvn package -Dmaven.test.skip=true

#~~~~~~~SNYK test~~~~~~~~~~~~
# download, configure and run snyk. Break build if vulns present, post results to `https://snyk.io/`
RUN curl -Lo ./snyk "https://github.com/snyk/snyk/releases/download/v1.210.0/snyk-linux"
RUN chmod -R +x ./snyk
#Auth set through environment variable
RUN ./snyk test --severity-threshold=medium
RUN ./snyk monitor

Set up docker scanning

Later in the build process, a docker image is created. Analyze it for vulnerabilities in buildspec.yml. First, pull the Snyk token snykAuthToken from the parameter store.

env:
  parameter-store:
    SNYK_AUTH_TOKEN: "snykAuthToken"

Next, in the prebuild phase, install Snyk.

phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - aws --version
      - $(aws ecr get-login --region $AWS_DEFAULT_REGION --no-include-email)
      - REPOSITORY_URI=$(aws ecr describe-repositories --repository-name petstore_frontend --query=repositories[0].repositoryUri --output=text)
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
      - IMAGE_TAG=${COMMIT_HASH:=latest}
      - PWD=$(pwd)
      - PWDUTILS=$(pwd)
      - curl -Lo ./snyk "https://github.com/snyk/snyk/releases/download/v1.210.0/snyk-linux"
      - chmod -R +x ./snyk

Next, in the build phase, pass the token to the docker compose command, where it is retrieved in the Dockerfile code you set up to test the application.

build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image...
      - cd modules/containerize-application
      - docker build --build-arg snyk_auth_token=$SNYK_AUTH_TOKEN -t $REPOSITORY_URI:latest.

You can further extend the build phase to authorize the Snyk instance for testing the Docker image that’s produced. If it passes, you can pass the results to Snyk for monitoring and reporting.

build:
    commands:
      - $PWDUTILS/snyk auth $SNYK_AUTH_TOKEN
      - $PWDUTILS/snyk test --docker $REPOSITORY_URI:latest
      - $PWDUTILS/snyk monitor --docker $REPOSITORY_URI:latest
      - docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG

For reference, a sample buildspec.yaml configured with Snyk is available in the sample repo. You can either copy this file and overwrite your existing buildspec.yaml or open an editor and replace the contents.

Testing the application

Now that services have been provisioned and Snyk tools have been integrated into your CI/CD pipeline, any new git commit triggers a fresh build and application scanning with Snyk detects vulnerabilities in your code.

In the CodeBuild console, you can look at your build history to see why your build failed, identify security vulnerabilities, and pinpoint how to fix them.

Testing /usr/src/app...
✗ Medium severity vulnerability found in org.primefaces:primefaces
Description: Cross-site Scripting (XSS)
Info: https://snyk.io/vuln/SNYK-JAVA-ORGPRIMEFACES-31642
Introduced through: org.primefaces:[email protected]
From: org.primefaces:[email protected]
Remediation:
Upgrade direct dependency org.primefaces:[email protected] to org.primefaces:[email protected] (triggers upgrades to org.primefaces:[email protected])
✗ Medium severity vulnerability found in org.primefaces:primefaces
Description: Cross-site Scripting (XSS)
Info: https://snyk.io/vuln/SNYK-JAVA-ORGPRIMEFACES-31643
Introduced through: org.primefaces:[email protected]
From: org.primefaces:[email protected]
Remediation:
Upgrade direct dependency org.primefaces:[email protected] to org.primefaces:[email protected] (triggers upgrades to org.primefaces:[email protected])
Organisation: sample-integrations
Package manager: maven
Target file: pom.xml
Open source: no
Project path: /usr/src/app
Tested 37 dependencies for known vulnerabilities, found 2 vulnerabilities, 2 vulnerable paths.
The command '/bin/sh -c ./snyk test' returned a non-zero code: 1
[Container] 2020/02/14 03:46:22 Command did not exit successfully docker build --build-arg snyk_auth_token=$SNYK_AUTH_TOKEN -t $REPOSITORY_URI:latest . exit status 1
[Container] 2020/02/14 03:46:22 Phase complete: BUILD Success: false
[Container] 2020/02/14 03:46:22 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: docker build --build-arg snyk_auth_token=$SNYK_AUTH_TOKEN -t $REPOSITORY_URI:latest .. Reason: exit status 1

Remediation

Once you remediate your vulnerabilities and check in your code, another build is triggered and an additional scan is performed by Snyk. This time, you should see the build pass with a status of Succeeded.

You can also drill down into the CodeBuild logs and see that Snyk successfully scanned the Docker Image and found no package dependency issues with your Docker container!

[Container] 2020/02/14 03:54:14 Running command $PWDUTILS/snyk test --docker $REPOSITORY_URI:latest
Testing 300326902600.dkr.ecr.us-west-2.amazonaws.com/petstore_frontend:latest...
Organisation: sample-integrations
Package manager: rpm
Docker image: 300326902600.dkr.ecr.us-west-2.amazonaws.com/petstore_frontend:latest
✓ Tested 190 dependencies for known vulnerabilities, no vulnerable paths found.

Reporting

Snyk provides detailed reports for your imported projects. You can navigate to Projects and choose View Report to set the frequency with which the project is checked for vulnerabilities. You can also choose View Report and then the Dependencies tab to see which libraries were used. Snyk offers a comprehensive database and remediation guidance for known vulnerabilities in their Vulnerability DB. Specifics on potential vulnerabilities that may exist in your code would be contingent on the particular open source dependencies used with your application.

Cleaning up

Remember to delete any resources you may have created in order to avoid additional costs. If you used the AWS CloudFormation templates provided here, you can safely remove them by deleting those stacks from the AWS CloudFormation Console.

Conclusion

In this post, you learned how to leverage various AWS services to build a fully automated CI/CD pipeline and cloud IDE development environment. You also learned how to utilize Snyk to seamlessly integrate with AWS and secure your open-source dependencies and container images. If you are interested in learning more about DevSecOps with Snyk and AWS, then I invite you to check out this workshop and watch this video.

 

About the Author

Author Photo

 

Jay is a Senior Partner Solutions Architect at AWS bringing over 20 years of experience in various technical roles. He holds a Master of Science degree in Computer Information Systems and is a subject matter expert and thought leader for strategic initiatives that help customers embrace a DevOps culture.

 

 

Continuous compliance monitoring with Chef InSpec and AWS Security Hub

Post Syndicated from Jonathan Rau original https://aws.amazon.com/blogs/security/continuous-compliance-monitoring-with-chef-inspec-and-aws-security-hub/

In this post, I will show you how to run a Chef InSpec scan with AWS Systems Manager and Systems Manager Run Command across your managed instances. InSpec is an open-source runtime framework that lets you create human-readable profiles to define security, compliance, and policy requirements and then test your Amazon Elastic Compute Cloud (Amazon EC2) instances against those profiles. InSpec profiles can also be used to make sure certain network ports aren’t reachable, to verify that certain packages are not installed, and/or to confirm that certain processes are running on your instances.

InSpec is integrated within AWS Systems Manager, an AWS service that you can use to view and control your infrastructure on AWS. InSpec compliance scans are run by using an AWS Systems Manager document (SSM document), which installs InSpec on your servers and removes InSpec after scans are completed.

In this post, you will create the supporting infrastructure to collect non-compliant findings from AWS Systems Manager Configuration Compliance and send them to AWS Security Hub via a Lambda function. You will also explore methods to correlate finding information in Security Hub for non-compliant resources.

Note: AWS Systems Manager (Systems Manager) was formerly known as “Amazon Simple Systems Manager (SSM)” and “Amazon EC2 Systems Manager (SSM).” The original abbreviated name of the service, “SSM,” is still reflected in various AWS resources. For more information, see Systems Manager Service Name History.

Solution overview

The following diagram shows the flow of events in the solution I describe in this post.
 

Figure 1: Architecture diagram

Figure 1: Architecture diagram

  1. Invoke an AWS-RunInSpecChecks document on-demand by using Run Command against your target instances (State Manager is another option for scheduling InSpec scans, but is not covered in this post).
  2. Systems Manager downloads the InSpec Ruby files from Amazon Simple Storage Service (Amazon S3), installs InSpec on your server, runs the scan, and removes InSpec when complete.
  3. AWS Systems Manager pushes scan results to the Compliance API and presents the information in the Systems Manager Compliance console, to include severity and compliance state.
  4. A CloudWatch Event is emitted for Compliance state changes.
  5. A CloudWatch Event Rule listens for these state changes and when detected, invokes a Lambda function.
  6. Lambda calls the Compliance APIs for additional data about which InSpec check failed.
  7. Lambda calls the EC2 APIs to further enrich the data about the non-compliant instance.
  8. Lambda maps these details to the AWS Security Finding Format and sends them to Security Hub.

To support the steps above, you will deploy a CloudFormation template that creates a CloudWatch Event Rule and a placeholder Lambda function. You will then create an InSpec profile, upload it to Amazon S3, and use Run Command to invoke an InSpec compliance scan.

When the scan completes, you can then search for the findings in Security Hub. You can create saved searches with insights in AWS Security Hub, and use different filtering to correlate InSpec compliance failures with other information from Amazon Inspector and Amazon GuardDuty.

Prerequisites

Getting started

Before creating your CloudFormation stack, you’ll upload your InSpec profile and Lambda function package to an S3 bucket.

To upload the Lambda package:

  1. Download the Lambda function ZIP file from GitHub.
  2. Save the file to your workstation. In your S3 bucket, choose Upload to upload the InSpecToSecurityHub.zip file.

The next step is to create and upload an InSpec profile to Amazon S3.

To create and upload an InSpec profile:

  1. From your workstation, create a file named Inspec_SSH.rb and paste in the code that follows:
    
    	control 'Linux instance check' do
        title 'SSH access'
        desc 'SSH port should not be open to the world'
        impact 0.9
        require 'rbconfig'
        is_windows = (RbConfig::CONFIG['host_os'] =~ /mswin|mingw|cygwin/)
        if ! is_windows
          describe port(22) do
            it { should be_listening }
            its('addresses') {should_not include '0.0.0.0'}
          end
        end
      end
    	

  2. Save your file, and in your S3 bucket, choose Upload to upload the Inspec_SSH.rb file.

Note: The InSpec profile in the example code above ensures that SSH (Port 22) is listening on your instance and that SSH access is not publicly available, as noted by {should_not include ‘0.0.0.0’}. For other InSpec profiles, see the DevSec chef-os-hardening project on GitHub for profiles to help you secure your instances and servers, or you can learn more about Compliance Automation with InSpec on the Chef website.

You will now deploy the CloudFormation template to finish setting up the solution.

To deploy the CloudFormation template:

  1. Download the CloudFormation template from GitHub and create a CloudFormation stack.

    Note: For more information about how to create a CloudFormation stack, see Getting Started with AWS CloudFormation in the AWS CloudFormation User Guide.

  2. Under Parameters, enter the name of the bucket you uploaded the package to, as shown in Figure 2, and finish creating your stack
     
    Figure 2: CloudFormation parameters

    Figure 2: CloudFormation parameters

After your stack is successfully created, you will run an InSpec scan.

Run an InSpec scan

Now that you have your Lambda function and InSpec profile ready, you will run your InSpec scan against one or more managed instances with Run Command.

  1. Navigate to the AWS Systems Manager Console and in the navigation pane, choose Run Command.
  2. In the search box, enter AWS-RunInspecChecks. In the search results, select AWS-RunInspecChecks.
  3. For Source Type, select S3. The Source Info format is:
    
    	{"path":"https://s3.amazonaws.com/BUCKET_NAME/PATH/ Inspec_SSH.rb"}
    	

    Change BUCKET_NAME, PATH, and the InSpec profile name as appropriate, and enter it in the Source Info box, as shown in Figure 3.

  4. Under Targets, select Choose instances manually and select one or more instances, as shown in Figure 3.
     
    Figure 3: Run Command interface

    Figure 3: Run Command interface

  5. Scroll down to Output options, and select S3, CloudWatch Logs, or both, then select Run.
  6. In the Run Command console for the specific Command invocation you just sent, you will see your instances and their status, as shown in Figure 4. Refresh the page after a few minutes to see if the invocation was successful.
     
    Figure 4: Run Command invocation status

    Figure 4: Run Command invocation status

  7. If you see anything other than Success, you can find information about different statuses in Understanding Command Statuses. You can also see Troubleshooting Systems Manager Run Command.
  8. In the AWS Systems Manager Console, navigate to the Compliance menu to see your full Compliance resources summary, as shown in Figure 5. You may also see additional compliance types and statuses. Only failed Custom:InSpec compliance types will make their way to Security Hub.
     
    Figure 5: SSM Compliance dashboard

    Figure 5: SSM Compliance dashboard

Analyze with Security Hub

After Lambda has sent your non-compliant InSpec results to Security Hub, you can create saved searches by using Security Hub insights to do basic correlation.

  1. Navigate to the Security Hub console and in the navigation pane, choose Findings.
  2. To find InSpec-related findings, select the search bar, and choose Product fields.
  3. For the Key field, enter Provider Name and for the Value field enter AWS Systems Manager Compliance, then choose Apply, as shown in Figure 6.
     
    Figure 6: User defined field - filter

    Figure 6: User defined field – filter

    This will give you your InSpec-related findings, as shown in Figure 7, but there is not enough data to correlate across to other findings from GuardDuty or Amazon Inspector.
     

    Figure 7: Security Hub InSpec findings

    Figure 7: Security Hub InSpec findings

  4. Navigate the Insights menu on the navigation pane and choose Create insight on the top-right.
  5. Select the search bar, enter Resource type, enter AwsEc2Instance and select Apply, as shown in Figure 8. This will group all findings related to EC2 instances together.
     
    Figure 8: Filter insights for EC2 instances

    Figure 8: Filter insights for EC2 instances

  6. Select the search bar again, select Group by, scroll down to select Severity label and select Apply. We now can see our total findings by severity across all EC2 instances, as shown in Figure 9.
     
    Figure 9: Insights grouped by severity

    Figure 9: Insights grouped by severity

  7. On the top right select Create insight, enter a name such as EC2 Instances by Severity and choose Create insight to save your insight.
  8. Navigate back to the main Insights menu and select your new Insight for more detailed graphics to appear on the right-hand side, as shown in Figure 10.
     
    Figure 10: Detailed Insight menu

    Figure 10: Detailed Insight menu

    You can hover over the values to get the exact count and select them to filter down further on details such as the ARN of the resource, the name of the product, or what account the finding originated from. This view can be used by security analysts who are looking to do fast correlation of findings for a resource or to take action on high severity findings first.

Conclusion

In this post, I showed you how to run InSpec scans to monitor the compliance of your instances against your policy requirements, as defined by InSpec profiles. InSpec can help identify when certain points are improperly configured or publicly accessible. By using Systems Manager, you can continuously monitor the compliance against these profiles with State Manager, and run these checks on demand by using Run Command. Systems Manager allows you to rapidly scale across your managed instances, and enriches instance data through the SSM Agent.

You also learned how to use Security Hub to perform correlation within the findings menu, so you can quickly search based on resource type, or the VPC and subnet that the EC2 instances are in. When you have an instance that fails compliance against an InSpec profile, you can look for any threat-related findings from GuardDuty, or vulnerability information from Amazon Inspector or partner integrations.

To avoid incurring additional charges from the resources created in this blog, delete the CloudFormation stack you deployed. For more information on Security Hub pricing, see the AWS Security Hub Pricing page.

If you have feedback about this blog post, submit comments in the Comments section below. If you have questions about this blog post, start a new thread on the Security Hub forum or contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Jonathan Rau

Jonathan is the Senior TPM for AWS Security Hub. He holds an AWS Certified Specialty-Security certification and is extremely passionate about cyber security, data privacy, and new emerging technologies, such as blockchain. He devotes personal time into research and advocacy about those same topics.

How to create certificates with custom extensions using AWS Certificate Manager Private CA

Post Syndicated from Josh Rosenthol original https://aws.amazon.com/blogs/security/how-to-create-certificates-with-custom-extensions-using-aws-certificate-manager-private-ca/

Digital certificates, also known as X.509 or TLS/SSL certificates, are used to prove the identity of entities like web servers or VPN users and to establish secure communication channels between them. In this blog post, I’ll discuss certificate extensions. You can use certificate extensions for applications beyond the common use case of identifying TLS server endpoints. These additional applications include code signing, signing Online Certificate Status Protocol (OCSP) responses, TLS clients for establishing two-way (mutual) authentication, and custom applications that include extensions you specify. I’ll also discuss how you can use AWS Identity and Access Management (IAM) policies to selectively control who can issue each kind of certificate. Finally, I’ll describe two use cases for certificate templates: OCSP signing and two-way TLS. A certificate template allows CA administrators and public key infrastructure (PKI) operators a way to control and specify X.509 certificate extensions for the certificates they issue with AWS Certificate Manager Private Certificate Authority (ACM Private CA).

What are certificate extensions?

Extension fields (“extensions”) define the usage of the certificate. The values of these fields — and the corresponding allowed usage of the certificates — can be very different. Certificates bind a subject name and public key for the named subject with a digital signature from a certificate authority. What makes one certificate useful for signing code, while another certificate is useful for terminating TLS are the extension fields in the certificate.

Digital certificates used on the internet (and with other modern applications) are defined in the internet standards specification RFC 5280. Extensions were introduced in v3 of the X.509 specification as a way to give certificates more flexibility. There are 19 extension types defined in RFC 5280, but the extensions widely used for defining certificate usage are basic constraints, key usage, and extended key usage.

  • Basic constraints indicate whether the certificate can be used to identify a certificate authority versus an end entity, such as a web server. Certificates that include basic constraints with a value of “CA=true” are CA certificates and are allowed to issue other certificates.
  • Key usage refers to the purpose of the public key contained in the certificate and the actions that can be performed with the key. An example of a key usage value is “digital signature.” If you use this value, it allows your public key to be used for verifying digital signatures (other than signatures on certificates and CRLs), such as those used for authentication and integrity, like when a TLS connection is established. The key usage extension is limited to nine key usage values that define the specific cryptographic functions the associated key pair can perform. Those values are: decipherOnly, encipherOnly, cRLSign, keyCertSign, keyAgreement, dataEncipherment, keyEncipherment, contentCommitment, nonRepudiation, and digitalSignature.
  • In contrast to key usage, the extended key usage extension defines the specific protocols and functions that the certificate can be used with. For example, one extended key usage value is “TLS web server authentication,” which indicates the public key can be used to terminate TLS as a server. There’s also “TLS web client authentication,” which indicates the key can be used to terminate TLS as a client, and “code signing,” which means the key can be used to validate digital signatures on software. RFC5246 defines the TLS protocol and the use of server and client certificates.

Available templates

ACM Private CA offers ten templates so you can control and specify the X.509 certificate extensions for the certificates you issue with ACM Private CA. The templates allow CA administrators to maintain control of specific X.509 certificate extensions — including key usage, extended key usage, and basic constraints — while allowing users to customize certificates with additional extensions.

Template Details

End-entity certificates

Use the end-entity certificate templates to create certificates for resources that are TLS clients or servers, such as web and application servers, API endpoints, or TLS clients like IoT (internet of things) devices and API users. All of the certificates issued by these template types have the key usage values of “digital signature” and “key encipherment.”

  • EndEntityCertificate — This template generates certificates that can be used as the server certificate in a one-way TLS session or as the server or client certificate in a two-way TLS handshake. This certificate has the extended key usage values of “TLS web server authentication” and “TLS client server authentication.”
  • EndEntityClientAuthCertificate — This template only generates certificates used by the client in two-way TLS. This certificate has an extended key usage value of “TLS Client Server Authentication.”
  • EndEntityServerAuthCertificate — This template only generates server certificates used for one-way or two-way TLS. This certificate has an extended key usage value of “TLS Web Server Authentication.”

OCSP signing certificates

An Online Certificate Status Protocol (OCSP) responder provides a mechanism to check if a certificate has been revoked. Clients that check certificate revocation status with OCSP require a response signed either by the CA or a certificate issued by the CA. Creating an OCSP signing certificate to sign OCSP responses allows you to offload response generation and signing from the CA to the responder or to another server. The ACM Private CA OCSPSigningCertificate template allows you to issue OCSP signing certificates that you can use to sign OCSP responses and operate your own OCSP responder. The key usage value for an OCSP certificate is “digital signature” and the extended key usage value is “OCSP Signing.”

Code signing certificates

Software developers use code signing certificates to digitally sign software, apps, drivers, and programs. Operating systems check code signatures before running the code to ensure it was signed by a trusted entity and has not been modified by an unauthorized third-party. The ACM Private CA CodeSigningCertificate template allows you to issue code signing certificates. The key usage value of a code signing certificate is “digital signatures” and the extended key usage value is “code signing.”

CSR passthrough certificates

A certificate signing request (CSR) is an identity document that contains a public key and a subject. You can also configure the CSR to contain a set of extension requests that the requested certificate should inclue. Normally, the CA ignores certificate extension requests included in the CSR rather than including them in the certificate it issues. In many cases, “ignore” is desired behavior because it allows the CA to remove unnecessary extensions and enforce the CA configurations. The templates described so far work this way, but there is another set of templates for which extensions from the CSR are passed to the certificate.

Private CA users can customize certificates by adding any X.509 extension to the certificate signing request (CSR) when they issue the certificate. If the CSR includes extensions that overlap with the extensions in the template, the value from the template will be used. The CA administrator can thus control the extensions in certificates by controlling access to the templates. One common example is for code signing certificates. For example, the subject directory attributes extension can be used to convey identification attributes such as the nationality of the certificate subject. If a customer is issuing certificates for different geographic locations from a single CA, they can include the SubjectDirectoryAttributes value in each individual CSR and have it correspond to the location of the domain for the certificate.

The ten certificate templates can be broken up into five pairs of templates, as shown in Table 1 below. For example, one pair would be EndEntityCertificate and EndEntityCertificate_CSRPassthrough. Each member of the pair issues the same certificate extension but with one major difference: the first template in the pair utilizes the CA values as the override for all the X509v3 fields. The second template (the CSRPassthrough template) allows CA administrators to maintain control of the most important and fundamental X.509 certificate extensions that determine how the certificate can be used, including key usage, extended key usage, and basic constraints.

The CSRPassthrough templates allow users making certificate requests to customize their certificates by including additional extensions when necessary. For example, CA users who want to build their own online certificate status protocol (OCSP) responders can include the location of the responder in certificates they issue by including a special authority information access (AIA) extension that clients use to identify the OCSP responder endpoint.

Table 1: The full list of supported templates

EndEntityCertificateEndEntityCertificate_CSRPassthrough
EndEntityClientAuthCertificateEndEntityClientAuthCertificate_CSRPassthrough
EndEntityServerAuthCertificateEndEntityServerAuthCertificate_CSRPassthrough
OCSPSigningCertificateOCSPSigningCertificate_CSRPassthrough
CodeSigningCertificateCodeSigningCertificate_CSRPassthrough

How to issue a certificate with a template

You can use ACM Private CA to issue a certificate with a template by using the ACM Private CA Command Line Interface (CLI). Here’s an example of a call to create a Server-Only TLS certificate:


aws acm-pca issue-certificate --certificate-authority-arn arn:aws:acm-pca:us-west-2:123456789012:certificate-authority/12345678-1234-1234-1234-123456789012 --csr file://C:\cert_1.csr --signing-algorithm "SHA256WITHRSA" --template-arn “arn:aws:acm-pca:::template/EndEntityServerAuthCertificate/V1” --validity Value=365,Type="DAYS"

Here are the pieces of this command:

  • The command is <issue-certificate>
  • The certificate authority ARN identifies the private CA that will be used to issue the certificate.
  • The CSR is a pointer to the file which contains the CSR.
  • The signing algorithm is used by the CA to sign the final certificate; in this example it’s SHA-256 with RSA.
  • The template ARN is for a server-only TLS certificate. The ACM Private CA documentation has the full list of templates.
  • The validity period of this certificate is set to 365 days.

This command creates a certificate with the extended key usage extension value of clientAuth. Other than that change, it’s identical to the EndEntityCertificate template as specified in the ACM Private CA User Guide.

How to configure user-based permissions

You can combine templates with Identity and Access Management (IAM) permissions to create roles with fine-grained access controls that are assigned to individuals or systems. In AWS, an IAM role has specific permissions that determine what the identity can and cannot do in AWS. For example, you might have a team in charge of building and maintaining OCSP infrastructure. The account this team uses could be configured so that it only has permission to use the OCSP signing template, as opposed to being able to issue any type of certificate. In other words, you can restrict their issuance of certificates to align with their job responsibilities. You could also use IAM policies to grant or deny the team permission to use the CSR Passthrough Template.

Here’s a sample of the IAM permissions which would deny the use of CSR Passthrough templates. This template might be used to ensure that no certificate requestors inserted custom values through a CSR passthrough template variant.


{
    "Effect": "Deny",
    "Action": [
        "acm-pca:IssueCertificate"
    ],
    "Resource": "arn:aws:acm-pca:*:*:certificate-authority/*",
    "Condition": {
        "StringLike": {
            "acm-pca:TemplateArn": [
                "arn:aws:acm-pca:::template/*CSRPassthrough*/V*"
            ]
        }
    }
},

Two use cases for certificate extensions

IoT devices with client-only TLS certificates

To authenticate and allow for secure communications with an IoT device, manufacturers frequently install TLS certificates on their devices. The production system that installs TLS certificates requires a CA in order to issue certificates with the appropriate extensions. Let’s say that a device manufacturer only wants their IoT devices to be able to initiate a TLS connection with the server. Using certificates with the client-only TLS template would prevent someone from trying to establish a connection with the device to start an inappropriate communication channel. The manufacturer also wants to enforce a fixed set of certificate extensions for the production environment, while allowing more flexibility in the extensions in locations that are considered more physically secure.

To achieve this, the manufacturer can restrict the issuing CA via IAM policies, enforcing the use of an End Entity Client Certificate (EndEntityClientAuthCertificate) that includes specific and fixed certificate extensions that will prevent the use of CSR Passthrough in the production environment. This configuration will prevent the certificate from being used as the server of a TLS transaction for inappropriate access from a third party. Not enabling CSR passthrough ensures the certificate fields are set to defined/required values.

This is the command to create an IoT certificate with client-only TLS:


aws acm-pca issue-certificate --certificate-authority-arn arn:aws:acm-pca:us-west-2:123456789012:certificate-authority/12345678-1234-1234-1234-123456789012 --csr file://C:\cert_2.csr --signing-algorithm "SHA256WITHRSA" --template-arn “arn:aws:acm-pca:::template/EndEntityClientAuthCertificate/V1” --validity Value=40,Type="YEARS"

This certificate is similar to the preceding certificate example, with two major changes: It will only work as a server-only certificate, and the validity is 40 years instead of one year. The 40-year validity is because IoT devices won’t return to the floor to be able to get a new certificate issued. After 40 years (which is beyond the expected lifespan of the device), the certificate would expire and the device would no longer be able to communicate.

In this example, the value of the ExtendedKeyUsage field is configured to “clientAuth.”

OCSP responder

One element of the construction of an OCSP responder is the OCSP signing certificate. (I’ve described the details of how the elements of an OCSP are built in the “OCSP signing certificates” section of this post.) When configuring an OCSP service, the administrator responsible for renewing that certificate should be given permission for that template. The OCSP signing certificate normally has a short duration, so the certificate needs to be renewed frequently. An important note is that an OCSP signing certificate generated by ACM Private CA only supports certificates issued by the same CA that created the OCSP signing certificate. It cannot be used to handle certificates up or down the CA hierarchy.

The command to generate a certificate for this use case is:


aws acm-pca issue-certificate --certificate-authority-arn arn:aws:acm-pca:us-west-2:123456789012:certificate-authority/12345678-1234-1234-1234-123456789012 --csr file://C:\cert_3.csr --signing-algorithm "SHA256WITHRSA" --template-arn “arn:aws:acm-pca:::template/OCSPSigningCertificate_CSRPassthrough/V1” --validity Value=365,Type="DAYS" --idempotency-token 1234

This certificate must be a CSR Passthrough because the Authority Information Access (AIA) will have the value ocsp for the accessMethod field and the URI address of the OCSP responder in the accessLocation field. Setting these two values requires CSR passthrough.

Conclusion

Certificates issued by Private CA can be used for a variety of use cases. TLS certificates are the most common. You can also generate certificates for other uses such as code signing. I’ve shown some examples of how to use templates in ACM to ensure that the certificates are created with the correct extensions. As your organization looks beyond TLS, or expands to the point where privilege for use matters, certificate templates can support the design and security guidelines of your environment.

To get started with ACM Private CA for TLS, OCSP or other use cases, read the getting started pages of the ACM Private CA documentation. If you have feedback about this blog post, submit comments in the Comments section below. If you have questions about this blog post, start a new thread on the Amazon AWS Certificate Manager forum or contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

Josh Rosenthol

Josh is a Product Manager who helps solve customer problems with public and private certificate and CAs from AWS. He enjoys listening to customers describe their use cases and translate them into improvements to AWS Certificate Manager and ACM Private CA.

Automated Response and Remediation with AWS Security Hub

Post Syndicated from Jonathan Rau original https://aws.amazon.com/blogs/security/automated-response-and-remediation-with-aws-security-hub/

AWS Security Hub is a service that gives you aggregated visibility into your security and compliance status across multiple AWS accounts. In addition to consuming findings from Amazon services and integrated partners, Security Hub gives you the option to create custom actions, which allow a customer to manually invoke a specific response or remediation action on a specific finding. You can send custom actions to Amazon CloudWatch Events as a specific event pattern, allowing you to create a CloudWatch Events rule that listens for these actions and sends them to a target service, such as a Lambda function or Amazon SQS queue.

By creating custom actions mapped to specific finding type and by developing a corresponding Lambda function for that custom action, you can achieve targeted, automated remediation for these findings. This allows a customer to specifically decide if he or she wants to invoke a remediation action on a specific finding. A customer can also use these Lambda functions as the target of fully automated remediation actions that do not require any human review.

In this blog post, I’ll show you how to build custom actions, CloudWatch Event rules, and Lambda functions for a dozen targeted actions that can help you remediate CIS AWS Foundations Benchmark-related compliance findings. I’ll also cover use cases for sending findings to an issue management system and for automating security patching. To promote rapid deployment and adoption of this solution, you’ll deploy a majority of the necessary components via AWS CloudFormation.

Note: The full repository for current and future response and remediation templates is hosted on GitHub and includes additional technical guidance for expanding the solution provided in this post.

Solution architecture

Figure 1 - Solution Architecture Overview

Figure 1 – Solution Architecture Overview

Figure 1 shows how a finding travels from an integrated service to a custom action:

  1. Integrated services send their findings to Security Hub.
  2. From the Security Hub console, you’ll choose a custom action for a finding. Each custom action is then emitted as a CloudWatch Event.
  3. The CloudWatch Event rule triggers a Lambda function. This function is mapped to a custom action based on the custom action’s ARN.
  4. Dependent on the particular rule, the Lambda function that is invoked will perform a remediation action on your behalf.

For the purpose of this blog post, I’ll refer to the end-to-end combination of a custom action, a CloudWatch Event rule, a Lambda function, plus any supporting services needed to perform a specific action as a “playbook.” To demonstrate how a remediation solution works end-to-end, I’ll show you how to build your first playbook manually. You’ll deploy the remainder of the playbooks via CloudFormation.

I’ll also show you how to modify four of the playbooks (three of which are appended by an asterisk), as they use AWS Lambda environment variables to perform their actions, we’ll walk through populating these later.

Based on feedback from Security Hub customers, the following controls from the CIS AWS Foundations Benchmark will be supported by this blog post:

  • 1.3 – “Ensure credentials unused for 90 days or greater are disabled”
  • 1.4 – “Ensure access keys are rotated every 90 days or less”
  • 1.5 – “Ensure IAM password policy requires at least one uppercase letter”
  • 1.6 – “Ensure IAM password policy requires at least one lowercase letter”
  • 1.7 – “Ensure IAM password policy requires at least one symbol”
  • 1.8 – “Ensure IAM password policy requires at least one number”
  • 1.9 – “Ensure IAM password policy requires a minimum length of 14 or greater”
  • 1.10 – “Ensure IAM password policy prevents password reuse”
  • 1.11 – “Ensure IAM password policy expires passwords within 90 days or less”
  • 2.2 – “Ensure CloudTrail log file validation is enabled”
  • 2.3 – “Ensure the S3 bucket CloudTrail logs to is not publicly accessible”
  • 2.4 – “Ensure CloudTrail trails are integrated with Amazon CloudWatch Logs”*
  • 2.6 – “Ensure S3 bucket access logging is enabled on the CloudTrail S3 bucket”*
  • 2.7 – “Ensure CloudTrail logs are encrypted at rest using AWS KMS CMKs”
  • 2.8 – “Ensure rotation for customer created CMKs is enabled”
  • 2.9 – “Ensure VPC flow logging is enabled in all VPCs”*
  • 4.1 – “Ensure no security groups allow ingress from 0.0.0.0/0 to port 22”
  • 4.2 – “Ensure no security groups allow ingress from 0.0.0.0/0 to port 3389”
  • 4.3 – “Ensure the default security group of every VPC restricts all traffic”

You’ll also deploy and modify an additional playbook, “Send findings to JIRA.” You can find the high-level description of each playbook in each custom action creation script, as well as in the CloudFormation resource descriptions.

Note: If you want to send Security Hub findings to a security information and event management tool (SIEM) such as Amazon ElasticSearch Service or a third-party solution, you must change the CloudWatch Events event pattern to match all findings and use different targets such as Amazon Kinesis Data Streams to Kinesis Data Firehose to load your SIEM. This process is out of scope for this post.

Prerequisites

Ensure you have Security Hub and AWS Config turned on in your Region. Also, note that the solution in this blog post is meant to support a single account and will not support cross-account remediation as deployed. Refer to the Knowledge Center article How can I configure a Lambda function to assume a role from another AWS account? for basic information on cross-account roles for Lambda.

For the playbook “Apply Security Patches,” your EC2 instances must be managed by Systems Manager. For more information on managed instances, see AWS Systems Manager Managed Instances in the AWS Systems Manager User Guide.

Manually create a remediation playbook

To demonstrate the end-to-end process of building a playbook, I’ll first show you how to create one manually, before you deploy the remaining playbooks via CloudFormation. You’ll build a playbook to remediate Control 2.7 of the AWS Foundations Benchmark, “ensure CloudTrail logs are encrypted at rest using AWS Key Management Service (KMS) Customer Managed Keys (CMK).” Configuring CloudTrail to use KMS encryption (called SSE-KMS) provides additional confidentiality controls on you log data. To access your CloudTrail logs, users must not only have S3 read permissions for the corresponding log bucket, they now must be granted decrypt permissions by the KMS key policy.

Important note: The way this remediation, and all other remediation code is written, you can only target one finding at a time via the Action Menu.

You’ll achieve automated remediation by using a Lambda function to create a new KMS CMK and alias which identifies the non-compliant CloudTrail trail. You’ll then attach a KMS key policy that only allows the AWS account that owns the trail to decrypt the logs by using the IAM condition for StringEquals: kms:CallerAccount. You only need to run this playbook once per non-compliant CloudTrail trail.

To get started, follow these steps:

  1. Navigate to the Security Hub console, select Settings from the navigation pane, then select the Custom Actions tab.
  2. Choose Create custom action and enter values for Action name, Description, and Custom action ID, then choose Create custom action again, as shown in Figure 2.

    For the purpose of this blog post, I’ll refer to my action name as “CIS 2.7 RR” where the “RR” stands for “Response and remediation.”
     

    Figure 2 - Create custom action

    Figure 2 – Create custom action

  3. Copy the Amazon resource number (ARN) down, as you’ll need it in step 11.
  4. Navigate to the Lambda console and select Create function.
  5. Enter a function name, choose Python 3.7 runtime, and under Permissions select Create a new role with basic Lambda permissions. Then choose Create function.
  6. Scroll down to Execution role and select the hyperlink under Existing role. This will open a new tab in the IAM console.
  7. From the IAM console, select Add inline policy, then select the JSON tab, paste in the below IAM policy JSON, and select Review Policy.
    
    {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "kmssid",
            "Action": [
              "kms:CreateAlias",
              "kms:CreateKey",
              "kms:PutKeyPolicy"
            ],
            "Effect": "Allow",
            "Resource": "*"
          },
          {
            "Sid": "cloudtrailsid",
            "Action": [
              "cloudtrail:UpdateTrail"
            ],
            "Effect": "Allow",
            "Resource": "*"
          }
        ]
      }
    

  8. Give the in-line policy a name and select Create policy.
  9. Back in the Lambda console, increase Timeout to 1 minute and Memory to 256MB. Scroll up to Function code, paste in the below code, and select Save.
    
     # Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
     # SPDX-License-Identifier: MIT-0
     #
     # Permission is hereby granted, free of charge, to any person obtaining a copy of this
     # software and associated documentation files (the "Software"), to deal in the Software
     # without restriction, including without limitation the rights to use, copy, modify,
     # merge, publish, distribute, sublicense, and/or sell copies of the Software, and to
     # permit persons to whom the Software is furnished to do so.
     #
     # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
     # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
     # PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
     # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
     # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
     # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    
    import boto3
    import json
    import time
    
    def lambda_handler(event, context):
        # parse non-compliant trail from Security Hub finding
        noncompliantTrail = str(event['detail']['findings'][0]['Resources'][0]['Details']['Other']['name'])
        # parse account ID from Security Hub finding, will be needed for Key Policy
        accountID = str(event['detail']['findings'][0]['AwsAccountId'])
        
        # import boto3 clients for KMS and CloudTrail
        kms = boto3.client('kms')
        cloudtrail = boto3.client('cloudtrail')
    
        # create a new KMS CMK to encrypt the non-compliant trail
        try:
            createKey = kms.create_key(
            Description='Generated by Security Hub to remediate CIS 2.7 Ensure CloudTrail logs are encrypted at rest using KMS CMKs',
            KeyUsage='ENCRYPT_DECRYPT',
            Origin='AWS_KMS'
            )
            # save key id as a variable
            cloudtrailKey = str(createKey['KeyMetadata']['KeyId'])
            print("Created Key" + " " + cloudtrailKey)
        except Exception as e:
            print(e)
            print("KMS CMK creation failed")
            raise
            
        # wait 2 seconds for key creation to propogate
        time.sleep(2)
    
        # attach an alias for easy identification to the key - must always begin with "alias/"
        try:
            createAlias = kms.create_alias(
            AliasName='alias/' + noncompliantTrail + '-CMK',
            TargetKeyId=cloudtrailKey
            )
            print(createAlias)
        except Exception as e:
            print(e)
            print("Failed to create KMS Alias")
            raise
        
        # wait 1 second
        time.sleep(1)
    
        # policy name for PutKeyPolicy is always "default"
        policyName = 'default'
        # set Key Policy as JSON object
        keyPolicy={
            "Version": "2012-10-17",
            "Id": "Key policy created by CloudTrail",
            "Statement": [
                {
                    "Sid": "Enable IAM User Permissions",
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": [ "arn:aws:iam::" + accountID + ":root" ]
                    },
                    "Action": "kms:*",
                    "Resource": "*"
                },
                {
                    "Sid": "Allow CloudTrail to encrypt logs",
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "cloudtrail.amazonaws.com"
                    },
                    "Action": "kms:GenerateDataKey*",
                    "Resource": "*",
                    "Condition": {
                        "StringLike": {
                            "kms:EncryptionContext:aws:cloudtrail:arn": "arn:aws:cloudtrail:*:" + accountID + ":trail/" + noncompliantTrail
                        }
                    }
                },
                {
                    "Sid": "Allow CloudTrail to describe key",
                    "Effect": "Allow",
                    "Principal": {
                        "Service": "cloudtrail.amazonaws.com"
                    },
                    "Action": "kms:DescribeKey",
                    "Resource": "*"
                },
                {
                    "Sid": "Allow principals in the account to decrypt log files",
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "*"
                    },
                    "Action": [
                        "kms:Decrypt",
                        "kms:ReEncryptFrom"
                    ],
                    "Resource": "*",
                    "Condition": {
                        "StringEquals": {
                            "kms:CallerAccount": accountID
                        },
                        "StringLike": {
                            "kms:EncryptionContext:aws:cloudtrail:arn": "arn:aws:cloudtrail:*:" + accountID + ":trail/" + noncompliantTrail
                        }
                    }
                },
                {
                    "Sid": "Allow alias creation during setup",
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "*"
                    },
                    "Action": "kms:CreateAlias",
                    "Resource": "*",
                    "Condition": {
                        "StringEquals": {
                            "kms:CallerAccount": accountID
                        }
                    }
                }
            ]
        }
        # attaches above key policy to key
        try:
            attachKeyPolicy = kms.put_key_policy(
                KeyId=cloudtrailKey,
                Policy=json.dumps(keyPolicy),
                PolicyName=policyName
            )
            print(attachKeyPolicy)
        except Exception as e:
            print(e)
            print("Failed to attach key policy to Key:" + " " + cloudtrailKey)
    
        # update CloudTrail with the new CMK
        try:
            encryptTrail = cloudtrail.update_trail(
            Name=noncompliantTrail,
            KmsKeyId=cloudtrailKey
            )
            print(encryptTrail)
            print("CloudTrail trail" + " " + noncompliantTrail + " " + "has been successfully encrypted!")
        except Exception as e:
            print(e)
            print("Failed to attach KMS CMK to CloudTrail")
    

  10. Navigate to the CloudWatch console, and choose Events, then Create rule.
  11. On the left, next to Event Pattern Preview, select Edit.
  12. Under Event Source, find Build custom event pattern and paste the JSON below into the text box. Replace the ARN under “resources” with the ARN of the custom action you created in step 3.
    
    {
      "source": [
        "aws.securityhub"
      ],
      "detail-type": [
        "Security Hub Findings - Custom Action"
      ],
      "resources": [
        "arn:aws:securityhub:us-west-2:123456789012:action/custom/test-action1"
      ]
    }
    

  13. On the same screen, under Targets on the right, select add Target, choose the Lambda function you created in step 4, then select Configure details.
  14. On the next screen, enter values for Name and Description, then select Create rule.
  15. Back in the Security Hub console, select Compliance standards from the menu on the left, then select View results to see the CIS AWS Benchmarks.
  16. Find rule 2.7 Ensure CloudTrail logs are encrypted at rest using KMS CMKs and select the hyperlink to see all findings related to that control.
  17. Select a finding that is FAILED, with a Resource type that reads AwsCloudTrailTrail.
  18. From the top right, select the Actions dropdown menu, then select CIS 2.7 RR (or whatever you named this action in step 2), as shown in Figure 3.

    Selecting this action will execute the rule you created in step 13, which will invoke the Lambda function you created in step 9.
     

    Figure 3 - Security Hub Custom Actions

    Figure 3 – Security Hub Custom Actions

  19. Navigate to the CloudTrail console to ensure your CloudTrail trail is updated with SSE-KMS.

This creation flow via the console is universal for all playbook development with Security Hub. You’ll use the Actions menu in the same way to trigger the playbooks you deploy via CloudFormation in the next section of this walkthrough.

Note: To monitor the actions that are taken by the playbooks’ Lambda functions, refer to the functions’ logs. Both success and error messages will appear to help you diagnose as needed.

Deploy remediation playbooks via CloudFormation

Download the CloudFormation template from GitHub and create a CloudFormation stack. For more information about how to create a CloudFormation stack, see Getting Started with AWS CloudFormation in the AWS CloudFormation User Guide.

After your stack has finished deploying, navigate to the Resources tab and select the hyperlink for each resource to be taken to their respective consoles. The logical ID for each resource will be prepended by the action the resource corresponds to. For example, in figure 4, CIS13RRCWEPermissions denotes CloudWatch Event permissions for AWS CIS Benchmark Control 1.3. This logical ID structure is used throughout the template.
 

Figure 4 - CloudFormation Resources

Figure 4 – CloudFormation Resources

Next, I’ll show you how to modify the Lambda functions associated with the following playbooks: “Send findings to JIRA,” CIS 2.4, CIS 2.6, and CIS 2.9.

Playbook modification: send findings to JIRA

Note: If you don’t currently have a JIRA Software Data Center deployment set up in your account, you can deploy one with a free evaluation period by following this Quick Start. If you are not interested in using JIRA or you use a different issue management tool, you can skip this section.

This playbook works by using the associated Lambda function to execute a Systems Manager automation document called AWS-CreateJiraIssue.

This Systems Manager document will in turn deploy a CloudFormation stack and create a custom Lambda function to map environmental variables (listed below) into JIRA via your playbook’s Lambda function.

Once complete, the CloudFormation stack will self-delete and the automation will be complete. To see this flow, refer to figure 5, below:
 

Figure 5 - Send to JIRA Architecture Diagram

Figure 5 – Send to JIRA Architecture Diagram

  1. A finding is selected and the custom action “Create JIRA Issue” is invoked, which triggers a CloudWatch Event.
  2. The CloudWatch Event rule will trigger a Lambda function.
  3. Lambda will invoke the AWS-CreateJiraIssue document via Systems Manager Automation.
  4. Systems Manager Automation will pass your Lambda environmental variables and Security Hub finding as document parameters.
  5. The document will create a CloudFormation Stack that contains another custom Lambda to invoke JIRA APIs for creating issues.
  6. The document’s Lambda function uses your parameters to create an issue in JIRA.
  7. JIRA will send back a response to note failure or success, and the CloudFormation stack will self-delete.

To get started, navigate to the Lambda console and find the function named SendToJIRA, then scroll down to Environmental variables: You should see 4, with the text “placeholder” next to each. The following steps walk you through how to populate them:

  1. Fill out the JIRA_API_PARAMETER field:
    1. Refer to Atlassian’s instructions to generate a JIRA API token
    2. Follow the Systems Manager Parameter Store walkthrough to create a parameter.
      1. In Step 6 of the linked instructions, choose Secure String and choose the default KMS key.
      2. In Step 7 of the linked instructions, paste in your newly generated JIRA API token.
    3. Copy the name of the parameter and paste it as the value of the JIRA_API_PARAMETER field.
  2. Fill out the JIRA_PROJECT field by pasting in your JIRA Software project’s Project Key.
  3. Fill out the JIRA_SECURITY_ISSUE_USER field:
    1. This value maps to a user in JIRA Software; refer to Atlassian’s instructions for how to create a user. Use the API key you generated in step 1.A as this user’s password.
    2. Paste the user name into the JIRA_SECURITY_ISSUE_USER field.
  4. Fill out the JIRA_URL field:
    1. From JIRA Software navigate to the System sub-menu of JIRA Administration and look for the Base URL field, underneath Settings – General Settings (see figure 6).
    2. Copy the Base URL value and paste it into the JIRA_URL field.
       
      Figure 6 - Base URL in JIRA

      Figure 6 – Base URL in JIRA

  5. When finished, select Save at the top of the Lambda console, then navigate to the Security Hub console and select Findings from the left-hand menu.
  6. Select any finding, and then, from the Actions dropdown in the top right, select Create JIRA Issue.
  7. Navigate to your JIRA instance and wait a few minutes for the new issue to appear, as shown in Figure 7.
     
    Figure 7 - Security Hub Finding in JIRA

    Figure 7 – Security Hub Finding in JIRA

    Note: If your issue hasn’t populated in JIRA after a few minutes, refer to the Systems Manager Automation console or CloudFormation console to find the stack that was created by Systems Manager and refer to the failure messages to troubleshoot further.

CIS 2.4 response & remediation playbook modification

CIS Control 2.4 is “Ensure CloudTrail trails are integrated with Amazon CloudWatch Logs.” The intent of this recommendation is to ensure that API activity recorded by CloudTrail is available to query in near real-time for the purpose of troubleshooting or security incident investigation with CloudWatch.

To send your CloudTrail logs to CloudWatch, the Lambda function for this playbook will create a brand new CloudWatch Logs group that has the name of the non-compliant CloudTrail trail in it for easy identification and then update your non-compliant CloudTrail trail to send its logs to the newly created log group.

To accomplish this, CloudTrail needs an IAM role and permissions to be allowed to publish logs to CloudWatch. To avoid creating multiple new IAM roles and policies via Lambda, you’ll populate the ARN of this IAM role in the Lambda environmental variables for this playbook.

Note: If you don’t currently have an IAM role for CloudTrail, follow these instructions from the CloudTrail user guide to create one.

To update this playbook:

  1. Navigate to the Lambda console and find the function named CIS_2-4_RR, then scroll down to Environmental variables.
  2. Find the variable CLOUDTRAIL_CW_LOGGING_ROLE_ARN with text that reads “placeholder” as the value.
  3. Paste the ARN of the IAM role into the field for this value, then select Save.

CIS 2.6 response & remediation playbook modification

CIS Control 2.6 is “Ensure S3 bucket access logging is enabled on the CloudTrail S3 bucket.” An access log record contains details about the request, such as the request type, the resources specified in the request worked, and the time and date the request was processed. AWS recommends that you enable bucket access logging on the CloudTrail S3 bucket. By enabling S3 bucket logging on target S3 buckets, you can capture all the events that might affect objects in the bucket. Configuring logs to be placed in a separate bucket enables centralized collection of access log information, which can be useful in security and incident response workflows.

Note: Security Hub supports CIS AWS Foundations controls only on resources in the same Region and owned by the same account as the one in which Security Hub is enabled and being used. For example, if you’re using Security Hub in the us-east-2 Region, and you’re storing CloudTrail logs in a bucket in the us-west-2 Region, Security Hub cannot find the bucket in the us-west-2 Region. The control returns a warning that the resource cannot be located. Similarly, if you’re aggregating logs from multiple accounts into a single bucket, the CIS control fails for all accounts except the account that owns the bucket.

To ensure the S3 bucket that contains your CloudTrail logs has access logging enabled, the Lambda function for this playbook invokes the Systems Manager document AWS-ConfigureS3BucketLogging. This document will enable access logging for that bucket. To avoid statically populating your S3 access logging bucket in the Lambda function’s code, you’ll pass that value in via an environmental variable.

Note: If you do not currently have an S3 bucket configured to receive access logs, follow the directions from the S3 user guide to create one.

To update this playbook:

  1. Navigate to the Lambda console and find the function named CIS_2-6_RR, then scroll down to Environmental variables.
  2. Find the variable ACCESS_LOGGING_BUCKET with text that reads ‘placeholder’ as the value.
  3. Paste the name of the S3 bucket that will receive the access logs into the field for this value and select Save.

CIS 2.9 response & remediation playbook modification

CIS Control 2.9 is “Ensure VPC flow logging is enabled in all VPCs.” The Amazon Virtual Private Cloud (VPC) flow logs feature enables you to capture information about the IP traffic going to and from network interfaces in your VPC. After you’ve created a flow log, you can view and retrieve its data in CloudWatch Logs. AWS recommends that you enable flow logging for packet rejects for VPCs. Flow logs provide visibility into network traffic that traverses the VPC and can detect anomalous traffic or provide insight into your security workflow.

To enable VPC flow logging for rejected packets, the Lambda function for this playbook will create a new CloudWatch Logs group. For easy identification, the name of the group will include the non-compliant VPC name. The Lambda function will programmatically update your VPC to enable flow logs to be sent to the newly created log group.

Similar to CloudTrail logging, VPC flow log need an IAM role and permissions to be allowed to publish logs to CloudWatch. To avoid creating multiple new IAM roles and policies via Lambda, you’ll populate the ARN of this IAM role in the Lambda environmental variables for this playbook.

Note: If you don’t currently have an IAM role that VPC flow logs can use to deliver logs to CloudWatch, follow the directions from the VPC user guide to create one.

To update this playbook:

  1. Navigate to the Lambda console and find the function named CIS_2-9_RR, then scroll down to Environmental variables.
  2. Find the variable flowLogRoleARN with text that reads ‘placeholder’ as the value.
  3. Paste in the ARN of the VPC flow logs IAM role the field for this value, and select Save.

Conclusion

In this blog post, I showed you how to create, deploy, and execute response and remediation playbooks for Security Hub. By combining custom actions, CloudWatch Event rules, and Lambda functions, you can quickly remediate non-compliant resources. I also showed you how to take pre-defined actions such as sending findings to JIRA, and how to deploy an additional seven playbooks via CloudFormation.

You can create playbooks that take other actions, such as updating Network Access Control Lists to help block malicious traffic from a TOR exit node via the UnauthorizedAccess:EC2/TorIPCaller finding from GuardDuty. Using playbooks with Security Hub is one more way to build toward the security and compliance of your AWS resources.

To avoid incurring additional charges from AWS resources, delete the CloudFormation stack you deployed as well as the resources you manually created for the CIS 2.7 response and remediation playbook.

If you have feedback about this blog post, submit comments in the Comments section below. If you have questions about this blog post, start a new thread on the Security Hub forum or contact AWS Support.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Jonathon Rau

Jonathan Rau

Jonathan is the Senior TPM for AWS Security Hub. He holds an AWS Certified Specialty-Security certification and is extremely passionate about cyber security, data privacy, and new emerging technologies, such as blockchain. He devotes personal time into research and advocacy about those same topics.

How to import AWS Config rules evaluations as findings in Security Hub

Post Syndicated from Omar Kahil original https://aws.amazon.com/blogs/security/how-to-import-aws-config-rules-evaluations-findings-security-hub/

In June at re:Inforce 2019, AWS announced the general availability of AWS Security Hub, a security service that enables customers to centrally view and manage compliance checks and security findings across their AWS accounts. AWS Security Hub imports security findings from AWS Guard Duty, Amazon Inspector, Amazon Macie, and over 30 AWS partner security solutions.

By default, when Security Hub is enabled, it deploys the CIS AWS Foundations standard in your account. CIS AWS Foundations are a set of security configuration best practices for hardening AWS accounts. In order for Security Hub to successfully run compliance checks against the rules included in the CIS AWS Foundations standard, AWS Config must be enabled in the same account where you enabled AWS Security Hub.

If you had previously created your own AWS Config rules before enabling AWS Security Hub, you will need to import them into AWS Security Hub. This will enable your security teams to have all compliance findings across accounts show up in Security Hub, along with the security findings from services like AWS Guard Duty, Amazon Inspector and Amazon Macie. In this post, we’ll show you how to set this up.

Solution Overview

You’ll use an AWS CloudFormation template to deploy the following components, so that you can import the findings of AWS Config rules into AWS Security Hub. Below are the main resources created by this template:

  • An AWS Lambda function with the necessary IAM role and IAM policies:

    Config-SecHub-Lambda

    This Lambda function will execute immediately after the CloudFormation stack creation. It will parse and import AWS Config Rule findings into Security Hub.

  • A CloudWatch Event rule with the necessary permission to invoke the AWS Lambda function:

    Config-Sechub-CW-Rule

    This CloudWatch Event rule will match the compliance change event from AWS Config and will route it to the Lambda function for processing.

Here’s the high level setup of the AWS services that will be created from the CloudFormation stack deployment:

Figure 1: Architecture diagram

Figure 1: Architecture diagram

  1. A new AWS Config rule is deployed in the account after you enable AWS Security Hub.
  2. The AWS Config rule reacts to resource configuration and compliance changes and send these change items to AWS CloudWatch.
  3. When AWS CloudWatch receives the compliance change, a CloudWatch event rule triggers the AWS Lambda function.
  4. The Lambda function parses the event received from CloudWatch and imports it into Security Hub as a finding.

Deploy the solution

  1. Log in to the AWS Management Console and select the us-east-1 Region (N.Virginia) for this sample deployment.
  2. Make sure that AWS Config and AWS Security Hub are enabled in the region where you want to deploy the template.
  3. Set up the sample deployment by selecting the Launch Stack button below.

The solution must be launched in the region where AWS Config and AWS Security Hub are enabled, and can be deployed in any region that support the resources that will be created.

Select this image to open a link that starts building the CloudFormation stack

Follow these steps to execute the stack:

  1. Leave the default location for the template and select Next.
  2. On the Specify stack details page, add your preferred stack name.
  3. On the Configure Stack options page, select the Next button.
  4. On the Review page, select the check box, the select the Create Stack button.
  5. Stack creation will take between 5 – 10 minutes. After the stack is created successfully, select the Resources tab to see the deployed resources.

    Figure 2: AWS CloudFormation Resources tab

    Figure 2: Select the Resources tab to see deployed resources

At this point, you’ve successfully created the following resources:

  • A Lambda function that will integrate AWS Config and Security Hub.
  • A Lambda execution role used by the Lambda function to call other services.
  • A CloudWatch Events rule that will trigger the Lambda function based on AWS Config compliance change events.
  • A service permission to allow Amazon CloudWatch to invoke the Lambda function.

Verify the solution

To be certain that everything is set up properly, look at the Lambda code that integrates AWS Config with AWS Security Hub by following the steps below from the console:

  1. Navigate to the AWS Lambda console.
  2. From the list of Lambda functions, open the function with the name Config-SecHub-Lambda to check that the function has been deployed successfully.

    Figure 3: AWS Lambda console

    Figure 3: Config-SecHub-Lambda

You can also verify that the CloudWatch Event rule has deployed successfully:

  1. Go to the Amazon CloudWatch service console.
  2. Under Events, select Rules. You should see a rule with the name Config-Sechub-CW-Rule.
     
    Figure 4: Look for Config-Sechub-CW-Rule

    Figure 4: Config-Sechub-CW-Rule

Test the solution

  1. Go to the AWS Config console.
  2. On the left, select Rules, then Add rule.
  3. In the search filter, enter ec2 as an input, then select the desired-instance-type rule to test.

    Figure 5: AWS Config "add rule" page

    Figure 5: AWS Config “add rule” page

  4. In the rule parameters, enter t2.micro as the value. Any other instance type launched in this account will violate this rule and be noncompliant. Then, select Save.
  5. Navigate to Amazon EC2 console, and under Create Instance, select Launch.
  6. Select any Amazon Linux AMI.

    Figure 6: Select an Amazon Linux AMI

    Figure 6: Select an Amazon Linux AMI

  7. To test the rule, select any instance type except t2.micro. In our example, we’ve selected t2.nano.

    Figure 7: Choose your instance type

    Figure 7: Choose your instance type

  8. Because you’re just testing the rule, select Review and Launch. Then choose Proceed without a key pair, check the “I acknowledge…” box, and select Launch Instances.
     
    Figure 8: Select "Launch Instances"

    Figure 8: Select “Launch Instances”

    You’ll need to wait few minutes until the instance is launched. You can check the EC2 console to view the instance status.

  9. Once the instance has launched, go back to the AWS Config console, and from the left-hand menu, select Rules again.
  10. On the Rules page, select the Noncompliant filter from the drop-down menu to view the rule you’ve created to check instance types.

    Figure 9: Select the "Noncompliant" filter on the AWS Config Rules page

    Figure 9: Select the “Noncompliant” filter on the AWS Config Rules page

  11. Under Rule name, look for your desired-instance-type rule. It will be followed by your number of noncompliant resources. In this case, there should only be 1 noncompliant resource that matches the EC2 instance you launched above. You can select the rule name to view more details about the noncompliant resource.

    Figure 10: Look for the desired-instance-type rule

    Figure 10: Look for the desired-instance-type rule

  12. Go to the AWS Security Hub console. On the left, select Findings.
  13. Select the search filter, and from the drop-down menu select Title as the filter, then desired-instance-type as the value. Then select Apply.

    Figure 11: Filter by title

    Figure 11: Filter by title

  14. You should now see your AWS Config rule under Findings in AWS Security Hub.

    Figure 12: Look for your Config rule under "Findings"

    Figure 12: Look for your Config rule under “Findings”

  15. After you’ve verified that the Lambda function is working, terminate the EC2 instance and delete the Config rule.

What’s happening in the background?

When you deploy the CloudFormation stack, a Lambda function is created along with other resources. This function integrates AWS Config with AWS Security Hub. Every time a new config rule is added, and as soon its compliance status is changed, the AWS CloudWatch Event rule will trigger the Lambda function based on the compliance event. Through this function, the CloudWatch Event is parsed to get the required data for ingestion, then the data is mapped and converted into AWS Security Finding format. After this, the BatchImportFindings API operation is used to import the findings into AWS Security Hub. You can also find the CloudFormation template and the code files in the linked GitHub repository.

Conclusion

In this blog post, we’ve shown you how to import an AWS Config rule to AWS Security Hub, so that your AWS Config rules show up alongside your other security findings. This allows you to more easily use AWS Config rules in your journey toward continuous compliance across your estate of AWS accounts. If you need help in planning, building or optimizing your infrastructure, please reach out to AWS Professional Services. If you have questions about this post, let us know in the Comments section below, or start a new thread on the AWS Security Hub forum.

Kahil author photo

Omar Kahil

Omar is a Professional Services consultant who helps customers adopt DevOps culture and best practices. He also works to simplify the adoption of AWS services by automating and implementing complex solutions.

Shahzad author photo

Muhammad Shahzad

Muhammad is a DevOps consultant in ProServe who enables customers to implement DevOps by explaining principles and integrating best practices in their organizations.