All posts by BK Das

Use the Snyk CLI to scan Python packages using AWS CodeCommit, AWS CodePipeline, and AWS CodeBuild

Post Syndicated from BK Das original https://aws.amazon.com/blogs/devops/snyk-cli-scan-python-codecommit-codepipeline-codebuild/

One of the primary advantages of working in the cloud is achieving agility in product development. You can adopt practices like continuous integration and continuous delivery (CI/CD) and GitOps to increase your ability to release code at quicker iterations. Development models like these demand agility from security teams as well. This means your security team has to provide the tooling and visibility to developers for them to fix security vulnerabilities as quickly as possible.

Vulnerabilities in cloud-native applications can be roughly classified into infrastructure misconfigurations and application vulnerabilities. In this post, we focus on enabling developers to scan vulnerable data around Python open-source packages using the Snyk Command Line Interface (CLI).

The world of package dependencies

Traditionally, code scanning is performed by the security team; they either ship the code to the scanning instance, or in some cases ship it to the vendor for vulnerability scanning. After the vendor finishes the scan, the results are provided to the security team and forwarded to the developer. The end-to-end process of organizing the repositories, sending the code to security team for scanning, getting results back, and remediating them is counterproductive to the agility of working in the cloud.

Let’s take an example of package A, which uses package B and C. To scan package A, you scan package B and C as well. Similar to package A having dependencies on B and C, packages B and C can have their individual dependencies too. So the dependencies for each package get complex and cumbersome to scan over time. The ideal method is to scan all the dependencies in one go, without having manual intervention to understand the dependencies between packages.

Building on the foundation of GitOps and Gitflow

GitOps was introduced in 2017 by Weaveworks as a DevOps model to implement continuous deployment for cloud-native applications. It focuses on the developer ability to ship code faster. Because security is a non-negotiable piece of any application, this solution includes security as part of the deployment process. We define the Snyk scanner as declarative and immutable AWS Cloud Development Kit (AWS CDK) code, which instructs new Python code committed to the repository to be scanned.

Another continuous delivery practice that we base this solution on is Gitflow. Gitflow is a strict branching model that enables project release by enforcing a framework for managing Git projects. As a brief introduction on Gitflow, typically you have a main branch, which is the code sent to production, and you have a development branch where new code is committed. After the code in development branch passes all tests, it’s merged to the main branch, thereby becoming the code in production. In this solution, we aim to provide this scanning capability in all your branches, providing security observability through your entire Gitflow.

AWS services used in this solution

We use the following AWS services as part of this solution:

  • AWS CDK – The AWS CDK is an open-source software development framework to define your cloud application resources using familiar programming languages. In this solution, we use Python to write our AWS CDK code.
  • AWS CodeBuild – CodeBuild is a fully managed build service in the cloud. CodeBuild compiles your source code, runs unit tests, and produces artifacts that are ready to deploy. CodeBuild eliminates the need to provision, manage, and scale your own build servers.
  • AWS CodeCommit – CodeCommit is a fully managed source control service that hosts secure Git-based repositories. It makes it easy for teams to collaborate on code in a secure and highly scalable ecosystem. CodeCommit eliminates the need to operate your own source control system or worry about scaling its infrastructure. You can use CodeCommit to securely store anything from source code to binaries, and it works seamlessly with your existing Git tools.
  • AWS CodePipeline – CodePipeline is a continuous delivery service you can use to model, visualize, and automate the steps required to release your software. You can quickly model and configure the different stages of a software release process. CodePipeline automates the steps required to release your software changes continuously.
  • Amazon EventBridge – EventBridge rules deliver a near-real-time stream of system events that describe changes in AWS resources. With simple rules that you can quickly set up, you can match events and route them to one or more target functions or streams.
  • AWS Systems Manager Parameter Store – Parameter Store, a capability of AWS Systems Manager, provides secure, hierarchical storage for configuration data management and secrets management. You can store data such as passwords, database strings, Amazon Machine Image (AMI) IDs, and license codes as parameter values.

Prerequisites

Before you get started, make sure you have the following prerequisites:

  • An AWS account (use a Region that supports CodeCommit, CodeBuild, Parameter Store, and CodePipeline)
  • A Snyk account
  • An existing CodeCommit repository you want to test on

Architecture overview

After you complete the steps in this post, you will have a working pipeline that scans your Python code for open-source vulnerabilities.

We use the Snyk CLI, which is available to customers on all plans, including the Free Tier, and provides the ability to programmatically scan repositories for vulnerabilities in open-source dependencies as well as base image recommendations for container images. The following reference architecture represents a general workflow of how Snyk performs the scan in an automated manner. The design uses DevSecOps principles of automation, event-driven triggers, and keeping humans out of the loop for its run.

As developers keep working on their code, they continue to commit their code to the CodeCommit repository. Upon each commit, a CodeCommit API call is generated, which is then captured using the EventBridge rule. You can customize this event rule for a specific event or feature branch you want to trigger the pipeline for.

When the developer commits code to the specified branch, that EventBridge event rule triggers a CodePipeline pipeline. This pipeline has a build stage using CodeBuild. This stage interacts with the Snyk CLI, and uses the token stored in Parameter Store. The Snyk CLI uses this token as authentication and starts scanning the latest code committed to the repository. When the scan is complete, you can review the results on the Snyk console.

This code is built for Python pip packages. You can edit the buildspec.yml to incorporate for any other language that Snyk supports.

The following diagram illustrates our architecture.

snyk architecture codepipeline

Code overview

The code in this post is written using the AWS CDK in Python. If you’re not familiar with the AWS CDK, we recommend reading Getting started with AWS CDK before you customize and deploy the code.

Repository URL: https://github.com/aws-samples/aws-cdk-codecommit-snyk

This AWS CDK construct uses the Snyk CLI within the CodeBuild job in the pipeline to scan the Python packages for open-source package vulnerabilities. The construct uses CodePipeline to create a two-stage pipeline: one source, and one build (the Snyk scan stage). The construct takes the input of the CodeCommit repository you want to scan, the Snyk organization ID, and Snyk auth token.

Resources deployed

This solution deploys the following resources:

For the deployment, we use the AWS CDK construct in the codebase cdk_snyk_construct/cdk_snyk_construct_stack.py in the AWS CDK stack cdk-snyk-stack. The construct requires the following parameters:

  • ARN of the CodeCommit repo you want to scan
  • Name of the repository branch you want to be monitored
  • Parameter Store name of the Snyk organization ID
  • Parameter Store name for the Snyk auth token

Set up the organization ID and auth token before deploying the stack. Because these are confidential and sensitive data, you should deploy them as a separate stack or manual process. In this solution, the parameters have been stored as a SecureString parameter type and encrypted using the AWS-managed KMS key.

You create the organization ID and auth token on the Snyk console. On the Settings page, choose General in the navigation page to add these parameters.

snyk settings console

 

You can retrieve the names of the parameters on the Systems Manager console by navigating to Parameter Store and finding the name on the Overview tab.

SSM Parameter Store

Create a requirements.txt file in the CodeCommit repository

We now create a repository in CodeCommit to store the code. For simplicity, we primarily store the requirements.txt file in our repository. In Python, a requirements file stores the packages that are used. Having clearly defined packages and versions makes it easier for development, especially in virtual environments.

For more information on the requirements file in Python, see Requirement Specifiers.

To create a CodeCommit repository, run the following AWS Command Line Interface (AWS CLI) command in your AWS accounts:

aws codecommit create-repository --repository-name snyk-repo \
--repository-description "Repository for Snyk to scan Python packages"

Now let’s create a branch called main in the repository using the following command:

aws codecommit create-branch --repository-name snyk-repo \
--branch-name main

After you create the repository, commit a file named requirements.txt with the following content. The following packages are pinned to a particular version that they have a vulnerability with. This file is our hypothetical vulnerable set of packages that have been committed into your development code.

PyYAML==5.3.1
Pillow==7.1.2
pylint==2.5.3
urllib3==1.25.8

 

For instructions on committing files in CodeCommit, see Connect to an AWS CodeCommit repository.

When you store the Snyk auth token and organization ID in Parameter Store, note the parameter names—you need to pass them as parameters during the deployment step.

Now clone the CDK code from the GitHub repository with the command below:

git clone https://github.com/aws-samples/aws-cdk-codecommit-snyk.git

After the cloning is complete you should see a directory named aws-cdk-codecommit-snyk on your machine.

When you’re ready to deploy, enter the aws-cdk-codecommit-snyk directory, and run the following command with the appropriate values:

cdk deploy cdk-snyk-stack \
--parameters RepoName=<name-of-codecommit-repo> \
--parameters RepoBranch=<branch-to-be-scanned>  \
--parameters SnykOrgId=<value> \
--parameters SnykAuthToken=<value>

After the stack deployment is complete, you can see a new pipeline in your AWS account, which is configured to be triggered every time a commit occurs on the main branch.

You can view the results of the scan on the Snyk console. After the pipeline runs, log in to snyk.io and you should see a project named as per your repository (see the following screenshot).

snyk dashboard

 

Choose the repo name to get a detailed view of the vulnerabilities found. Depending on what packages you put in your requirements.txt, your report will differ from the following screenshot.

snyk-vuln-details

 

To fix the vulnerability identified, you can change the version of these packages in the requirements.txt file. The edited requirements file should look like the following:

PyYAML==5.4
Pillow==8.2.0
pylint==2.6.1
urllib3==1.25.9

After you update the requirements.txt file in your repository, push your changes back to the CodeCommit repository you created earlier on the main branch. The push starts the pipeline again.

After the commit is performed to the targeted branch, you don’t see the vulnerability reported on the Snyk dashboard because the pinned version 5.4 doesn’t contain that vulnerability.

Clean up

To avoid accruing further cost for the resources deployed in this solution, run cdk destroy to remove all the AWS resources you deployed through CDK.

As the CodeCommit repository was created using AWS CLI, the following command deletes the CodeCommit repository:

aws codecommit delete-repository --repository-name snyk-repo

Conclusion

In this post, we provided a solution so developers can self- remediate vulnerabilities in their code by monitoring it through Snyk. This solution provides observability, agility, and security for your Python application by following DevOps principles.

A similar architecture has been used at NFL to shift-left the security of their code. According to the shift-left design principle, security should be moved closer to the developers to identify and remediate security issues earlier in the development cycle. NFL has implemented a similar architecture which made the total process, from committing code on the branch to remediating 15 times faster than their previous code scanning setup.

Here’s what NFL has to say about their experience:

“NFL used Snyk to scan Python packages for a service launch. Traditionally it would have taken 10days to scan the packages through our existing process but with Snyk we were able to follow DevSecOps principles and get the scans completed, and reviewed within matter of days. This simplified our time to market while maintaining visibility into our security posture.” – Joe Steinke (Director, Data Solution Architect)

How to create auto-suppression rules in AWS Security Hub

Post Syndicated from BK Das original https://aws.amazon.com/blogs/security/how-to-create-auto-suppression-rules-in-aws-security-hub/

AWS Security Hub gives you a comprehensive view of your security alerts and security posture across your AWS accounts. With Security Hub, you have a single place that aggregates, organizes, and prioritizes your security alerts, or findings, from multiple AWS services. Security Hub lets you assign workflow statuses to these findings, which are NEW, NOTIFIED, SUPPRESSED, or RESOLVED. These statuses allow you to categorize which findings are open and need your attention.

In this blog post, we show how you can create automated suppression rules for specific types of findings in AWS Security Hub, such as ones that are an accepted risk by design, or have a compensating control. By automatically suppressing these findings that don’t require follow-up action from your security team, you can concentrate on investigating and remediating findings that are not yet resolved.

As an example of a finding that you may want to suppress, suppose that your development environment doesn’t need to have Amazon Virtual Private Cloud (VPC) Flow Logs enabled because it does not contain any sensitive data (that is, it is an accepted risk). However, your production environment must have VPC Flow Logs enabled. You can use this solution to automatically suppress the development environment findings regarding VPC Flow Logs not being enabled. Then, you can focus on responding to and remediating findings regarding the production environment VPC Flow Logs that are not enabled.

This solution uses an Amazon EventBridge rule to evaluate Security Hub findings based on predefined filters. An AWS Lambda function is the target of the rule, and is triggered to perform the suppression. The Lambda function calls the Security Hub BatchUpdateFindings API action to set the finding of interest to the SUPPRESSED status.

Prerequisites

This solution assumes that you have Security Hub and AWS Config enabled in your administrator and member AWS accounts. AWS Config is required to execute the rules that will generate the findings. You will also need to enable the AWS Foundational Security Best Practices standard, because the examples in this post rely on those findings. You should ensure that you have configured your administrator account to aggregate your Security Hub findings from across your AWS accounts.

Solution overview

In Security Hub, the status of an investigation of a finding is tracked using the workflow status attribute. The workflow status for new findings is initially set to NEW. You can change the workflow status of a finding either by selecting it in the AWS Security Hub console, or by automating the change of workflow status by using AWS CLI or Security Hub API. After the owner of the finding’s resource is notified to take action, you can set the workflow status to NOTIFIED. After a finding is remediated, you can set the workflow status to RESOLVED. If the finding is not a concern for your given environment and does not require any action, then you can set the workflow status to SUPPRESSED.

In this solution, we show you how to automatically set the workflow status to SUPPRESSED for expected findings, by using EventBridge event patterns that trigger on Security Hub findings that match your defined criteria. The event pattern can match on fields of the findings such as account number, AWS Region, and Amazon Resource Names (ARNs). The Lambda function triggers on findings that match all defined criteria, and then sets the workflow status to SUPPRESSED for all matched findings using the BatchUpdateFindings Security Hub API action.

Solution architecture

 

Figure 1: Solution architecture overview

Figure 1: Solution architecture overview

Figure 1 shows the administrator account aggregating the Security Hub findings from the member accounts.

  1. Security Hub generates findings in the member accounts, then forwards the findings to the administrator account to be evaluated.
  2. In the administrator account, Security Hub evaluates every finding (whether generated or forwarded) against EventBridge rules.
  3. If a finding satisfies any of the defined EventBridge rule conditions, EventBridge triggers a Lambda function in the same Region. The EventBridge event bus delivers the finding to the Lambda function.
  4. The Lambda function in the administrator account performs the finding suppression evaluation, and sets the Security Hub workflow status of the finding to SUPPRESSED.

This architecture uses one Lambda function per Region. You can group together multiple suppression rules into the same EventBridge pattern when they apply to the same group of AWS accounts. You can also configure multiple separate EventBridge event patterns when a suppression rule shouldn’t apply to an account.

Implementation

First, we show how to write the EventBridge event pattern. You use the CDK to define the event rule and pattern. The following example code will suppress Security Hub findings that originate in the development accounts for VPC flow logs that aren’t enabled. The solution will filter new findings only.

In the following example, replace <account-id-1> and <account-id-2> with your own information.

event_pattern_obj = events.EventPattern(
            source=["aws.securityhub"],
            detail_type=["Security Hub Findings - Imported"],
            detail= {
                "findings": {
                    "GeneratorId": [
                        "aws-foundational-security-best-practices/v/1.0.0/EC2.6"
                    ],
                    "AwsAccountId": [
                        "<account-id-1>",
                        "<account-id-2>"
                    ],
                    "Workflow": {
                        "Status": [
                            "NEW"
                        ]
                    }
                }
            }
        )

Second, you define the EventBridge rule that will match on the defined pattern.

        vpc_flow_log_dev_account_event_rule = events.Rule(
                    self,
                    'vpc-flow-logs-development-account-eventbridge-rule',
                    description='VPC flow logs in development account finding suppression',
                    rule_name='vpc-flow-logs-development-account-sechub-rule',
                    event_pattern=event_pattern_obj
                    )

Finally, the EventBridge rule triggers the suppression Lambda function.

 vpc_flow_log_dev_account_event_rule.add_target(lambda_targets.LambdaFunction(security_hub_suppression_lambda))

Solution deployment

You can deploy the solution through either the AWS Management Console or the AWS Cloud Development Kit (AWS CDK).

To deploy the solution by using the AWS Management Console

In your security account, launch the template by choosing the following Launch Stack button.
Select the Launch Stack button to launch the template

To deploy the solution by using the AWS CDK

You can find the latest code on GitHub, where you can also contribute to the sample code. The following commands show how to deploy the solution by using the AWS CDK. First, the CDK initializes your environment and uploads the Lambda assets to Amazon Simple Storage Service (Amazon S3). Then, you can deploy the solution to your account. For <account_id>, specify the account number, or comma separated list of account numbers, that you want the suppression rule to apply to.

cdk bootstrap

cdk deploy sechub-finding-suppression --parameters GeneratorIds=<generator_ids> --parameters AccountNumbers=<account_ids>

To test the solution

  1. Create a VPC that does not have flow logs enabled. We have included a test VPC that you can deploy with the following command: 
    cdk deploy vpc-test-suppression

  2. Verify that the Security Hub finding EC2.6 has been suppressed in the parent account and the target account. You might need to wait a few minutes for the AWS Config recorder to detect the newly created resource and then to manually trigger the following AWS Config rule: 
    securityhub-vpc-flow-logs-enabled-*

  3. After verifying the suppression, delete the test VPC you created to test the suppression rule: 
    cdk destroy vpc-test-suppression

Next steps

You can configure EventBridge rules and patterns to suppress all of your findings that are accepted risk, by design, or that have a compensating control. For example, if you are performing IAM authentication by using Amazon RDS Proxy, you could consider suppressing the control [RDS.10] IAM authentication should be configured for RDS instances. You can also consider creating event patterns that filter based on resource tags, such as filtering VPCs based on tags rather than account numbers for [EC2.6] VPC flow logging should be enabled in all VPCs.

Summary

In this blog post, we showed how you can automatically suppress specific findings by using the Security Hub BatchUpdateFindings API action. We showed you how to configure EventBridge patterns and rules in order to trigger a Lambda function that calls this API action to suppress your expected findings. After you follow the steps in this blog post for automatic Security Hub suppression, your console view in Security Hub will only show findings that are not suppressed.

If you have feedback about this post, submit comments in the Comments section below.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Author

BK Das

BK works as a Senior Security Architect with AWS Professional Services. He love to solve security problems for his customers, and help them feel comfortable within AWS. Outside of work, BK loves to play computer games, and go on long drives.

Author

Josh Joy

Josh is a Senior Security Consultant with the AWS Global Security Practice, a part of our Worldwide Professional Services Organization. Josh helps customers improve their security posture as they migrate their most sensitive workloads to AWS. Josh enjoys diving deep and working backwards in order to help customers achieve positive outcomes.

Author

Moumita Saha

Moumita is a Security Consultant with AWS Professional Services working to help enterprise customers secure their workloads in the cloud. She assists customers in secure cloud migration, designing automated solutions to protect against cyber threats in the cloud. She is passionate about cyber security, data privacy, and new, emerging cloud-security technologies.