Post Syndicated from Andy Robinson original https://aws.amazon.com/blogs/security/how-to-export-aws-security-hub-findings-to-csv-format/

AWS Security Hub is a central dashboard for security, risk management, and compliance findings from AWS Audit Manager, AWS Firewall Manager, Amazon GuardDuty, IAM Access Analyzer, Amazon Inspector, and many other AWS and third-party services. You can use the insights from Security Hub to get an understanding of your compliance posture across multiple AWS accounts. It is not unusual for a single AWS account to have more than a thousand Security Hub findings. Multi-account and multi-Region environments may have tens or hundreds of thousands of findings. With so many findings, it is important for you to get a summary of the most important ones. Navigating through duplicate findings, false positives, and benign positives can take time.

In this post, we demonstrate how to export those findings to comma separated values (CSV) formatted files in an Amazon Simple Storage Service (Amazon S3) bucket. You can analyze those files by using a spreadsheet, database applications, or other tools. You can use the CSV formatted files to change a set of status and workflow values to align with your organizational requirements, and update many or all findings at once in Security Hub.

The solution described in this post, called CSV Manager for Security Hub, uses an AWS Lambda function to export findings to a CSV object in an S3 bucket, and another Lambda function to update Security Hub findings by modifying selected values in the downloaded CSV file from an S3 bucket. You use an Amazon EventBridge scheduled rule to perform periodic exports (for example, once a week). CSV Manager for Security Hub also has an update function that allows you to update the workflow, customer-specific notation, and other customer-updatable values for many or all findings at once. If you’ve set up a Region aggregator in Security Hub, you should configure the primary CSV Manager for Security Hub stack to export findings only from the aggregator Region. However, you may configure other CSV Manager for Security Hub stacks that export findings from specific Regions or from all applicable Regions in specific accounts. This allows application and account owners to view their own Security Hub findings without having access to other findings for the organization.

How it works

CSV Manager for Security Hub has two main features:

• Export Security Hub findings to a CSV object in an S3 bucket
• Update Security Hub findings from a CSV object in an S3 bucket

Overview of the export function

The overview of the export function CsvExporter is shown in Figure 1.

Figure 1: Architecture diagram of the export function

Figure 1 shows the following numbered steps:

1. In the AWS Management Console, you invoke the CsvExporter Lambda function with a test event.
2. The export function calls the Security Hub GetFindings API action and gets a list of findings to export from Security Hub.
3. The export function converts the most important fields to identify and sort findings to a 37-column CSV format (which includes 12 updatable columns) and writes to an S3 bucket.

Overview of the update function

To update existing Security Hub findings that you previously exported, you can use the update function CsvUpdater to modify the respective rows and columns of the CSV file you exported, as shown in Figure 2. There are 12 modifiable columns out of 37 (any changes to other columns are ignored), which are described in more detail in Step 3: View or update findings in the CSV file later in this post.

Figure 2: Architecture diagram of the update function

Figure 2 shows the following numbered steps:

1. You download the CSV file that the CsvExporter function generated from the S3 bucket and update as needed.
2. You upload the CSV file that contains your updates to the S3 bucket.
3. In the AWS Management Console, you invoke the CsvUpdater Lambda function with a test event containing the URI of the CSV file.
4. CsvUpdater reads the updated CSV file from the S3 bucket.
5. CsvUpdater identifies the minimum set of updates and invokes the Security Hub BatchUpdateFindings API action.

Step 1: Use the CloudFormation template to deploy the solution

You can set up and use CSV Manager for Security Hub by using either AWS CloudFormation or the AWS Cloud Development Kit (AWS CDK).

To deploy the solution (AWS CDK)

You can find the latest code in the aws-security-hub-csv-manager GitHub repository, where you can also contribute to the sample code. The following commands show how to deploy the solution by using the AWS CDK. First, the AWS CDK initializes your environment and uploads the AWS Lambda assets to an S3 bucket. Then, you deploy the solution to your account by using the following commands. Replace <INSERT_AWS_ACCOUNT> with your account number, and replace <INSERT_REGION> with the AWS Region that you want the solution deployed to, for example us-east-1.

cdk bootstrap aws://<INSERT_AWS_ACCOUNT>/<INSERT_REGION>
cdk deploy

To deploy the solution (CloudFormation)

1. Choose the following Launch Stack button to open the AWS CloudFormation console pre-loaded with the template for this solution:

   

2. In the Parameters section, as shown in Figure 3, enter your values.
   

   Figure 3: CloudFormation template variables

   1. For What folder for CSV Manager for Security Hub Lambda code, leave the default Code. For What folder for CSV Manager for Security Hub exports, leave the default Findings.

      These are the folders within the S3 bucket that the CSV Manager for Security Hub CloudFormation template creates to store the Lambda code, as well as where the findings are exported by the Lambda function.

   2. For Frequency, for this solution you can leave the default value cron(0 8 ? * SUN *). This default causes automatic exports to occur every Sunday at 8:00 AM local time using an EventBridge scheduled rule. For more information about how to update this value to meet your needs, see Schedule Expressions for Rules in the Amazon CloudWatch Events User Guide.
   3. The values you enter for the Regions field depend on whether you have configured an aggregation Region in Security Hub.
      • If you have configured an aggregation Region, enter only that Region code, for example eu-north-1, as shown in Figure 3.
      • If you haven’t configured an aggregation Region, enter a comma-separated list of Regions in which you have enabled Security Hub, for example us-east-1, eu-west-1, eu-west-2.
      • If you would like to export findings from all Regions where Security Hub is enabled, leave the Regions field blank. Regions where Security Hub is not enabled will generate a message and will be skipped.
3. Choose Next.

The CloudFormation stack deploys the necessary resources, including an EventBridge scheduling rule, AWS System Managers Automation documents, an S3 bucket, and Lambda functions for exporting and updating Security Hub findings.

After you deploy the CloudFormation stack

After you create the CSV Manager for Security Hub stack, you can do the following:

1. Perform the export function to write some or all Security Hub findings to a CSV file by following the instructions in Step 2: Export Security Hub findings to a CSV file later in this post.
2. Perform a bulk update of Security Hub findings by following the instructions in Step 3: View or update findings in the CSV file later in this post. You can make changes to one or more of the 12 updatable columns of the CSV file, and perform the update function to update some or all Security Hub findings.

Step 2: Export Security Hub findings to a CSV file

You can export Security Hub findings from the AWS Lambda console. To do this, you create a test event and invoke the CsvExporter Lambda function. CsvExporter exports all Security Hub findings from all applicable Regions to a single CSV file in the S3 bucket for CSV Manager for Security Hub.

To export Security Hub findings to a CSV file

1. In the AWS Lambda console, find the CsvExporter Lambda function and select it.
2. On the Code tab, choose the down arrow at the right of the Test button, as shown in Figure 4, and select Configure test event.
   

   Figure 4: The down arrow at the right of the Test button

3. To create an empty test event, on the Configure test event page, do the following:
   1. Choose Create a new event.
   2. Enter an event name; in this example we used testEvent.
   3. For Template, leave the default hello-world.
   4. For Event JSON, enter the JSON object {} as shown in Figure 5.
   

   Figure 5: Creating an empty test event

4. Choose Save to save the empty test event.
5. To invoke the Lambda function, choose the Test button, as shown in Figure 6.
   

   Figure 6: Test button to invoke the Lambda function

6. On the Execution Results tab, note the following details, which you will need for the next step.

   {
   "message": "Export succeeded", 
   "bucket": DOC-EXAMPLE-BUCKET,
   "exportKey”: DOC-EXAMPLE-OBJECT,
   "resultCode": 200
   }

7. Locate the CSV object that matches the value of “exportKey” (in this example, DOC-EXAMPLE-OBJECT) in the S3 bucket that matches the value of “bucket” (in this example, DOC-EXAMPLE-BUCKET).

Now you can view or update the findings in the CSV file, as described in the next section.

Step 3: (Optional) Using filters to limit CSV results

In your test event, you can specify any filter that is accepted by the GetFindings API action. You do this by adding a filter key to your test event. The filter key can either contain the word HighActive (which is a predefined filter configured as a default for selecting active high-severity and critical findings, as shown in Figure 8), or a JSON filter object.

Figure 8 depicts an example JSON filter that performs the same filtering as the HighActive predefined filter.

To use filters to limit CSV results

1. In the AWS Lambda console, find the CsvExporter Lambda function and select it.
2. On the Code tab, choose the down arrow at the right of the Test button, as shown in Figure 7, and select Configure test event.
   

   Figure 7: The down arrow at the right of the Test button

3. To create a test event containing a filter, on the Configure test event page, do the following:
   1. Choose Create a new event.
   2. Enter an event name; in this example we used filterEvent.
   3. For Template, select testEvent,
   4. For Event JSON, enter the following JSON object, as shown in Figure 8.

      {
         "SeverityLabel":[
            {
               "Value":"CRITICAL",
               "Comparison":"EQUALS"
            },
            {
               "Value":"HIGH",
               "Comparison":"EQUALS"
            }
         ],
         "RecordState":[
            {
               "Comparison":"EQUALS",
               "Value":"ACTIVE"
            }
         ]
      }

      

      Figure 8: Test button to invoke the Lambda function

   5. Choose Save.
4. To invoke the Lambda function, choose the Test button as shown in Figure 9.
   

   Figure 9: Test button to invoke the Lambda function

5. On the Execution Results tab, note the following details, which you will need for the next step.

   {
   "message": "Export succeeded", 
   "bucket": DOC-EXAMPLE-BUCKET,
   "exportKey": DOC-EXAMPLE-OBJECT,
   "resultCode": 200
   }

6. Locate the CSV object that matches the value of “exportKey” (in this example, DOC-EXAMPLE-OBJECT) in the S3 bucket that matches the value of “bucket” (in this example, DOC-EXAMPLE-BUCKET).

The results in this CSV file should be a filtered set of Security Hub findings according to the filter you specified above. You can now proceed to step 4 if you want to view or update findings.

Step 4: View or update findings in the CSV file

You can use any program that allows you to view or edit CSV files, such as Microsoft Excel. The first row in the CSV file are the column names. These column names correspond to fields in the JSON objects that are returned by the GetFindings API action.

Warning: Do not modify the first two columns, Id (column A) or ProductArn (column B). If you modify these columns, Security Hub will not be able to locate the finding to update, and any other changes to that finding will be discarded.

You can locally modify any of the columns in the CSV file, but only 12 columns out of 37 columns will actually be updated if you use CsvUpdater to update Security Hub findings. The following are the 12 columns you can update. These correspond to columns C through N in the CSV file.

* These columns are stored inside the UserDefinedFields field of the updated findings. The column names imply a certain kind of information, but you can put any information you wish.

** These columns are stored inside the Severity field of the updated findings. These values have a fixed format and will be rejected if they do not meet that format.

Columns with fixed text values (L, M, N) in the previous table can be specified in mixed case and without underscores—they will be converted to all uppercase and underscores added in the CsvUpdater Lambda function. For example, “false positive” will be converted to “FALSE_POSITIVE”.

Step 5: Create a test event and update Security Hub by using the CSV file

If you want to update Security Hub findings, make your changes to columns C through N as described in the previous table. After you make your changes in the CSV file, you can update the findings in Security Hub by using the CSV file and the CsvUpdater Lambda function.

Use the following procedure to create a test event and run the CsvUpdater Lambda function.

To create a test event and run the CsvUpdater Lambda function

1. In the AWS Lambda console, find the CsvUpdater Lambda function and select it.
2. On the Code tab, choose the down arrow to the right of the Test button, as shown in Figure 10, and select Configure test event.
   

   Figure 10: The down arrow to the right of the Test button

3. To create a test event as shown in Figure 11, on the Configure test event page, do the following:
   1. Choose Create a new event.
   2. Enter an event name; in this example we used testEvent.
   3. For Template, leave the default hello-world.
   4. For Event JSON, enter the following:

      {
      "input": <s3ObjectUri>,
      "primaryRegion": <aggregationRegionName>
      }

      Replace <s3ObjectUri> with the full URI of the S3 object where the updated CSV file is located.

      Replace <aggregationRegionName> with your Security Hub aggregation Region, or the primary Region in which you initially enabled Security Hub.

      

      Figure 11: Create and save a test event for the CsvUpdater Lambda function

4. Choose Save.
5. Choose the Test button, as shown in Figure 12, to invoke the Lambda function.
   

   Figure 12: Test button to invoke the Lambda function

6. To verify that the Lambda function ran successfully, on the Execution Results tab, review the results for “message”: “Success”, as shown in the following example. Note that the results may be thousands of lines long.

   {
   "message": "Success",
   "details": {
   "processed": [{"Id": arn:aws:securityhub:us-east-1: 111122223333:subscription/cis-aws-foundations-benchmark/v/1.2.0/1.7/finding/6d543b22-6a3d-405c-ae7f-224469bde7d2, "ProductArn": arn:aws:securityhub:us-east-1::product/aws/securityhub}, … ],
   "unprocessed": [],
   "message": "Updated succeeded",
   "success": true
   },
   "input": s3://DOC-EXAMPLE-BUCKET/DOC-EXAMPLE-OBJECT,
   "resultCode": 200
   }

   The processed array lists every successfully updated finding by Id and ProductArn.

   If any of the findings were not successfully updated, their Id and ProductArn appear in the unprocessed array. In the previous example, no findings were unprocessed.

   The value s3://DOC-EXAMPLE-BUCKET/DOC-EXAMPLE-OBJECT is the URI of the S3 object from which your updates were read.

Cleaning up

To avoid incurring future charges, first delete the CloudFormation stack that you deployed in Step 1: Use the CloudFormation template to deploy the solution. Next, you need to manually delete the S3 bucket deployed with the stack. For instructions, see Deleting a bucket in the Amazon Simple Storage Service User Guide.

Conclusion

In this post, we showed you how you can export Security Hub findings to a CSV file in an S3 bucket and update the exported findings by using CSV Manager for Security Hub. We showed you how you can automate this process by using AWS Lambda, Amazon S3, and AWS Systems Manager. Full documentation for CSV Manager for Security Hub is available in the aws-security-hub-csv-manager GitHub repository. You can also investigate other ways to manage Security Hub findings by checking out our blog posts about Security Hub integration with Amazon OpenSearch Service, Amazon QuickSight, Slack, PagerDuty, Jira, or ServiceNow.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the Security Hub re:Post. To learn more or get started, visit AWS Security Hub.

Want more AWS Security news? Follow us on Twitter.