Tag Archives: Amazon CloudWatch Events

How to Query Personally Identifiable Information with Amazon Macie

Post Syndicated from Chad Woolf original https://aws.amazon.com/blogs/security/how-to-query-personally-identifiable-information-with-amazon-macie/

Amazon Macie logo

In August 2017 at the AWS Summit New York, AWS launched a new security and compliance service called Amazon Macie. Macie uses machine learning to automatically discover, classify, and protect sensitive data in AWS. In this blog post, I demonstrate how you can use Macie to help enable compliance with applicable regulations, starting with data retention.

How to query retained PII with Macie

Data retention and mandatory data deletion are common topics across compliance frameworks, so knowing what is stored and how long it has been or needs to be stored is of critical importance. For example, you can use Macie for Payment Card Industry Data Security Standard (PCI DSS) 3.2, requirement 3, “Protect stored cardholder data,” which mandates a “quarterly process for identifying and securely deleting stored cardholder data that exceeds defined retention.” You also can use Macie for ISO 27017 requirement 12.3.1, which calls for “retention periods for backup data.” In each of these cases, you can use Macie’s built-in queries to identify the age of data in your Amazon S3 buckets and to help meet your compliance needs.

To get started with Macie and run your first queries of personally identifiable information (PII) and sensitive data, follow the initial setup as described in the launch post on the AWS Blog. After you have set up Macie, walk through the following steps to start running queries. Start by focusing on the S3 buckets that you want to inventory and capture important compliance related activity and data.

To start running Macie queries:

  1. In the AWS Management Console, launch the Macie console (you can type macie to find the console).
  2. Click Dashboard in the navigation pane. This shows you an overview of the risk level and data classification type of all inventoried S3 buckets, categorized by date and type.
    Screenshot of "Dashboard" in the navigation pane
  3. Choose S3 objects by PII priority. This dashboard lets you sort by PII priority and PII types.
    Screenshot of "S3 objects by PII priority"
  4. In this case, I want to find information about credit card numbers. I choose the magnifying glass for the type cc_number (note that PII types can be used for custom queries). This view shows the events where PII classified data has been uploaded to S3. When I scroll down, I see the individual files that have been identified.
    Screenshot showing the events where PII classified data has been uploaded to S3
  5. Before looking at the files, I want to continue to build the query by only showing items with high priority. To do so, I choose the row called Object PII Priority and then the magnifying glass icon next to High.
    Screenshot of refining the query for high priority events
  6. To view the results matching these queries, I scroll down and choose any file listed. This shows vital information such as creation date, location, and object access control list (ACL).
  7. The piece I am most interested in this case is the Object PII details line to understand more about what was found in the file. In this case, I see name and credit card information, which is what caused the high priority. Scrolling up again, I also see that the query fields have updated as I interacted with the UI.
    Screenshot showing "Object PII details"

Let’s say that I want to get an alert every time Macie finds new data matching this query. This alert can be used to automate response actions by using AWS Lambda and Amazon CloudWatch Events.

  1. I choose the left green icon called Save query as alert.
    Screenshot of "Save query as alert" button
  2. I can customize the alert and change things like category or severity to fit my needs based on the alert data.
  3. Another way to find the information I am looking for is to run custom queries. To start using custom queries, I choose Research in the navigation pane.
    1. To learn more about custom Macie queries and what you can do on the Research tab, see Using the Macie Research Tab.
  4. I change the type of query I want to run from CloudTrail data to S3 objects in the drop-down list menu.
    Screenshot of choosing "S3 objects" from the drop-down list menu
  5. Because I want PII data, I start typing in the query box, which has an autocomplete feature. I choose the pii_types: query. I can now type the data I want to look for. In this case, I want to see all files matching the credit card filter so I type cc_number and press Enter. The query box now says, pii_types:cc_number. I press Enter again to enable autocomplete, and then I type AND pii_types:email to require both a credit card number and email address in a single object.
    The query looks for all files matching the credit card filter ("cc_number")
  6. I choose the magnifying glass to search and Macie shows me all S3 objects that are tagged as PII of type Credit Cards. I can further specify that I only want to see PII of type Credit Card that are classified as High priority by adding AND and pii_impact:high to the query.
    Screenshot showing narrowing the query results furtherAs before, I can save this new query as an alert by clicking Save query as alert, which will be triggered by data matching the query going forward.

Advanced tip

Try the following advanced queries using Lucene query syntax and save the queries as alerts in Macie.

  • Use a regular-expression based query to search for a minimum of 10 credit card numbers and 10 email addresses in a single object:
    • pii_explain.cc_number:/([1-9][0-9]|[0-9]{3,}) distinct Credit Card Numbers.*/ AND pii_explain.email:/([1-9][0-9]|[0-9]{3,}) distinct Email Addresses.*/
  • Search for objects containing at least one credit card, name, and email address that have an object policy enabling global access (searching for S3 AllUsers or AuthenticatedUsers permissions):
    • (object_acl.Grants.Grantee.URI:”http\://acs.amazonaws.com/groups/global/AllUsers” OR  object_acl.Grants.Grantee.URI:”http\://acs.amazonaws.com/groups/global/AllUsers”) AND (pii_types.cc_number AND pii_types.email AND pii_types.name)

These are two ways to identify and be alerted about PII by using Macie. In a similar way, you can create custom alerts for various AWS CloudTrail events by choosing a different data set on which to run the queries again. In the examples in this post, I identified credit cards stored in plain text (all data in this post is example data only), determined how long they had been stored in S3 by viewing the result details, and set up alerts to notify or trigger actions on new sensitive data being stored. With queries like these, you can build a reliable data validation program.

If you have comments about this post, submit them in the “Comments” section below. If you have questions about how to use Macie, start a new thread on the Macie forum or contact AWS Support.

-Chad

Automating Amazon EBS Snapshot Management with AWS Step Functions and Amazon CloudWatch Events

Post Syndicated from Andy Katz original https://aws.amazon.com/blogs/compute/automating-amazon-ebs-snapshot-management-with-aws-step-functions-and-amazon-cloudwatch-events/

Brittany Doncaster, Solutions Architect

Business continuity is important for building mission-critical workloads on AWS. As an AWS customer, you might define recovery point objectives (RPO) and recovery time objectives (RTO) for different tier applications in your business. After the RPO and RTO requirements are defined, it is up to your architects to determine how to meet those requirements.

You probably store persistent data in Amazon EBS volumes, which live within a single Availability Zone. And, following best practices, you take snapshots of your EBS volumes to back up the data on Amazon S3, which provides 11 9’s of durability. If you are following these best practices, then you’ve probably recognized the need to manage the number of snapshots you keep for a particular EBS volume and delete older, unneeded snapshots. Doing this cleanup helps save on storage costs.

Some customers also have policies stating that backups need to be stored a certain number of miles away as part of a disaster recovery (DR) plan. To meet these requirements, customers copy their EBS snapshots to the DR region. Then, the same snapshot management and cleanup has to also be done in the DR region.

All of this snapshot management logic consists of different components. You would first tag your snapshots so you could manage them. Then, determine how many snapshots you currently have for a particular EBS volume and assess that value against a retention rule. If the number of snapshots was greater than your retention value, then you would clean up old snapshots. And finally, you might copy the latest snapshot to your DR region. All these steps are just an example of a simple snapshot management workflow. But how do you automate something like this in AWS? How do you do it without servers?

One of the most powerful AWS services released in 2016 was Amazon CloudWatch Events. It enables you to build event-driven IT automation, based on events happening within your AWS infrastructure. CloudWatch Events integrates with AWS Lambda to let you execute your custom code when one of those events occurs. However, the actions to take based on those events aren’t always composed of a single Lambda function. Instead, your business logic may consist of multiple steps (like in the case of the example snapshot management flow described earlier). And you may want to run those steps in sequence or in parallel. You may also want to have retry logic or exception handling for each step.

AWS Step Functions serves just this purpose―to help you coordinate your functions and microservices. Step Functions enables you to simplify your effort and pull the error handling, retry logic, and workflow logic out of your Lambda code. Step Functions integrates with Lambda to provide a mechanism for building complex serverless applications. Now, you can kick off a Step Functions state machine based on a CloudWatch event.

In this post, I discuss how you can target Step Functions in a CloudWatch Events rule. This allows you to have event-driven snapshot management based on snapshot completion events firing in CloudWatch Event rules.

As an example of what you could do with Step Functions and CloudWatch Events, we’ve developed a reference architecture that performs management of your EBS snapshots.

Automating EBS Snapshot Management with Step Functions

This architecture assumes that you have already set up CloudWatch Events to create the snapshots on a schedule or that you are using some other means of creating snapshots according to your needs.

This architecture covers the pieces of the workflow that need to happen after a snapshot has been created.

  • It creates a CloudWatch Events rule to invoke a Step Functions state machine execution when an EBS snapshot is created.
  • The state machine then tags the snapshot, cleans up the oldest snapshots if the number of snapshots is greater than the defined number to retain, and copies the snapshot to a DR region.
  • When the DR region snapshot copy is completed, another state machine kicks off in the DR region. The new state machine has a similar flow and uses some of the same Lambda code to clean up the oldest snapshots that are greater than the defined number to retain.
  • Also, both state machines demonstrate how you can use Step Functions to handle errors within your workflow. Any errors that are caught during execution result in the execution of a Lambda function that writes a message to an SNS topic. Therefore, if any errors occur, you can subscribe to the SNS topic and get notified.

The following is an architecture diagram of the reference architecture:

Creating the Lambda functions and Step Functions state machines

First, pull the code from GitHub and use the AWS CLI to create S3 buckets for the Lambda code in the primary and DR regions. For this example, assume that the primary region is us-west-2 and the DR region is us-east-2. Run the following commands, replacing the italicized text in <> with your own unique bucket names.

git clone https://github.com/awslabs/aws-step-functions-ebs-snapshot-mgmt.git

cd aws-step-functions-ebs-snapshot-mgmt/

aws s3 mb s3://<primary region bucket name> --region us-west-2

aws s3 mb s3://<DR region bucket name> --region us-east-2

Next, use the Serverless Application Model (SAM), which uses AWS CloudFormation to deploy the Lambda functions and Step Functions state machines in the primary and DR regions. Replace the italicized text in <> with the S3 bucket names that you created earlier.

aws cloudformation package --template-file PrimaryRegionTemplate.yaml --s3-bucket <primary region bucket name>  --output-template-file tempPrimary.yaml --region us-west-2

aws cloudformation deploy --template-file tempPrimary.yaml --stack-name ebsSnapshotMgmtPrimary --capabilities CAPABILITY_IAM --region us-west-2

aws cloudformation package --template-file DR_RegionTemplate.yaml --s3-bucket <DR region bucket name> --output-template-file tempDR.yaml  --region us-east-2

aws cloudformation deploy --template-file tempDR.yaml --stack-name ebsSnapshotMgmtDR --capabilities CAPABILITY_IAM --region us-east-2

CloudWatch event rule verification

The CloudFormation templates deploy the following resources:

  • The Lambda functions that are coordinated by Step Functions
  • The Step Functions state machine
  • The SNS topic
  • The CloudWatch Events rules that trigger the state machine execution

So, all of the CloudWatch event rules have been created for you by performing the preceding commands. The next section demonstrates how you could create the CloudWatch event rule manually. To jump straight to testing the workflow, see the “Testing in your Account” section. Otherwise, you begin by setting up the CloudWatch event rule in the primary region for the createSnapshot event and also the CloudWatch event rule in the DR region for the copySnapshot command.

First, open the CloudWatch console in the primary region.

Choose Create Rule and create a rule for the createSnapshot command, with your newly created Step Function state machine as the target.

For Event Source, choose Event Pattern and specify the following values:

  • Service Name: EC2
  • Event Type: EBS Snapshot Notification
  • Specific Event: createSnapshot

For Target, choose Step Functions state machine, then choose the state machine created by the CloudFormation commands. Choose Create a new role for this specific resource. Your completed rule should look like the following:

Choose Configure Details and give the rule a name and description.

Choose Create Rule. You now have a CloudWatch Events rule that triggers a Step Functions state machine execution when the EBS snapshot creation is complete.

Now, set up the CloudWatch Events rule in the DR region as well. This looks almost same, but is based off the copySnapshot event instead of createSnapshot.

In the upper right corner in the console, switch to your DR region. Choose CloudWatch, Create Rule.

For Event Source, choose Event Pattern and specify the following values:

  • Service Name: EC2
  • Event Type: EBS Snapshot Notification
  • Specific Event: copySnapshot

For Target, choose Step Functions state machine, then select the state machine created by the CloudFormation commands. Choose Create a new role for this specific resource. Your completed rule should look like in the following:

As in the primary region, choose Configure Details and then give this rule a name and description. Complete the creation of the rule.

Testing in your account

To test this setup, open the EC2 console and choose Volumes. Select a volume to snapshot. Choose Actions, Create Snapshot, and then create a snapshot.

This results in a new execution of your state machine in the primary and DR regions. You can view these executions by going to the Step Functions console and selecting your state machine.

From there, you can see the execution of the state machine.

Primary region state machine:

DR region state machine:

I’ve also provided CloudFormation templates that perform all the earlier setup without using git clone and running the CloudFormation commands. Choose the Launch Stack buttons below to launch the primary and DR region stacks in Dublin and Ohio, respectively. From there, you can pick up at the Testing in Your Account section above to finish the example. All of the code for this example architecture is located in the aws-step-functions-ebs-snapshot-mgmt AWSLabs repo.

Launch EBS Snapshot Management into Ireland with CloudFormation
Primary Region eu-west-1 (Ireland)

Launch EBS Snapshot Management into Ohio with CloudFormation
DR Region us-east-2 (Ohio)

Summary

This reference architecture is just an example of how you can use Step Functions and CloudWatch Events to build event-driven IT automation. The possibilities are endless:

  • Use this pattern to perform other common cleanup type jobs such as managing Amazon RDS snapshots, old versions of Lambda functions, or old Amazon ECR images—all triggered by scheduled events.
  • Use Trusted Advisor events to identify unused EC2 instances or EBS volumes, then coordinate actions on them, such as alerting owners, stopping, or snapshotting.

Happy coding and please let me know what useful state machines you build!

Automate Your IT Operations Using AWS Step Functions and Amazon CloudWatch Events

Post Syndicated from Andy Katz original https://aws.amazon.com/blogs/compute/automate-your-it-operations-using-aws-step-functions-and-amazon-cloudwatch-events/


Rob Percival, Associate Solutions Architect

Are you interested in reducing the operational overhead of your AWS Cloud infrastructure? One way to achieve this is to automate the response to operational events for resources in your AWS account.

Amazon CloudWatch Events provides a near real-time stream of system events that describe the changes and notifications for your AWS resources. From this stream, you can create rules to route specific events to AWS Step Functions, AWS Lambda, and other AWS services for further processing and automated actions.

In this post, learn how you can use Step Functions to orchestrate serverless IT automation workflows in response to CloudWatch events sourced from AWS Health, a service that monitors and generates events for your AWS resources. As a real-world example, I show automating the response to a scenario where an IAM user access key has been exposed.

Serverless workflows with Step Functions and Lambda

Step Functions makes it easy to develop and orchestrate components of operational response automation using visual workflows. Building automation workflows from individual Lambda functions that perform discrete tasks lets you develop, test, and modify the components of your workflow quickly and seamlessly. As serverless services, Step Functions and Lambda also provide the benefits of more productive development, reduced operational overhead, and no costs incurred outside of when the workflows are actively executing.

Example workflow

As an example, this post focuses on automating the response to an event generated by AWS Health when an IAM access key has been publicly exposed on GitHub. This is a diagram of the automation workflow:

AWS proactively monitors popular code repository sites for IAM access keys that have been publicly exposed. Upon detection of an exposed IAM access key, AWS Health generates an AWS_RISK_CREDENTIALS_EXPOSED event in the AWS account related to the exposed key. A configured CloudWatch Events rule detects this event and invokes a Step Functions state machine. The state machine then orchestrates the automated workflow that deletes the exposed IAM access key, summarizes the recent API activity for the exposed key, and sends the summary message to an Amazon SNS topic to notify the subscribers―in that order.

The corresponding Step Functions state machine diagram of this automation workflow can be seen below:

While this particular example focuses on IT automation workflows in response to the AWS_RISK_CREDENTIALS_EXPOSEDevent sourced from AWS Health, it can be generalized to integrate with other events from these services, other event-generating AWS services, and even run on a time-based schedule.

Walkthrough

To follow along, use the code and resources found in the aws-health-tools GitHub repo. The code and resources include an AWS CloudFormation template, in addition to instructions on how to use it.

Launch Stack into N. Virginia with CloudFormation

The Step Functions state machine execution starts with the exposed keys event details in JSON, a sanitized example of which is provided below:

{
    "version": "0",
    "id": "121345678-1234-1234-1234-123456789012",
    "detail-type": "AWS Health Event",
    "source": "aws.health",
    "account": "123456789012",
    "time": "2016-06-05T06:27:57Z",
    "region": "us-east-1",
    "resources": [],
    "detail": {
        "eventArn": "arn:aws:health:us-east-1::event/AWS_RISK_CREDENTIALS_EXPOSED_XXXXXXXXXXXXXXXXX",
        "service": "RISK",
        "eventTypeCode": "AWS_RISK_CREDENTIALS_EXPOSED",
        "eventTypeCategory": "issue",
        "startTime": "Sat, 05 Jun 2016 15:10:09 GMT",
        "eventDescription": [
            {
                "language": "en_US",
                "latestDescription": "A description of the event is provided here"
            }
        ],
        "affectedEntities": [
            {
                "entityValue": "ACCESS_KEY_ID_HERE"
            }
        ]
    }
}

After it’s invoked, the state machine execution proceeds as follows.

Step 1: Delete the exposed IAM access key pair

The first thing you want to do when you determine that an IAM access key has been exposed is to delete the key pair so that it can no longer be used to make API calls. This Step Functions task state deletes the exposed access key pair detailed in the incoming event, and retrieves the IAM user associated with the key to look up API activity for the user in the next step. The user name, access key, and other details about the event are passed to the next step as JSON.

This state contains a powerful error-handling feature offered by Step Functions task states called a catch configuration. Catch configurations allow you to reroute and continue state machine invocation at new states depending on potential errors that occur in your task function. In this case, the catch configuration skips to Step 3. It immediately notifies your security team that errors were raised in the task function of this step (Step 1), when attempting to look up the corresponding IAM user for a key or delete the user’s access key.

Note: Step Functions also offers a retry configuration for when you would rather retry a task function that failed due to error, with the option to specify an increasing time interval between attempts and a maximum number of attempts.

Step 2: Summarize recent API activity for key

After you have deleted the access key pair, you’ll want to have some immediate insight into whether it was used for malicious activity in your account. Another task state, this step uses AWS CloudTrail to look up and summarize the most recent API activity for the IAM user associated with the exposed key. The summary is in the form of counts for each API call made and resource type and name affected. This summary information is then passed to the next step as JSON. This step requires information that you obtained in Step 1. Step Functions ensures the successful completion of Step 1 before moving to Step 2.

Step 3: Notify security

The summary information gathered in the last step can provide immediate insight into any malicious activity on your account made by the exposed key. To determine this and further secure your account if necessary, you must notify your security team with the gathered summary information.

This final task state generates an email message providing in-depth detail about the event using the API activity summary, and publishes the message to an SNS topic subscribed to by the members of your security team.

If the catch configuration of the task state in Step 1 was triggered, then the security notification email instead directs your security team to log in to the console and navigate to the Personal Health Dashboard to view more details on the incident.

Lessons learned

When implementing this use case with Step Functions and Lambda, consider the following:

  • One of the most important parts of implementing automation in response to operational events is to ensure visibility into the response and resolution actions is retained. Step Functions and Lambda enable you to orchestrate your granular response and resolution actions that provides direct visibility into the state of the automation workflow.
  • This basic workflow currently executes these steps serially with a catch configuration for error handling. More sophisticated workflows can leverage the parallel execution, branching logic, and time delay functionality provided by Step Functions.
  • Catch and retry configurations for task states allow for orchestrating reliable workflows while maintaining the granularity of each Lambda function. Without leveraging a catch configuration in Step 1, you would have had to duplicate code from the function in Step 3 to ensure that your security team was notified on failure to delete the access key.
  • Step Functions and Lambda are serverless services, so there is no cost for these services when they are not running. Because this IT automation workflow only runs when an IAM access key is exposed for this account (which is hopefully rare!), the total monthly cost for this workflow is essentially $0.

Conclusion

Automating the response to operational events for resources in your AWS account can free up the valuable time of your engineers. Step Functions and Lambda enable granular IT automation workflows to achieve this result while gaining direct visibility into the orchestration and state of the automation.

For more examples of how to use Step Functions to automate the operations of your AWS resources, or if you’d like to see how Step Functions can be used to build and orchestrate serverless applications, visit Getting Started on the Step Functions website.

Build Serverless AWS CodeCommit Workflows using Amazon CloudWatch Events and JGit

Post Syndicated from Chris Barclay original https://aws.amazon.com/blogs/devops/build-serverless-aws-codecommit-workflows-using-amazon-cloudwatch-events-and-jgit/

Sam Dengler is a Solutions Architect at Amazon Web Services

Summary

Amazon CloudWatch Events now supports AWS CodeCommit Repository State Changes event types for activities like pushing new code to a repository. Using these new event types, customers can build Amazon CloudWatch Event rules to match AWS CodeCommit events and route them to one or more targets like an Amazon SNS Topic, AWS Step Functions state machine, or AWS Lambda function to trigger automated workflows to process repository changes.

In this blog, I will provide three examples for using AWS Lambda and JGit to build cost-effective serverless solutions to securely process AWS CodeCommit repository state changes:

  • Replicate CodeCommit Repository
  • Enforce Git Commit Message Policy
  • Backup Git Archive to Amazon S3

Source code and Amazon CloudFormation templates for the examples are located in the following GitHub repository: https://github.com/awslabs/serverless-codecommit-examples.

AWS CodeCommit CloudWatch Events

Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in AWS resources. Below is an example Amazon CloudWatch Event for one of the new AWS CodeCommit Repository State Changes event types, referenceUpdated. Any change to the repository will trigger a referenceUpdated event, however triggers for particular branches can be filtered using the referenceType and referenceName fields in the event details.

{
    "version": "0",
    "id": "01234567-0123-0123-0123-012345678901",
    "detail-type": "CodeCommit Repository State Change",
    "source": "aws.codecommit",
    "account": "123456789012",
    "time": "2017-06-12T10:23:43Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:codecommit:us-east-1:123456789012:myRepo"
    ],
    "detail": {
        "event": "referenceUpdated",
        "repositoryName": "myRepo",
        "referenceType": "head",
        "referenceName": "myBranch",
        "commitId": "3e5983EXAMPLE",
        "oldCommitId": "1a7813EXAMPLE"
    }
}

We will use the Amazon CloudWatch Event’s fields to create a pattern to match the events for which we will trigger a target, in this case an AWS Lambda function.

Accessing AWS CodeCommit using HTTPS URLs

The HTTPS URL method for accessing an AWS CodeCommit repository is particularly suited to a serverless solution because an Amazon Lambda execution container already provides temporary AWS IAM key credentials associated to the function’s AWS IAM Execution Role. The function’s Execution Role is associated to one or more AWS IAM Policies, in which you specify permissions allowing the function to access AWS resources. For each function, we will limit the AWS IAM polices to only the AWS CodeCommit repository, Amazon S3 bucket, or Amazon SNS topics, following the AWS best practice to grant least privileged access.

For example, the AWS IAM Policy snippet below restricts the Amazon Lambda function to pull from the source repository and push to the target repository.

Policies:
  - Version: '2012-10-17'
    Statement:
      - Effect: Allow
        Resource: !Sub 'arn:aws:codecommit:${AWS::Region}:${AWS::AccountId}:${SourceRepositoryName}'
        Action:
          - 'codecommit:GetRepository'
          - 'codecommit:GitPull'
      - Effect: Allow
        Resource: !Sub 'arn:aws:codecommit:${TargetRepositoryRegion}:${AWS::AccountId}:${TargetRepositoryName}'
        Action:
          - 'codecommit:GetRepository'
          - 'codecommit:GitPush'

When using the HTTPS URL access method, a credential helper is configured for the Git client, which executes the “aws codecommit credential-helper” command to provide a SigV4 compatible user name and password using AWS IAM credentials (see more). When using JGit as the Git client, a CredentialsProvider can be supplied to Git commands to achieve the same result.

The Spring Cloud Config project provides an implementation of the JGit CredentialsProvider for AwsCodeCommit (source), which conveniently uses the AWS DefaultAWSCredentialsProviderChain to discover AWS credentials in the standard priority order supported by Amazon Lambda. The AwsCodeCommit.calculateCodeCommitPassword method is particularly interesting to review as an example of SigV4 transformation logic.

Cloning a repository is repeated across examples, and the functionality has been delegated to a supporting CloneCommandBuilder Class below.

public class CloneCommandBuilder {

    private File directory;

    public CloneCommandBuilder() throws IOException {
        directory = Files.createTempDirectory(null).toFile();
    }

    public CloneCommand buildCloneCommand(String sourceUrl) {
        return buildCloneCommand(sourceUrl, new AwsCodeCommitCredentialProvider());
    }

    public CloneCommand buildCloneCommand(String sourceUrl,
            AwsCodeCommitCredentialProvider credentialsProvider) {

        return new CloneCommand().setDirectory(directory)
                .setURI(sourceUrl)
                .setCredentialsProvider(credentialsProvider)
                .setBare(true);
    }
}

Next we’ll look at some examples to process the repository events using Amazon Lambda.

Example 1: Replicate CodeCommit Repository

Customers often need to replicate commits from one repository to another to support disaster recovery or cross region CI/CD pipelines. In this example, the Amazon Lambda function will clone a repository from the source and push to the target. This example is intended to update an existing target repository, which should not be empty before configuring replication prior to configuring replication.

Please note, Amazon Lambda functions are limited to 1.5GB memory, 512MB ephemeral disk capacity (“/tmp” space), and a 5 minute execution time. If your repository is unable to be processed within these limits, please see the Replicating and Automating Sync-Ups for a Repository with AWS CodeCommit blog article for an alternative approach to replicate repositories using an Amazon EC2 instance.

Let’s take a look at some code!

public class ReplicateRepositoryHandler
        implements RequestHandler<CodeCommitEvent, HandlerResponse> {

    private static Logger logger = Logger.getLogger(ReplicateRepositoryHandler.class);
    private final String targetUrl;
    private final AwsCodeCommitCredentialProvider credentialsProvider;

    public ReplicateRepositoryHandler() {
        String targetName = System.getenv("TARGET_REPO_NAME");
        String targetRegion = System.getenv("TARGET_REPO_REGION");

        CodeCommitMetadata target = new CodeCommitMetadata(targetName, targetRegion);
        targetUrl = target.getCloneUrlHttp();
        credentialsProvider = new AwsCodeCommitCredentialProvider();
    }

    // ...

On instantiation, the Amazon Lambda function discovers the target AWS CodeCommit repository HTTPS URL by querying the repository metadata using the target repository name and region. This discovery process is repeated across the examples, and the code has been delegated to the CodeCommitMetadata class below.

public class CodeCommitMetadata {

    private RepositoryMetadata repositoryMetadata;

    public CodeCommitMetadata(String repoName, String repoRegion) {
        AWSCodeCommitClientBuilder builder = AWSCodeCommitClientBuilder.standard();
        AWSCodeCommit client = builder.withRegion(repoRegion).build();

        GetRepositoryRequest request = new GetRepositoryRequest();
        request.withRepositoryName(repoName);

        GetRepositoryResult result = client.getRepository(request);
        repositoryMetadata = result.getRepositoryMetadata();
    }

    public String getCloneUrlHttp() {
        return repositoryMetadata.getCloneUrlHttp();
    }
}

When the Amazon Lambda function is triggered by the AWS CloudWatch Event, the source repository name and region in the event are used to discover the source repository HTTPS URL. We use JGit to clone the source repository from this URL into a local repository stored in a temporary directory in the Amazon Lambda execution container.

public HandlerResponse handleRequest(CodeCommitEvent event, Context context) {
    try {
        String sourceName = event.getDetail().getRepositoryName();
        String sourceRegion = event.getRegion();
   
        // clone source repository
        CodeCommitMetadata source = new CodeCommitMetadata(sourceName, sourceRegion);
        String sourceUrl = source.getCloneUrlHttp();
        Git git = new CloneCommandBuilder().buildCloneCommand(sourceUrl).call();

        // ...

Once we’ve cloned the local repository to the Amazon Lambda execution container, the last step is to set the target AWS CodeCommit repository as a new remote location and push the local references using the reference specification “+refs/*:refs/*”.

// push target repository
git.push().setCredentialsProvider(credentialsProvider)
          .setRemote(targetUrl)
          .setRefSpecs(new RefSpec("+refs/*:refs/*"))
          .call();
// ...

In the next example, we’ll review how we can build a Lambda function to enforce commit message policies.

Example 2: Enforce Git Commit Message Policy

Some customers choose to enforce policies on a Git repository to maintain code quality. In this example, we use the same tools described above to clone a repository and validate the commit messages from the Git log using a regular expression.

public class PolicyEnforcerHandler implements RequestHandler<CodeCommitEvent, HandlerResponse> {

    private static Logger logger = LoggerFactory.getLogger(ArchiveRepositoryHandler.class);

    private final String mainBranch;
    private final String snsTopicArn;
    private final Pattern pattern;
    private final AmazonSNS snsClient;

    public PolicyEnforcerHandler() {
        mainBranch = System.getenv("MAIN_BRANCH_NAME");
        snsTopicArn = System.getenv("SNS_TOPIC_ARN");

        String messageRegex = System.getenv("MESSAGE_REGEX");
        pattern = Pattern.compile(messageRegex);

        String snsRegion = snsTopicArn.split(":")[3];
        snsClient = AmazonSNSClientBuilder.standard().withRegion(snsRegion).build();
    }

    // ...

On instantiation, the Amazon Lambda function compiles the regular expression for message validation and creates an Amazon SNS client to send notifications.

@Override
public HandlerResponse handleRequest(CodeCommitEvent event, Context context) {
    String sourceName = event.getDetail().getRepositoryName();
    String sourceRegion = event.getRegion();
    String commitId = event.getDetail().getCommitId();
    String oldCommitId = event.getDetail().getOldCommitId();

    try {
        // clone source repository
        CodeCommitMetadata source = new CodeCommitMetadata(sourceName, sourceRegion);
        String sourceUrl = source.getCloneUrlHttp();
        Git git = new CloneCommandBuilder().buildCloneCommand(sourceUrl).call();

        // ...

When the Amazon Lambda function is triggered by the AWS CloudWatch Event, the process to clone the repository is the same, discovering the AWS CodeCommit HTTPS URL from the AWS CloudWatch Event and cloning a bare Git repository to the Amazon Lambda execution container.

// use the OldCommitId, or default to the main branch
String toGitReference = Optional.ofNullable(oldCommitId).orElse(mainBranch);
Repository repository = git.getRepository();
ObjectId to = repository.resolve(toGitReference);
ObjectId from = repository.resolve(commitId);

// ...

JGit RevWalk is used to determine the range of commits over which to validate the message policy. When commits are added to an existing branch, AWS CodeCommit will emit an referenceUpdated event, which includes commitId and oldCommitId fields that establish the range of commits.

When commits are added to a new branch, AWS CodeCommit will emit a referenceCreated event, which includes a commitId but not the oldCommitId. In this case, we will use the main branch name to determine the common ancestry of the commit chains, called the merge base, in order to establish the range of commits.

// create a RevWalk and set the range of commits
try (RevWalk walk = new RevWalk(repository)) {
    walk.markStart(walk.parseCommit(from));
    walk.markUninteresting(walk.parseCommit(to));

    // iterate the list of commits and validate each message
    for (RevCommit commit : walk) {
        Matcher matcher = pattern.matcher(commit.getShortMessage());

        // publish a message to the topic if the message does not match
        if (!matcher.find()) {
            String message = buildMessage(commit);
            logger.info(message);
            snsClient.publish(snsTopicArn, message);
        }
    }

    walk.dispose();
}

// ...

Once the range has been established, we iterate the list of commit messages, testing each against the message policy regular expression. If the message does not match the regular expression, then it is out of compliance from the policy, and a message is published to the Amazon SNS topic for notification.

In the next example, I’ll review how to backup an archive of the files in a Git repository.

Example 3: Backup Git Archive to Amazon S3

The previous examples have focused on the bare Git repository objects, however there are some use cases for processing the files in the Git repository at a particular reference. In this example, I’ll build a Lambda function to create a zip of the files in the repository and store it in Amazon S3 as a backup.

public class ArchiveRepositoryHandler
        implements RequestHandler<CodeCommitEvent, HandlerResponse> {

    private static Logger logger = LoggerFactory.getLogger(ArchiveRepositoryHandler.class);

    private final String ZIP_FORMAT = "zip"
    private final String targetS3Bucket;
    private final AmazonS3 s3Client;

    public ArchiveRepositoryHandler() {
        targetS3Bucket = System.getenv("TARGET_S3_BUCKET");
        s3Client = AmazonS3ClientBuilder.defaultClient();
        ArchiveCommand.registerFormat(ZIP_FORMAT, new ZipFormat());
    }

    // ...

On instantiation, the Amazon Lambda function creates an Amazon S3 client and registers the ZipFormat with JGit.

@Override
public HandlerResponse handleRequest(CodeCommitEvent event, Context context) {
    String sourceName = event.getDetail().getRepositoryName();
    String sourceRegion = event.getRegion();
    String commitId = event.getDetail().getCommitId();

    try {
        // clone source repository
        CodeCommitMetadata source = new CodeCommitMetadata(sourceName, sourceRegion);
        String sourceUrl = source.getCloneUrlHttp();
        Git git = new CloneCommandBuilder().buildCloneCommand(sourceUrl).call();

        // ...

When the Amazon Lambda function is triggered by the AWS CloudWatch Event, the process to clone the repository is the same, discovering the AWS CodeCommit HTTPS URL from the AWS CloudWatch Event and cloning a bare Git repository to the Amazon Lambda execution container.

// create and upload archive for commitId
File file = Files.createTempFile(null, null).toFile();
try (OutputStream out = new FileOutputStream(file)) {
    ObjectId objectId = git.getRepository().resolve(commitId);
    git.archive().setTree(objectId)
                 .setFormat(ZIP_FORMAT)
                 .setOutputStream(out)
                 .call();

    String key = sourceName + "." + commitId + "." + ZIP_FORMAT;
    s3Client.putObject(targetS3Bucket, key, file);
}

// ...

Once the repository has been cloned, we use a JGit ArchiveCommand to create a zip artifact representing the working files of repository at the commit triggering the event. The generated zip artifact is then uploaded to Amazon S3 using the repository name and commit shortId as the key.

Conclusion

AWS CloudWatch Event’s support for AWS CodeCommit Repository State Changes event types opens possibilities to build event-driven source code workflow automation using the same AWS CloudWatch Events service that acts as an event bus across many AWS services. Combining this new capability with Amazon Lambda, the JGit client, and AWS IAM policy controls provides builders with a set of tools to build serverless solutions that securely access AWS resources, scale on demand, and are cost effective.

In this blog, I’ve demonstrated three example solutions built using these tools, however AWS CodeCommit’s integration with AWS CloudWatch Events allows you to integrate with other AWS CloudWatch Events targets, like Amazon SQS or AWS Step Functions.

I encourage you to visit the GitHub repository (https://github.com/awslabs/serverless-codecommit-examples), which has instructions to launch these examples in your own AWS account. Please share your ideas and questions in the comments below, or submit pull requests and issues to the GitHub repository!

How to Create an AMI Builder with AWS CodeBuild and HashiCorp Packer – Part 2

Post Syndicated from Heitor Lessa original https://aws.amazon.com/blogs/devops/how-to-create-an-ami-builder-with-aws-codebuild-and-hashicorp-packer-part-2/

Written by AWS Solutions Architects Jason Barto and Heitor Lessa

 
In Part 1 of this post, we described how AWS CodeBuild, AWS CodeCommit, and HashiCorp Packer can be used to build an Amazon Machine Image (AMI) from the latest version of Amazon Linux. In this post, we show how to use AWS CodePipeline, AWS CloudFormation, and Amazon CloudWatch Events to continuously ship new AMIs. We use Ansible by Red Hat to harden the OS on the AMIs through a well-known set of security controls outlined by the Center for Internet Security in its CIS Amazon Linux Benchmark.

You’ll find the source code for this post in our GitHub repo.

At the end of this post, we will have the following architecture:

Requirements

 
To follow along, you will need Git and a text editor. Make sure Git is configured to work with AWS CodeCommit, as described in Part 1.

Technologies

 
In addition to the services and products used in Part 1 of this post, we also use these AWS services and third-party software:

AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.

Amazon CloudWatch Events enables you to react selectively to events in the cloud and in your applications. Specifically, you can create CloudWatch Events rules that match event patterns, and take actions in response to those patterns.

AWS CodePipeline is a continuous integration and continuous delivery service for fast and reliable application and infrastructure updates. AWS CodePipeline builds, tests, and deploys your code every time there is a code change, based on release process models you define.

Amazon SNS is a fast, flexible, fully managed push notification service that lets you send individual messages or to fan out messages to large numbers of recipients. Amazon SNS makes it simple and cost-effective to send push notifications to mobile device users or email recipients. The service can even send messages to other distributed services.

Ansible is a simple IT automation system that handles configuration management, application deployment, cloud provisioning, ad-hoc task-execution, and multinode orchestration.

Getting Started

 
We use CloudFormation to bootstrap the following infrastructure:

Component Purpose
AWS CodeCommit repository Git repository where the AMI builder code is stored.
S3 bucket Build artifact repository used by AWS CodePipeline and AWS CodeBuild.
AWS CodeBuild project Executes the AWS CodeBuild instructions contained in the build specification file.
AWS CodePipeline pipeline Orchestrates the AMI build process, triggered by new changes in the AWS CodeCommit repository.
SNS topic Notifies subscribed email addresses when an AMI build is complete.
CloudWatch Events rule Defines how the AMI builder should send a custom event to notify an SNS topic.
Region AMI Builder Launch Template
N. Virginia (us-east-1)
Ireland (eu-west-1)

After launching the CloudFormation template linked here, we will have a pipeline in the AWS CodePipeline console. (Failed at this stage simply means we don’t have any data in our newly created AWS CodeCommit Git repository.)

Next, we will clone the newly created AWS CodeCommit repository.

If this is your first time connecting to a AWS CodeCommit repository, please see instructions in our documentation on Setup steps for HTTPS Connections to AWS CodeCommit Repositories.

To clone the AWS CodeCommit repository (console)

  1. From the AWS Management Console, open the AWS CloudFormation console.
  2. Choose the AMI-Builder-Blogpost stack, and then choose Output.
  3. Make a note of the Git repository URL.
  4. Use git to clone the repository.

For example: git clone https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/AMI-Builder_repo

To clone the AWS CodeCommit repository (CLI)

# Retrieve CodeCommit repo URL
git_repo=$(aws cloudformation describe-stacks --query 'Stacks[0].Outputs[?OutputKey==`GitRepository`].OutputValue' --output text --stack-name "AMI-Builder-Blogpost")

# Clone repository locally
git clone ${git_repo}

Bootstrap the Repo with the AMI Builder Structure

 
Now that our infrastructure is ready, download all the files and templates required to build the AMI.

Your local Git repo should have the following structure:

.
├── ami_builder_event.json
├── ansible
├── buildspec.yml
├── cloudformation
├── packer_cis.json

Next, push these changes to AWS CodeCommit, and then let AWS CodePipeline orchestrate the creation of the AMI:

git add .
git commit -m "My first AMI"
git push origin master

AWS CodeBuild Implementation Details

 
While we wait for the AMI to be created, let’s see what’s changed in our AWS CodeBuild buildspec.yml file:

...
phases:
  ...
  build:
    commands:
      ...
      - ./packer build -color=false packer_cis.json | tee build.log
  post_build:
    commands:
      - egrep "${AWS_REGION}\:\sami\-" build.log | cut -d' ' -f2 > ami_id.txt
      # Packer doesn't return non-zero status; we must do that if Packer build failed
      - test -s ami_id.txt || exit 1
      - sed -i.bak "s/<<AMI-ID>>/$(cat ami_id.txt)/g" ami_builder_event.json
      - aws events put-events --entries file://ami_builder_event.json
      ...
artifacts:
  files:
    - ami_builder_event.json
    - build.log
  discard-paths: yes

In the build phase, we capture Packer output into a file named build.log. In the post_build phase, we take the following actions:

  1. Look up the AMI ID created by Packer and save its findings to a temporary file (ami_id.txt).
  2. Forcefully make AWS CodeBuild to fail if the AMI ID (ami_id.txt) is not found. This is required because Packer doesn’t fail if something goes wrong during the AMI creation process. We have to tell AWS CodeBuild to stop by informing it that an error occurred.
  3. If an AMI ID is found, we update the ami_builder_event.json file and then notify CloudWatch Events that the AMI creation process is complete.
  4. CloudWatch Events publishes a message to an SNS topic. Anyone subscribed to the topic will be notified in email that an AMI has been created.

Lastly, the new artifacts phase instructs AWS CodeBuild to upload files built during the build process (ami_builder_event.json and build.log) to the S3 bucket specified in the Outputs section of the CloudFormation template. These artifacts can then be used as an input artifact in any later stage in AWS CodePipeline.

For information about customizing the artifacts sequence of the buildspec.yml, see the Build Specification Reference for AWS CodeBuild.

CloudWatch Events Implementation Details

 
CloudWatch Events allow you to extend the AMI builder to not only send email after the AMI has been created, but to hook up any of the supported targets to react to the AMI builder event. This event publication means you can decouple from Packer actions you might take after AMI completion and plug in other actions, as you see fit.

For more information about targets in CloudWatch Events, see the CloudWatch Events API Reference.

In this case, CloudWatch Events should receive the following event, match it with a rule we created through CloudFormation, and publish a message to SNS so that you can receive an email.

Example CloudWatch custom event

[
        {
            "Source": "com.ami.builder",
            "DetailType": "AmiBuilder",
            "Detail": "{ \"AmiStatus\": \"Created\"}",
            "Resources": [ "ami-12cd5guf" ]
        }
]

Cloudwatch Events rule

{
  "detail-type": [
    "AmiBuilder"
  ],
  "source": [
    "com.ami.builder"
  ],
  "detail": {
    "AmiStatus": [
      "Created"
    ]
  }
}

Example SNS message sent in email

{
    "version": "0",
    "id": "f8bdede0-b9d7...",
    "detail-type": "AmiBuilder",
    "source": "com.ami.builder",
    "account": "<<aws_account_number>>",
    "time": "2017-04-28T17:56:40Z",
    "region": "eu-west-1",
    "resources": ["ami-112cd5guf "],
    "detail": {
        "AmiStatus": "Created"
    }
}

Packer Implementation Details

 
In addition to the build specification file, there are differences between the current version of the HashiCorp Packer template (packer_cis.json) and the one used in Part 1.

Variables

  "variables": {
    "vpc": "{{env `BUILD_VPC_ID`}}",
    "subnet": "{{env `BUILD_SUBNET_ID`}}",
         “ami_name”: “Prod-CIS-Latest-AMZN-{{isotime \”02-Jan-06 03_04_05\”}}”
  },
  • ami_name: Prefixes a name used by Packer to tag resources during the Builders sequence.
  • vpc and subnet: Environment variables defined by the CloudFormation stack parameters.

We no longer assume a default VPC is present and instead use the VPC and subnet specified in the CloudFormation parameters. CloudFormation configures the AWS CodeBuild project to use these values as environment variables. They are made available throughout the build process.

That allows for more flexibility should you need to change which VPC and subnet will be used by Packer to launch temporary resources.

Builders

  "builders": [{
    ...
    "ami_name": “{{user `ami_name`| clean_ami_name}}”,
    "tags": {
      "Name": “{{user `ami_name`}}”,
    },
    "run_tags": {
      "Name": “{{user `ami_name`}}",
    },
    "run_volume_tags": {
      "Name": “{{user `ami_name`}}",
    },
    "snapshot_tags": {
      "Name": “{{user `ami_name`}}",
    },
    ...
    "vpc_id": "{{user `vpc` }}",
    "subnet_id": "{{user `subnet` }}"
  }],

We now have new properties (*_tag) and a new function (clean_ami_name) and launch temporary resources in a VPC and subnet specified in the environment variables. AMI names can only contain a certain set of ASCII characters. If the input in project deviates from the expected characters (for example, includes whitespace or slashes), Packer’s clean_ami_name function will fix it.

For more information, see functions on the HashiCorp Packer website.

Provisioners

  "provisioners": [
    {
        "type": "shell",
        "inline": [
            "sudo pip install ansible"
        ]
    }, 
    {
        "type": "ansible-local",
        "playbook_file": "ansible/playbook.yaml",
        "role_paths": [
            "ansible/roles/common"
        ],
        "playbook_dir": "ansible",
        "galaxy_file": "ansible/requirements.yaml"
    },
    {
      "type": "shell",
      "inline": [
        "rm .ssh/authorized_keys ; sudo rm /root/.ssh/authorized_keys"
      ]
    }

We used shell provisioner to apply OS patches in Part 1. Now, we use shell to install Ansible on the target machine and ansible-local to import, install, and execute Ansible roles to make our target machine conform to our standards.

Packer uses shell to remove temporary keys before it creates an AMI from the target and temporary EC2 instance.

Ansible Implementation Details

 
Ansible provides OS patching through a custom Common role that can be easily customized for other tasks.

CIS Benchmark and Cloudwatch Logs are implemented through two Ansible third-party roles that are defined in ansible/requirements.yaml as seen in the Packer template.

The Ansible provisioner uses Ansible Galaxy to download these roles onto the target machine and execute them as instructed by ansible/playbook.yaml.

For information about how these components are organized, see the Playbook Roles and Include Statements in the Ansible documentation.

The following Ansible playbook (ansible</playbook.yaml) controls the execution order and custom properties:

---
- hosts: localhost
  connection: local
  gather_facts: true    # gather OS info that is made available for tasks/roles
  become: yes           # majority of CIS tasks require root
  vars:
    # CIS Controls whitepaper:  http://bit.ly/2mGAmUc
    # AWS CIS Whitepaper:       http://bit.ly/2m2Ovrh
    cis_level_1_exclusions:
    # 3.4.2 and 3.4.3 effectively blocks access to all ports to the machine
    ## This can break automation; ignoring it as there are stronger mechanisms than that
      - 3.4.2 
      - 3.4.3
    # CloudWatch Logs will be used instead of Rsyslog/Syslog-ng
    ## Same would be true if any other software doesn't support Rsyslog/Syslog-ng mechanisms
      - 4.2.1.4
      - 4.2.2.4
      - 4.2.2.5
    # Autofs is not installed in newer versions, let's ignore
      - 1.1.19
    # Cloudwatch Logs role configuration
    logs:
      - file: /var/log/messages
        group_name: "system_logs"
  roles:
    - common
    - anthcourtney.cis-amazon-linux
    - dharrisio.aws-cloudwatch-logs-agent

Both third-party Ansible roles can be easily configured through variables (vars). We use Ansible playbook variables to exclude CIS controls that don’t apply to our case and to instruct the CloudWatch Logs agent to stream the /var/log/messages log file to CloudWatch Logs.

If you need to add more OS or application logs, you can easily duplicate the playbook and make changes. The CloudWatch Logs agent will ship configured log messages to CloudWatch Logs.

For more information about parameters you can use to further customize third-party roles, download Ansible roles for the Cloudwatch Logs Agent and CIS Amazon Linux from the Galaxy website.

Committing Changes

 
Now that Ansible and CloudWatch Events are configured as a part of the build process, commiting any changes to the AWS CodeComit Git Repository will triger a new AMI build process that can be followed through the AWS CodePipeline console.

When the build is complete, an email will be sent to the email address you provided as a part of the CloudFormation stack deployment. The email serves as notification that an AMI has been built and is ready for use.

Summary

 
We used AWS CodeCommit, AWS CodePipeline, AWS CodeBuild, Packer, and Ansible to build a pipeline that continuously builds new, hardened CIS AMIs. We used Amazon SNS so that email addresses subscribed to a SNS topic are notified upon completion of the AMI build.

By treating our AMI creation process as code, we can iterate and track changes over time. In this way, it’s no different from a software development workflow. With that in mind, software patches, OS configuration, and logs that need to be shipped to a central location are only a git commit away.

Next Steps

 
Here are some ideas to extend this AMI builder:

  • Hook up a Lambda function in Cloudwatch Events to update EC2 Auto Scaling configuration upon completion of the AMI build.
  • Use AWS CodePipeline parallel steps to build multiple Packer images.
  • Add a commit ID as a tag for the AMI you created.
  • Create a scheduled Lambda function through Cloudwatch Events to clean up old AMIs based on timestamp (name or additional tag).
  • Implement Windows support for the AMI builder.
  • Create a cross-account or cross-region AMI build.

Cloudwatch Events allow the AMI builder to decouple AMI configuration and creation so that you can easily add your own logic using targets (AWS Lambda, Amazon SQS, Amazon SNS) to add events or recycle EC2 instances with the new AMI.

If you have questions or other feedback, feel free to leave it in the comments or contribute to the AMI Builder repo on GitHub.

Visualize and Monitor Amazon EC2 Events with Amazon CloudWatch Events and Amazon Kinesis Firehose

Post Syndicated from Karan Desai original https://aws.amazon.com/blogs/big-data/visualize-and-monitor-amazon-ec2-events-with-amazon-cloudwatch-events-and-amazon-kinesis-firehose/

Monitoring your AWS environment is important for security, performance, and cost control purposes. For example, by monitoring and analyzing API calls made to your Amazon EC2 instances, you can trace security incidents and gain insights into administrative behaviors and access patterns. The kinds of events you might monitor include console logins, Amazon EBS snapshot creation/deletion/modification, VPC creation/deletion/modification, and instance reboots, etc.

In this post, I show you how to build a near real-time API monitoring solution for EC2 events using Amazon CloudWatch Events and Amazon Kinesis Firehose. Please be sure to have Amazon CloudTrail enabled in your account.

  • CloudWatch Events offers a near real-time stream of system events that describe changes in AWS resources. CloudWatch Events now supports Kinesis Firehose as a target.
  • Kinesis Firehose is a fully managed service for continuously capturing, transforming, and delivering data in minutes to storage and analytics destinations such as Amazon S3, Amazon Kinesis Analytics, Amazon Redshift, and Amazon Elasticsearch Service.

Walkthrough

For this walkthrough, you create a CloudWatch event rule that matches specific EC2 events such as:

  • Starting, stopping, and terminating an instance
  • Creating and deleting VPC route tables
  • Creating and deleting a security group
  • Creating, deleting, and modifying instance volumes and snapshots

Your CloudWatch event target is a Kinesis Firehose delivery stream that delivers this data to an Elasticsearch cluster, where you set up Kibana for visualization. Using this solution, you can easily load and visualize EC2 events in minutes without setting up complicated data pipelines.

Set up the Elasticsearch cluster

Create the Amazon ES domain in the Amazon ES console, or by using the create-elasticsearch-domain command in the AWS CLI.

This example uses the following configuration:

  • Domain Name: esLogSearch
  • Elasticsearch Version: 1
  • Instance Count: 2
  • Instance type:elasticsearch
  • Enable dedicated master: true
  • Enable zone awareness: true
  • Restrict Amazon ES to an IP-based access policy

Other settings are left as the defaults.

Create a Kinesis Firehose delivery stream

In the Kinesis Firehose console, create a new delivery stream with Amazon ES as the destination. For detailed steps, see Create a Kinesis Firehose Delivery Stream to Amazon Elasticsearch Service.

Set up CloudWatch Events

Create a rule, and configure the event source and target. You can choose to configure multiple event sources with several AWS resources, along with options to specify specific or multiple event types.

In the CloudWatch console, choose Events.

For Service Name, choose EC2.

In Event Pattern Preview, choose Edit and copy the pattern below. For this walkthrough, I selected events that are specific to the EC2 API, but you can modify it to include events for any of your AWS resources.

 

{
	"source": [
		"aws.ec2"
	],
	"detail-type": [
		"AWS API Call via CloudTrail"
	],
	"detail": {
		"eventSource": [
			"ec2.amazonaws.com"
		],
		"eventName": [
			"RunInstances",
			"StopInstances",
			"StartInstances",
			"CreateFlowLogs",
			"CreateImage",
			"CreateNatGateway",
			"CreateVpc",
			"DeleteKeyPair",
			"DeleteNatGateway",
			"DeleteRoute",
			"DeleteRouteTable",
"CreateSnapshot",
"DeleteSnapshot",
			"DeleteVpc",
			"DeleteVpcEndpoints",
			"DeleteSecurityGroup",
			"ModifyVolume",
			"ModifyVpcEndpoint",
			"TerminateInstances"
		]
	}
}

The following screenshot shows what your event looks like in the console.

Next, choose Add target and select the delivery stream that you just created.

Set up Kibana on the Elasticsearch cluster

Amazon ES provides a default installation of Kibana with every Amazon ES domain. You can find the Kibana endpoint on your domain dashboard in the Amazon ES console. You can restrict Amazon ES access to an IP-based access policy.

In the Kibana console, for Index name or pattern, type log. This is the name of the Elasticsearch index.

For Time-field name, choose @time.

To view the events, choose Discover.

The following chart demonstrates the API operations and the number of times that they have been triggered in the past 12 hours.

Summary

In this post, you created a continuous, near real-time solution to monitor various EC2 events such as starting and shutting down instances, creating VPCs, etc. Likewise, you can build a continuous monitoring solution for all the API operations that are relevant to your daily AWS operations and resources.

With Kinesis Firehose as a new target for CloudWatch Events, you can retrieve, transform, and load system events to the storage and analytics destination of your choice in minutes, without setting up complicated data pipelines.

If you have any questions or suggestions, please comment below.

Additional Reading

Learn how to build a serverless architecture to analyze Amazon CloudFront access logs using AWS Lambda, Amazon Athena, and Amazon Kinesis Analytics

 

 

 

How to Create an AMI Builder with AWS CodeBuild and HashiCorp Packer

Post Syndicated from Jason Barto original https://aws.amazon.com/blogs/devops/how-to-create-an-ami-builder-with-aws-codebuild-and-hashicorp-packer/

It’s an operational and security best practice to create and maintain custom Amazon Machine Images. Because it’s also a best practice to maintain infrastructure as code, it makes sense to use automated tooling to script the creation and configuration of AMIs that are used to quickly launch Amazon EC2 instances.

In this first of two posts, we’ll use AWS CodeBuild to programmatically create AMIs for use in our environment. As a part of the AMI generation, we will apply OS patches, configure a banner statement, and install some common software, forming a solid base for future Amazon EC2-based deployments.

Requirements

You will need Git and a text editor.

Technologies

AWS CodeBuild is a fully managed build service that enables customers to compile source code, run tests, and produce software packages that are ready to deploy. It is elastic, scalable, and provides curated build environments for programming languages such as Java, Ruby, Python, Go, and Node.js. With AWS CodeBuild, you don’t need to provision, manage, and scale your own build servers. For more information, see the AWS CodeBuild User Guide.

HashiCorp Packer is an open source utility designed to automate the creation of identical virtual machine images for multiple platforms from a single source configuration. For more information, see the HashiCorp Packer documentation.

Getting Started

We need to host the AWS CodeBuild and HashiCorp Packer project somewhere. AWS CodeBuild can use Amazon S3, AWS CodeCommit, or GitHub as source code repositories. For this tutorial, sign in to the AWS Management Console and create an AWS CodeCommit repository. If you have not used AWS CodeCommit before, we recommend that you start with the Git with AWS CodeCommit Tutorial.

Create an AWS CodeBuild Project in the Console

Create an AWS CodeBuild project. It will be used later to build our HashiCorp Packer template.

  1. From the AWS Management Console, open the AWS CodeBuild console.
  2. If you have no AWS CodeBuild projects, choose Get Started or, on the Build Projects page, choose Create project.
  3. On the Create project page, type a name for your AWS CodeBuild project (for example, AMI_Builder).
  4. For Source provider, choose AWS CodeCommit. From the Repository drop-down list, select the repository you created in the Getting Started step.

AWS CodeBuild uses containers to execute the project’s build instructions and build your project. You can specify custom container images hosted in Amazon EC2 Container Registry or DockerHub, but this tutorial uses the default managed Ubuntu container.

  1. For Environment Image, select Use an image managed by AWS CodeBuild. For Operating system, choose Ubuntu. For Runtime, choose Base. For Version, choose aws/codebuild/ubuntu-base:14.04.

The next section of the page tells AWS CodeBuild where to find commands that will be executed as a part of the build. For this project, the build commands are stored in the buildspec.yml file in your repository.

  1. For Build specification, accept the default.

CodeBuild project configuration

The buildspec.yml file will use HashiCorp Packer to execute a template and generate an Amazon EC2 AMI. There is no binary output or build result that will be used as an artifact.

  1. For Artifacts type, choose No artifacts.

In the Service Role section the service role you select here will provide the AWS CodeBuild container with permissions to your AWS account. HashiCorp Packer needs permissions to create a temporary EC2 instance and an AMI, delete an EC2 instance, and perform other EC2-related actions. .

For the purposes of this post, allow AWS CodeBuild to create a service role for you. You will update it later with the permissions required by HashiCorp Packer.

  1. For Service Role choose the default of Create a service role in your account.
  2. Choose Continue.
  3. Review the settings on the next page, and then choose Save.

Update Service Role Permissions

You have now created an AWS CodeBuild project that is connected to an AWS CodeCommit repository and is ready to build our source code. Now we need to grant AWS CodeBuild the additional permissions it will need to create, delete, and image an EC2 instance.

  1. Open the IAM console and click the AWS CodeBuild service role, created in the last section, to display its summary page.
  2. On the summary page, under Permissions, expand Inline policies, and click the link to create a policy.
  3. Choose Custom Policy, and then choose Select.
  4. Copy and paste the IAM policy from the HashiCorp Packer documentation into the text area. Type a name for the policy (for example, codebuild-AMI_Builder-ec2-permissions).
  5. Choose Validate Policy, and then choose Apply Policy to link the policy with the service role.

CodeBuild project permissions

AWS CodeBuild should now have the permissions required to create AMIs.

Initial Commit

The next step is to create the HashiCorp Packer template and build specification. Check out a copy of the Git repository and create two files:

  • The HashiCorp Packer template, amazon-linux_packer-template.json.
  • The AWS CodeBuild configuration file, buildspec.yml.

Create a HashiCorp Packer Template

A HashiCorp Packer template is a JSON-formatted document that provides HashiCorp Packer with information about how to build a machine image. A HashiCorp Packer template can be composed of many keys. In this post, the focus is on the builders and provisioners keys. For more information, see Templates on the HashiCorp Packer website.

Using a text editor, create a HashiCorp Packer template named amazon-linux_packer-template.json that contains three sections: variables, builders, and provisioners.

The variables section defines two variables, aws_region and aws_ami_name. These variables can be reused later in the template. They are defined by using an environment variable {{env `AWS_REGION`}} and an internal HashiCorp Packer function {{isotime \"02Jan2006\"}} . For more information about HashiCorp Packer functions, see Template Engine on the HashiCorp Packer website.

The builders section configures the amazon-ebs builder to deploy an instance into the same region in which AWS CodeBuild is running, using a t2.micro instance, and to connect to the instance with a username of ec2-user. The source_ami_filter is being used to find the latest version of Amazon Linux available for the target region. The selected source AMI will be the base for your custom AMI. After the EC2 instance has been provisioned, amazon-ebs will create an AMI using the value of the aws_ami_name variable as the AMI name.

{
    "variables": {
        "aws_region": "{{env `AWS_REGION`}}",
        "aws_ami_name": "amazon-linux_{{isotime \"02Jan2006\"}}"
    },

    "builders": [{
        "type": "amazon-ebs",
        "region": "{{user `aws_region`}}",
        "instance_type": "t2.micro",
        "ssh_username": "ec2-user",
        "ami_name": "{{user `aws_ami_name`}}",
        "ami_description": "Customized Amazon Linux",
        "associate_public_ip_address": "true",
        "source_ami_filter": {
            "filters": {
                "virtualization-type": "hvm",
                "name": "amzn-ami*-ebs",
                "root-device-type": "ebs"
            },
            "owners": ["137112412989", "591542846629", "801119661308", "102837901569", "013907871322", "206029621532", "286198878708", "443319210888"],
            "most_recent": true
        }
    }],

    "provisioners": [
        {
            "type": "shell",
            "inline": [
                "sudo yum update -y",
				"sudo /usr/sbin/update-motd --disable",
                "echo 'No unauthorized access permitted' | sudo tee /etc/motd",
                "sudo rm /etc/issue",
                "sudo ln -s /etc/motd /etc/issue",
                "sudo yum install -y elinks screen"
            ]
        }
    ]
}

The provisioners section provides shell commands that Packer will use to configure the virtual machine after it has been created by the builders, but before it is captured as a machine image. The provisioners section updates the package management system and cleans up temporary keys before exiting. For more information, see Provisioners on the HashiCorp Packer website.

AWS CodeBuild and the containers it executes operate outside of a customer’s VPC. Any actions performed as a part of your build project will not have access to private IP addresses in your VPC. As a result a public IP address must be assigned to the EC2 instance so that HashiCorp Packer can remotely connect to it and configure the system. If the instance is not going to be launched in your default VPC, you must provide HashiCorp Packer with the VPC and a public-facing subnet. You will also need to be able to use SSH to connect to the EC2 instance from the internet.

To protect against malicious behavior and prevent unnecessary access, during the short life of the EC2 instance, HashiCorp Packer will use a newly created key pair and security group to deploy the EC2 instance. They will be deleted after the AMI has been created.

The source_ami_filter is an alternative to source_ami, which states which AMI should be used to create the EC2 instance. source_ami_filter will query the AMIs available in your region and select the one that best matches the filter criteria. In this example, source_ami_filter is configured to find the most recent AMI that matches the expression amzn-ami*-ebs. The filter has been restricted to match AMIs owned by a limited set of AWS accounts. The owners listed are accounts owned and managed by AWS.

Create a Build Specification

AWS CodeBuild uses the buildspec.yml file to execute user-defined commands at various phases of the build process. This file tells AWS CodeBuild how to use HashiCorp Packer to execute the template.

The buildspec.yml must contain the following:

---
version: 0.2

phases:
  pre_build:
    commands:
      - echo "Installing HashiCorp Packer..."
      - curl -qL -o packer.zip https://releases.hashicorp.com/packer/0.12.3/packer_0.12.3_linux_amd64.zip && unzip packer.zip
      - echo "Installing jq..."
      - curl -qL -o jq https://stedolan.github.io/jq/download/linux64/jq && chmod +x ./jq
      - echo "Validating amazon-linux_packer-template.json"
      - ./packer validate amazon-linux_packer-template.json
  build:
    commands:
      ### HashiCorp Packer cannot currently obtain the AWS CodeBuild-assigned role and its credentials
      ### Manually capture and configure the AWS CLI to provide HashiCorp Packer with AWS credentials
      ### More info here: https://github.com/mitchellh/packer/issues/4279
      - echo "Configuring AWS credentials"
      - curl -qL -o aws_credentials.json http://169.254.170.2/$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI > aws_credentials.json
      - aws configure set region $AWS_REGION
      - aws configure set aws_access_key_id `./jq -r '.AccessKeyId' aws_credentials.json`
      - aws configure set aws_secret_access_key `./jq -r '.SecretAccessKey' aws_credentials.json`
      - aws configure set aws_session_token `./jq -r '.Token' aws_credentials.json`
      - echo "Building HashiCorp Packer template, amazon-linux_packer-template.json"
      - ./packer build amazon-linux_packer-template.json
  post_build:
    commands:
      - echo "HashiCorp Packer build completed on `date`"

During pre_build, this build file configures the AWS CodeBuild container with the tools it will need to execute the Packer template. The first tool is HashiCorp Packer, which is available for download from the HashiCorp website. The second tool is the jq utitility, used to parse JSON files. The last command in this phase validates the HashiCorp Packer template to ensure there are no syntax errors.

In the build phase, the build file configures the build container’s AWS credentials using the metadata URL. This metadata will provide the AWS credentials provided by the IAM role assigned to the AWS CodeBuild project earlier to HashiCorp Packer so that it can create EC2 resources on your behalf. As a final step the HashiCorp Packer template is executed, resulting in the building of an EC2 AMI.

Execute the AWS CodeBuild Project

Commit the new files to your repository and push them to AWS CodeCommit. You are now ready to have AWS CodeBuild create the AMI.

  1. From the AWS Management Console, navigate to the AWS CodeBuild console.
  2. In the list of build projects, choose the project you created, and then choose Start build.
  3. In Start new build, choose which branch and revision of your AWS CodeCommit repository should be used to build your AMI.
  4. From the Branch drop-down list, choose master. This should populate the Source version field with the latest commit you pushed to the repository.
  5. Choose Start build. AWS CodeBuild will execute the buildspec.yml file in your repository.

CodeBuild project start

You will be taken to a page that provides status and results for each phase of the build. Succeeded should be displayed for every stage. If an error occurred, you can review the build logs at the bottom of the page for any error messages.

CodeBuild project status

During the build, HashiCorp Packer will generate a temporary key pair, launch an EC2 instance, remotely connect to the instance using the generated key pair, and then provision the machine, before converting the instance to an AMI and tearing down everything that was created to build the AMI. After these steps are complete, the build log should contain output that shows an AMI with an ID of something like ami-1a2b3cde was created.

This AMI can now be used to create EC2 instances on demand or as part of an Auto Scaling group to elastically scale your infrastructure.

Next Steps

We hope you found the information in this first post helpful and will use it as a starting point for your own infrastructure as code pipeline. In this post, we created a portion of your infrastructure as code in the form of a HashiCorp Packer template. We used AWS CodeBuild to create an AMI based on the template. Your next steps could include:

  • Building more than one HashiCorp Packer template. Multiple HashiCorp Packer templates can be used to build custom AMIs based on different source AMIs, perhaps one for Ubuntu and another for Microsoft Windows.
  • Adding AWS CodePipeline to monitor your AWS CodeCommit repository. When a new version is committed, AWS CodePipeline will call AWS CodeBuild and re-create your AMIs automatically.
  • Using CloudWatch Events to implement automated compliance. You can use an AWS Lambda function to verify that all EC2 instances launched in your account are using one of your preconfigured custom AMIs.

In the next post in this two-part series, we’ll use AWS services such as AWS CodePipeline, AWS CloudFormation, and Amazon Cloudwatch Events to continuously release new AMIs from end to end.

Implementing DevSecOps Using AWS CodePipeline

Post Syndicated from Ramesh Adabala original https://aws.amazon.com/blogs/devops/implementing-devsecops-using-aws-codepipeline/

DevOps is a combination of cultural philosophies, practices, and tools that emphasizes collaboration and communication between software developers and IT infrastructure teams while automating an organization’s ability to deliver applications and services rapidly, frequently, and more reliably.

CI/CD stands for continuous integration and continuous deployment. These concepts represent everything related to automation of application development and the deployment pipeline — from the moment a developer adds a change to a central repository until that code winds up in production.

DevSecOps covers security of and in the CI/CD pipeline, including automating security operations and auditing. The goals of DevSecOps are to:

  • Embed security knowledge into DevOps teams so that they can secure the pipelines they design and automate.
  • Embed application development knowledge and automated tools and processes into security teams so that they can provide security at scale in the cloud.

The Security Cloud Adoption Framework (CAF) whitepaper provides prescriptive controls to improve the security posture of your AWS accounts. These controls are in line with a DevOps blog post published last year about the control-monitor-fix governance model.

Security CAF controls are grouped into four categories:

  • Directive: controls establish the governance, risk, and compliance models on AWS.
  • Preventive: controls protect your workloads and mitigate threats and vulnerabilities.
  • Detective: controls provide full visibility and transparency over the operation of your deployments in AWS.
  • Responsive: controls drive remediation of potential deviations from your security baselines.

To embed the DevSecOps discipline in the enterprise, AWS customers are automating CAF controls using a combination of AWS and third-party solutions.

In this blog post, I will show you how to use a CI/CD pipeline to automate preventive and detective security controls. I’ll use an example that show how you can take the creation of a simple security group through the CI/CD pipeline stages and enforce security CAF controls at various stages of the deployment. I’ll use AWS CodePipeline to orchestrate the steps in a continuous delivery pipeline.

These resources are being used in this example:

  • An AWS CloudFormation template to create the demo pipeline.
  • A Lambda function to perform the static code analysis of the CloudFormation template.
  • A Lambda function to perform dynamic stack validation for the security groups in scope.
  • An S3 bucket as the sample code repository.
  • An AWS CloudFormation source template file to create the security groups.
  • Two VPCs to deploy the test and production security groups.

These are the high-level security checks enforced by the pipeline:

  • During the Source stage, static code analysis for any open security groups. The pipeline will fail if there are any violations.
  • During the Test stage, dynamic analysis to make sure port 22 (SSH) is open only to the approved IP CIDR range. The pipeline will fail if there are any violations.

demo_pipeline1

 

These are the pipeline stages:

1. Source stage: In this example, the pipeline gets the CloudFormation code that creates the security group from S3, the code repository service.

This stage passes the CloudFormation template and pipeline name to a Lambda function, CFNValidateLambda. This function performs the static code analysis. It uses the regular expression language to find patterns and identify security group policy violations. If it finds violations, then Lambda fails the pipeline and includes the violation details.

Here is the regular expression that Lambda function using for static code analysis of the open SSH port:

"^.*Ingress.*(([fF]rom[pP]ort|[tT]o[pP]ort).\s*:\s*u?.(22).*[cC]idr[iI]p.\s*:\s*u?.((0\.){3}0\/0)|[cC]idr[iI]p.\s*:\s*u?.((0\.){3}0\/0).*([fF]rom[pP]ort|[tT]o[pP]ort).\s*:\s*u?.(22))"

2. Test stage: After the static code analysis is completed successfully, the pipeline executes the following steps:

a. Create stack: This step creates the stack in the test VPC, as described in the test configuration.

b. Stack validation: This step triggers the StackValidationLambda Lambda function. It passes the stack name and pipeline name in the event parameters. Lambda validates the security group for the following security controls. If it finds violations, then Lambda deletes the stack, stops the pipeline, and returns an error message.

The following is the sample Python code used by AWS Lambda to check if the SSH port is open to the approved IP CIDR range (in this example, 72.21.196.67/32):

for n in regions:
    client = boto3.client('ec2', region_name=n)
    response = client.describe_security_groups(
        Filters=[{'Name': 'tag:aws:cloudformation:stack-name', 'Values': [stackName]}])
    for m in response['SecurityGroups']:
        if "72.21.196.67/32" not in str(m['IpPermissions']):
            for o in m['IpPermissions']:
                try:
                    if int(o['FromPort']) <= 22 <= int(o['ToPort']):
                        result = False
                        failReason = "Found Security Group with port 22 open to the wrong source IP range"
                        offenders.append(str(m['GroupId']))
                except:
                    if str(o['IpProtocol']) == "-1":
                        result = False
                        failReason = "Found Security Group with port 22 open to the wrong source IP range"
                        offenders.append(str(n) + " : " + str(m['GroupId']))

c. Approve test stack: This step creates a manual approval task for stack review. This step could be eliminated for automated deployments.

d. Delete test stack: After all the stack validations are successfully completed, this step deletes the stack in the test environment to avoid unnecessary costs.

3. Production stage: After the static and dynamic security checks are completed successfully, this stage creates the stack in the production VPC using the production configuration supplied in the template.

a. Create change set: This step creates the change set for the resources in the scope.

b. Execute change set: This step executes the change set and creates/updates the security group in the production VPC.

 

Source code and CloudFormation template

You’ll find the source code at https://github.com/awslabs/automating-governance-sample/tree/master/DevSecOps-Blog-Code

basic-sg-3-cfn.json creates the pipeline in AWS CodePipeline with all the stages previously described. It also creates the static code analysis and stack validation Lambda functions.

The CloudFormation template points to a shared S3 bucket. The codepipeline-lambda.zip file contains the Lambda functions. Before you run the template, upload the zip file to your S3 bucket and then update the CloudFormation template to point to your S3 bucket location.

The CloudFormation template uses the codepipe-single-sg.zip file, which contains the sample security group and test and production configurations. Update these configurations with your VPC details, and then upload the modified zip file to your S3 bucket.

Update these parts of the code to point to your S3 bucket:

 "S3Bucket": {
      "Default": "codepipeline-devsecops-demo",
      "Description": "The name of the S3 bucket that contains the source artifact, which must be in the same region as this stack",
      "Type": "String"
    },
    "SourceS3Key": {
      "Default": "codepipe-single-sg.zip",
      "Description": "The file name of the source artifact, such as myfolder/myartifact.zip",
      "Type": "String"
    },
    "LambdaS3Key": {
      "Default": "codepipeline-lambda.zip",
      "Description": "The file name of the source artifact of the Lambda code, such as myfolder/myartifact.zip",
      "Type": "String"
    },
	"OutputS3Bucket": {
      "Default": "codepipeline-devsecops-demo",
      "Description": "The name of the output S3 bucket that contains the processed artifact, which must be in the same region as this stack",
      "Type": "String"
    },

After the stack is created, AWS CodePipeline executes the pipeline and starts deploying the sample CloudFormation template. In the default template, security groups have wide-open ports (0.0.0.0/0), so the pipeline execution will fail. Update the CloudFormation template in codepipe-single-sg.zip with more restrictive ports and then upload the modified zip file to S3 bucket. Open the AWS CodePipeline console, and choose the Release Change button. This time the pipeline will successfully create the security groups.

demo_pipeline2

You could expand the security checks in the pipeline to include other AWS resources, not just security groups. The following table shows the sample controls you could enforce in the pipeline using the static and dynamic analysis Lambda functions.

demo_pipeline3
If you have feedback about this post, please add it to the Comments section below. If you have questions about implementing the example used in this post, please open a thread on the Developer Tools forum.

In Case You Missed These: AWS Security Blog Posts from January, February, and March

Post Syndicated from Craig Liebendorfer original https://aws.amazon.com/blogs/security/in-case-you-missed-these-aws-security-blog-posts-from-january-february-and-march/

Image of lock and key

In case you missed any AWS Security Blog posts published so far in 2017, they are summarized and linked to below. The posts are shown in reverse chronological order (most recent first), and the subject matter ranges from protecting dynamic web applications against DDoS attacks to monitoring AWS account configuration changes and API calls to Amazon EC2 security groups.

March

March 22: How to Help Protect Dynamic Web Applications Against DDoS Attacks by Using Amazon CloudFront and Amazon Route 53
Using a content delivery network (CDN) such as Amazon CloudFront to cache and serve static text and images or downloadable objects such as media files and documents is a common strategy to improve webpage load times, reduce network bandwidth costs, lessen the load on web servers, and mitigate distributed denial of service (DDoS) attacks. AWS WAF is a web application firewall that can be deployed on CloudFront to help protect your application against DDoS attacks by giving you control over which traffic to allow or block by defining security rules. When users access your application, the Domain Name System (DNS) translates human-readable domain names (for example, www.example.com) to machine-readable IP addresses (for example, 192.0.2.44). A DNS service, such as Amazon Route 53, can effectively connect users’ requests to a CloudFront distribution that proxies requests for dynamic content to the infrastructure hosting your application’s endpoints. In this blog post, I show you how to deploy CloudFront with AWS WAF and Route 53 to help protect dynamic web applications (with dynamic content such as a response to user input) against DDoS attacks. The steps shown in this post are key to implementing the overall approach described in AWS Best Practices for DDoS Resiliency and enable the built-in, managed DDoS protection service, AWS Shield.

March 21: New AWS Encryption SDK for Python Simplifies Multiple Master Key Encryption
The AWS Cryptography team is happy to announce a Python implementation of the AWS Encryption SDK. This new SDK helps manage data keys for you, and it simplifies the process of encrypting data under multiple master keys. As a result, this new SDK allows you to focus on the code that drives your business forward. It also provides a framework you can easily extend to ensure that you have a cryptographic library that is configured to match and enforce your standards. The SDK also includes ready-to-use examples. If you are a Java developer, you can refer to this blog post to see specific Java examples for the SDK. In this blog post, I show you how you can use the AWS Encryption SDK to simplify the process of encrypting data and how to protect your encryption keys in ways that help improve application availability by not tying you to a single region or key management solution.

March 21: Updated CJIS Workbook Now Available by Request
The need for guidance when implementing Criminal Justice Information Services (CJIS)–compliant solutions has become of paramount importance as more law enforcement customers and technology partners move to store and process criminal justice data in the cloud. AWS services allow these customers to easily and securely architect a CJIS-compliant solution when handling criminal justice data, creating a durable, cost-effective, and secure IT infrastructure that better supports local, state, and federal law enforcement in carrying out their public safety missions. AWS has created several documents (collectively referred to as the CJIS Workbook) to assist you in aligning with the FBI’s CJIS Security Policy. You can use the workbook as a framework for developing CJIS-compliant architecture in the AWS Cloud. The workbook helps you define and test the controls you operate, and document the dependence on the controls that AWS operates (compute, storage, database, networking, regions, Availability Zones, and edge locations).

March 9: New Cloud Directory API Makes It Easier to Query Data Along Multiple Dimensions
Today, we made available a new Cloud Directory API, ListObjectParentPaths, that enables you to retrieve all available parent paths for any directory object across multiple hierarchies. Use this API when you want to fetch all parent objects for a specific child object. The order of the paths and objects returned is consistent across iterative calls to the API, unless objects are moved or deleted. In case an object has multiple parents, the API allows you to control the number of paths returned by using a paginated call pattern. In this blog post, I use an example directory to demonstrate how this new API enables you to retrieve data across multiple dimensions to implement powerful applications quickly.

March 8: How to Access the AWS Management Console Using AWS Microsoft AD and Your On-Premises Credentials
AWS Directory Service for Microsoft Active Directory, also known as AWS Microsoft AD, is a managed Microsoft Active Directory (AD) hosted in the AWS Cloud. Now, AWS Microsoft AD makes it easy for you to give your users permission to manage AWS resources by using on-premises AD administrative tools. With AWS Microsoft AD, you can grant your on-premises users permissions to resources such as the AWS Management Console instead of adding AWS Identity and Access Management (IAM) user accounts or configuring AD Federation Services (AD FS) with Security Assertion Markup Language (SAML). In this blog post, I show how to use AWS Microsoft AD to enable your on-premises AD users to sign in to the AWS Management Console with their on-premises AD user credentials to access and manage AWS resources through IAM roles.

March 7: How to Protect Your Web Application Against DDoS Attacks by Using Amazon Route 53 and an External Content Delivery Network
Distributed Denial of Service (DDoS) attacks are attempts by a malicious actor to flood a network, system, or application with more traffic, connections, or requests than it is able to handle. To protect your web application against DDoS attacks, you can use AWS Shield, a DDoS protection service that AWS provides automatically to all AWS customers at no additional charge. You can use AWS Shield in conjunction with DDoS-resilient web services such as Amazon CloudFront and Amazon Route 53 to improve your ability to defend against DDoS attacks. Learn more about architecting for DDoS resiliency by reading the AWS Best Practices for DDoS Resiliency whitepaper. You also have the option of using Route 53 with an externally hosted content delivery network (CDN). In this blog post, I show how you can help protect the zone apex (also known as the root domain) of your web application by using Route 53 to perform a secure redirect to prevent discovery of your application origin.

Image of lock and key

February

February 27: Now Generally Available – AWS Organizations: Policy-Based Management for Multiple AWS Accounts
Today, AWS Organizations moves from Preview to General Availability. You can use Organizations to centrally manage multiple AWS accounts, with the ability to create a hierarchy of organizational units (OUs). You can assign each account to an OU, define policies, and then apply those policies to an entire hierarchy, specific OUs, or specific accounts. You can invite existing AWS accounts to join your organization, and you can also create new accounts. All of these functions are available from the AWS Management Console, the AWS Command Line Interface (CLI), and through the AWS Organizations API.To read the full AWS Blog post about today’s launch, see AWS Organizations – Policy-Based Management for Multiple AWS Accounts.

February 23: s2n Is Now Handling 100 Percent of SSL Traffic for Amazon S3
Today, we’ve achieved another important milestone for securing customer data: we have replaced OpenSSL with s2n for all internal and external SSL traffic in Amazon Simple Storage Service (Amazon S3) commercial regions. This was implemented with minimal impact to customers, and multiple means of error checking were used to ensure a smooth transition, including client integration tests, catching potential interoperability conflicts, and identifying memory leaks through fuzz testing.

February 22: Easily Replace or Attach an IAM Role to an Existing EC2 Instance by Using the EC2 Console
AWS Identity and Access Management (IAM) roles enable your applications running on Amazon EC2 to use temporary security credentials. IAM roles for EC2 make it easier for your applications to make API requests securely from an instance because they do not require you to manage AWS security credentials that the applications use. Recently, we enabled you to use temporary security credentials for your applications by attaching an IAM role to an existing EC2 instance by using the AWS CLI and SDK. To learn more, see New! Attach an AWS IAM Role to an Existing Amazon EC2 Instance by Using the AWS CLI. Starting today, you can attach an IAM role to an existing EC2 instance from the EC2 console. You can also use the EC2 console to replace an IAM role attached to an existing instance. In this blog post, I will show how to attach an IAM role to an existing EC2 instance from the EC2 console.

February 22: How to Audit Your AWS Resources for Security Compliance by Using Custom AWS Config Rules
AWS Config Rules enables you to implement security policies as code for your organization and evaluate configuration changes to AWS resources against these policies. You can use Config rules to audit your use of AWS resources for compliance with external compliance frameworks such as CIS AWS Foundations Benchmark and with your internal security policies related to the US Health Insurance Portability and Accountability Act (HIPAA), the Federal Risk and Authorization Management Program (FedRAMP), and other regimes. AWS provides some predefined, managed Config rules. You also can create custom Config rules based on criteria you define within an AWS Lambda function. In this post, I show how to create a custom rule that audits AWS resources for security compliance by enabling VPC Flow Logs for an Amazon Virtual Private Cloud (VPC). The custom rule meets requirement 4.3 of the CIS AWS Foundations Benchmark: “Ensure VPC flow logging is enabled in all VPCs.”

February 13: AWS Announces CISPE Membership and Compliance with First-Ever Code of Conduct for Data Protection in the Cloud
I have two exciting announcements today, both showing AWS’s continued commitment to ensuring that customers can comply with EU Data Protection requirements when using our services.

February 13: How to Enable Multi-Factor Authentication for AWS Services by Using AWS Microsoft AD and On-Premises Credentials
You can now enable multi-factor authentication (MFA) for users of AWS services such as Amazon WorkSpaces and Amazon QuickSight and their on-premises credentials by using your AWS Directory Service for Microsoft Active Directory (Enterprise Edition) directory, also known as AWS Microsoft AD. MFA adds an extra layer of protection to a user name and password (the first “factor”) by requiring users to enter an authentication code (the second factor), which has been provided by your virtual or hardware MFA solution. These factors together provide additional security by preventing access to AWS services, unless users supply a valid MFA code.

February 13: How to Create an Organizational Chart with Separate Hierarchies by Using Amazon Cloud Directory
Amazon Cloud Directory enables you to create directories for a variety of use cases, such as organizational charts, course catalogs, and device registries. Cloud Directory offers you the flexibility to create directories with hierarchies that span multiple dimensions. For example, you can create an organizational chart that you can navigate through separate hierarchies for reporting structure, location, and cost center. In this blog post, I show how to use Cloud Directory APIs to create an organizational chart with two separate hierarchies in a single directory. I also show how to navigate the hierarchies and retrieve data. I use the Java SDK for all the sample code in this post, but you can use other language SDKs or the AWS CLI.

February 10: How to Easily Log On to AWS Services by Using Your On-Premises Active Directory
AWS Directory Service for Microsoft Active Directory (Enterprise Edition), also known as Microsoft AD, now enables your users to log on with just their on-premises Active Directory (AD) user name—no domain name is required. This new domainless logon feature makes it easier to set up connections to your on-premises AD for use with applications such as Amazon WorkSpaces and Amazon QuickSight, and it keeps the user logon experience free from network naming. This new interforest trusts capability is now available when using Microsoft AD with Amazon WorkSpaces and Amazon QuickSight Enterprise Edition. In this blog post, I explain how Microsoft AD domainless logon works with AD interforest trusts, and I show an example of setting up Amazon WorkSpaces to use this capability.

February 9: New! Attach an AWS IAM Role to an Existing Amazon EC2 Instance by Using the AWS CLI
AWS Identity and Access Management (IAM) roles enable your applications running on Amazon EC2 to use temporary security credentials that AWS creates, distributes, and rotates automatically. Using temporary credentials is an IAM best practice because you do not need to maintain long-term keys on your instance. Using IAM roles for EC2 also eliminates the need to use long-term AWS access keys that you have to manage manually or programmatically. Starting today, you can enable your applications to use temporary security credentials provided by AWS by attaching an IAM role to an existing EC2 instance. You can also replace the IAM role attached to an existing EC2 instance. In this blog post, I show how you can attach an IAM role to an existing EC2 instance by using the AWS CLI.

February 8: How to Remediate Amazon Inspector Security Findings Automatically
The Amazon Inspector security assessment service can evaluate the operating environments and applications you have deployed on AWS for common and emerging security vulnerabilities automatically. As an AWS-built service, Amazon Inspector is designed to exchange data and interact with other core AWS services not only to identify potential security findings but also to automate addressing those findings. Previous related blog posts showed how you can deliver Amazon Inspector security findings automatically to third-party ticketing systems and automate the installation of the Amazon Inspector agent on new Amazon EC2 instances. In this post, I show how you can automatically remediate findings generated by Amazon Inspector. To get started, you must first run an assessment and publish any security findings to an Amazon Simple Notification Service (SNS) topic. Then, you create an AWS Lambda function that is triggered by those notifications. Finally, the Lambda function examines the findings and then implements the appropriate remediation based on the type of issue.

February 6: How to Simplify Security Assessment Setup Using Amazon EC2 Systems Manager and Amazon Inspector
In a July 2016 AWS Blog post, I discussed how to integrate Amazon Inspector with third-party ticketing systems by using Amazon Simple Notification Service (SNS) and AWS Lambda. This AWS Security Blog post continues in the same vein, describing how to use Amazon Inspector to automate various aspects of security management. In this post, I show you how to install the Amazon Inspector agent automatically through the Amazon EC2 Systems Manager when a new Amazon EC2 instance is launched. In a subsequent post, I will show you how to update EC2 instances automatically that run Linux when Amazon Inspector discovers a missing security patch.

Image of lock and key

January

January 30: How to Protect Data at Rest with Amazon EC2 Instance Store Encryption
Encrypting data at rest is vital for regulatory compliance to ensure that sensitive data saved on disks is not readable by any user or application without a valid key. Some compliance regulations such as PCI DSS and HIPAA require that data at rest be encrypted throughout the data lifecycle. To this end, AWS provides data-at-rest options and key management to support the encryption process. For example, you can encrypt Amazon EBS volumes and configure Amazon S3 buckets for server-side encryption (SSE) using AES-256 encryption. Additionally, Amazon RDS supports Transparent Data Encryption (TDE). Instance storage provides temporary block-level storage for Amazon EC2 instances. This storage is located on disks attached physically to a host computer. Instance storage is ideal for temporary storage of information that frequently changes, such as buffers, caches, and scratch data. By default, files stored on these disks are not encrypted. In this blog post, I show a method for encrypting data on Linux EC2 instance stores by using Linux built-in libraries. This method encrypts files transparently, which protects confidential data. As a result, applications that process the data are unaware of the disk-level encryption.

January 27: How to Detect and Automatically Remediate Unintended Permissions in Amazon S3 Object ACLs with CloudWatch Events
Amazon S3 Access Control Lists (ACLs) enable you to specify permissions that grant access to S3 buckets and objects. When S3 receives a request for an object, it verifies whether the requester has the necessary access permissions in the associated ACL. For example, you could set up an ACL for an object so that only the users in your account can access it, or you could make an object public so that it can be accessed by anyone. If the number of objects and users in your AWS account is large, ensuring that you have attached correctly configured ACLs to your objects can be a challenge. For example, what if a user were to call the PutObjectAcl API call on an object that is supposed to be private and make it public? Or, what if a user were to call the PutObject with the optional Acl parameter set to public-read, therefore uploading a confidential file as publicly readable? In this blog post, I show a solution that uses Amazon CloudWatch Events to detect PutObject and PutObjectAcl API calls in near-real time and helps ensure that the objects remain private by making automatic PutObjectAcl calls, when necessary.

January 26: Now Available: Amazon Cloud Directory—A Cloud-Native Directory for Hierarchical Data
Today we are launching Amazon Cloud Directory. This service is purpose-built for storing large amounts of strongly typed hierarchical data. With the ability to scale to hundreds of millions of objects while remaining cost-effective, Cloud Directory is a great fit for all sorts of cloud and mobile applications.

January 24: New SOC 2 Report Available: Confidentiality
As with everything at Amazon, the success of our security and compliance program is primarily measured by one thing: our customers’ success. Our customers drive our portfolio of compliance reports, attestations, and certifications that support their efforts in running a secure and compliant cloud environment. As a result of our engagement with key customers across the globe, we are happy to announce the publication of our new SOC 2 Confidentiality report. This report is available now through AWS Artifact in the AWS Management Console.

January 18: Compliance in the Cloud for New Financial Services Cybersecurity Regulations
Financial regulatory agencies are focused more than ever on ensuring responsible innovation. Consequently, if you want to achieve compliance with financial services regulations, you must be increasingly agile and employ dynamic security capabilities. AWS enables you to achieve this by providing you with the tools you need to scale your security and compliance capabilities on AWS. The following breakdown of the most recent cybersecurity regulations, NY DFS Rule 23 NYCRR 500, demonstrates how AWS continues to focus on your regulatory needs in the financial services sector.

January 9: New Amazon GameDev Blog Post: Protect Multiplayer Game Servers from DDoS Attacks by Using Amazon GameLift
In online gaming, distributed denial of service (DDoS) attacks target a game’s network layer, flooding servers with requests until performance degrades considerably. These attacks can limit a game’s availability to players and limit the player experience for those who can connect. Today’s new Amazon GameDev Blog post uses a typical game server architecture to highlight DDoS attack vulnerabilities and discusses how to stay protected by using built-in AWS Cloud security, AWS security best practices, and the security features of Amazon GameLift. Read the post to learn more.

January 6: The Top 10 Most Downloaded AWS Security and Compliance Documents in 2016
The following list includes the 10 most downloaded AWS security and compliance documents in 2016. Using this list, you can learn about what other people found most interesting about security and compliance last year.

January 6: FedRAMP Compliance Update: AWS GovCloud (US) Region Receives a JAB-Issued FedRAMP High Baseline P-ATO for Three New Services
Three new services in the AWS GovCloud (US) region have received a Provisional Authority to Operate (P-ATO) from the Joint Authorization Board (JAB) under the Federal Risk and Authorization Management Program (FedRAMP). JAB issued the authorization at the High baseline, which enables US government agencies and their service providers the capability to use these services to process the government’s most sensitive unclassified data, including Personal Identifiable Information (PII), Protected Health Information (PHI), Controlled Unclassified Information (CUI), criminal justice information (CJI), and financial data.

January 4: The Top 20 Most Viewed AWS IAM Documentation Pages in 2016
The following 20 pages were the most viewed AWS Identity and Access Management (IAM) documentation pages in 2016. I have included a brief description with each link to give you a clearer idea of what each page covers. Use this list to see what other people have been viewing and perhaps to pique your own interest about a topic you’ve been meaning to research.

January 3: The Most Viewed AWS Security Blog Posts in 2016
The following 10 posts were the most viewed AWS Security Blog posts that we published during 2016. You can use this list as a guide to catch up on your blog reading or even read a post again that you found particularly useful.

January 3: How to Monitor AWS Account Configuration Changes and API Calls to Amazon EC2 Security Groups
You can use AWS security controls to detect and mitigate risks to your AWS resources. The purpose of each security control is defined by its control objective. For example, the control objective of an Amazon VPC security group is to permit only designated traffic to enter or leave a network interface. Let’s say you have an Internet-facing e-commerce website, and your security administrator has determined that only HTTP (TCP port 80) and HTTPS (TCP 443) traffic should be allowed access to the public subnet. As a result, your administrator configures a security group to meet this control objective. What if, though, someone were to inadvertently change this security group’s rules and enable FTP or other protocols to access the public subnet from any location on the Internet? That expanded access could weaken the security posture of your assets. Consequently, your administrator might need to monitor the integrity of your company’s security controls so that the controls maintain their desired effectiveness. In this blog post, I explore two methods for detecting unintended changes to VPC security groups. The two methods address not only control objectives but also control failures.

If you have questions about or issues with implementing the solutions in any of these posts, please start a new thread on the forum identified near the end of each post.

– Craig

Announcing the AWS Health Tools Repository

Post Syndicated from Ana Visneski original https://aws.amazon.com/blogs/aws/announcing-the-aws-health-tools-repository/

Tipu Qureshi and Ram Atur join us today with really cool news about a Git repository for AWS Health / Personal Health Dashboard.

-Ana

Today, we’re happy to release the AWS Health Tools repository, a community-based source of tools to automate remediation actions and customize Health alerts.

The AWS Health service provides personalized information about events that can affect your AWS infrastructure, guides you through scheduled changes, and accelerates the troubleshooting of issues that affect your AWS resources and accounts.  The AWS Health API also powers the Personal Health Dashboard, which gives you a personalized view into the performance and availability of the AWS services underlying your AWS resources. You can use Amazon CloudWatch Events to detect and react to changes in the status of AWS Personal Health Dashboard (AWS Health) events.

AWS Health Tools takes advantage of the integration of AWS Health, Amazon CloudWatch Events and AWS Lambda to implement customized automation in response to events regarding your AWS infrastructure. As an example, you can use AWS Health Tools to pause your deployments that are part of AWS CodePipeline when a CloudWatch event is generated in response to an AWS Health issue.

AWSHealthToolsArchitecture

The AWS Health Tools repository empowers customers to effectively utilize AWS Health events by tapping in to the collective ingenuity and expertise of the AWS community. The repository is free, public, and hosted on an independent platform. Furthermore, the repository contains full source code, allowing you to learn and contribute. We look forward to working together to leverage the combined wisdom and lessons learned by our experts and experts in the broader AWS user base.

Here’s a sample of the AWS Health tools that you now have access to:

To get started using these tools in your AWS account, see the readme file on GitHub. We encourage you to use this repository to share with the AWS community the AWS Health Tools you have written

-Tipu Qureshi and Ram Atur

How to Simplify Security Assessment Setup Using Amazon EC2 Systems Manager and Amazon Inspector

Post Syndicated from Eric Fitzgerald original https://aws.amazon.com/blogs/security/how-to-simplify-security-assessment-setup-using-ec2-systems-manager-and-amazon-inspector/

In a July 2016 AWS Blog post, I discussed how to integrate Amazon Inspector with third-party ticketing systems by using Amazon Simple Notification Service (SNS) and AWS Lambda.

This AWS Security Blog post continues in the same vein, describing how to use Amazon Inspector to automate various aspects of security management. In this post, I show you how to install the Amazon Inspector agent automatically through the Amazon EC2 Systems Manager when a new Amazon EC2 instance is launched. In a subsequent post, I will show you how to update EC2 instances automatically that run Linux when Amazon Inspector discovers a missing security patch.

An overview of EC2 Systems Manager and EC2 Simple Systems Manager (SSM)

Amazon EC2 Systems Manager is a set of services that makes it easy to manage your Windows or Linux hosts running on EC2 instances. EC2 Systems Manager does this through an agent called EC2 Simple Systems Manager (SSM), which is installed on your instances. With SSM on your EC2 instances, you can save yourself an SSH or RDP session to the instance to perform management tasks.

With EC2 Systems Manager, you can perform various tasks at scale through a simple API, CLI, or EC2 Run Command. The EC2 Run Command can execute a Unix shell script on Linux instances or a Windows PowerShell script on Windows instances. When you use EC2 Systems Manager to run a script on an EC2 instance, the output is piped to a text file in Amazon S3 for you automatically. Therefore, you can examine the output without visiting the system or inventing your own mechanism for capturing console output.

The solution

Step 1: Enable EC2 Systems Manager and install the EC2 SSM agent

Setting up EC2 Systems Manager is relatively straightforward, but you must set up EC2 Systems Manager at the time you launch the instance. This is because the SSM agent will use an instance role to communicate with the EC2 Systems Manager securely. When launched with the appropriately configured IAM role, the EC2 instance is provided with a set of credentials that allows the SSM agent to perform actions on behalf of the account owner. The policy on the IAM role determines the permissions associated with these credentials.

The easiest way I have found to do this is to create the role, and then each time you launch an instance, associate the role with the instance and provide the SSM agent installation script in the instance’s user data in the launch wizard or API. Here’s how:

  1. Create an instance role so that the on-instance SSM agent can communicate with EC2 Systems Manager. If you already need an instance role for some other purpose, use the IAM console to attach the AmazonEC2RoleforSSM managed policy to your existing role.
  2. When launching the instance with the EC2 launch wizard, associate the role you just created with the new instance.
  3. When launching the instance with the EC2 launch wizard, provide the appropriate script as user data for your operating system and architecture to install the SSM agent as the instance is launched. To see this process and scripts in full, see Installing the SSM Agent.

Note: You must change the scripts slightly when copying them from the instructions to the EC2 user data: the word region in the curl command must be replaced with the AWS region code (for example, us-east-1).

When your instance starts, the SSM agent is installed. Having the SSM agent on the instance is the key component to the automated installation of the Amazon Inspector agent on the instance.

Step 2: Automatically install the Amazon Inspector agent when new EC2 instances are launched

Let’s assume that you will install the SSM agent when you first launch your instances. With that assumption in mind, you have two methods for installing the Amazon Inspector agent.

Method 1: Install the Amazon Inspector agent with user data

Just as we did above with the SSM agent, we can use the user data feature of EC2 to execute the Amazon Inspector agent installation script during instance launch. This is useful if you have decided not to install the SSM agent, but it is more work than necessary if you are in the habit of deploying the SSM agent at the launch of an instance.

To install the Amazon Inspector agent with user data on Linux systems, simply add the following commands to the User data box in the instance launch wizard (as shown in the following screenshot). This script works without modification on any Linux distribution that Amazon Inspector supports.

#!/bin/bash
cd /tmp
curl -O https://d1wk0tztpsntt1.cloudfront.net/linux/latest/install
chmod +x /tmp/install
/tmp/install

Note: If you are adding these commands to existing user data, be sure that only the first line of user data is #!/bin/bash. You should not have multiple copies of this line.

Finish launching the EC2 instance and the Amazon Inspector agent is installed as the instance is starting for the first time. To read more about this process, see Working with AWS Agents on Linux-based Operating Systems.

Method 2: Install the Amazon Inspector agent whenever a new EC2 instance starts

In environments that launch new instances continually, installing the Amazon Inspector agent automatically when an instance starts prevents some additional work. As we discussed in the previous method, you need to modify your instance launch process to include the EC2 SSM agent. This means you need to configure your instances with an EC2 Systems Manager role, as well as run the EC2 SSM agent.

First, create an IAM role that gives your Lambda function the permissions it needs to deploy the Amazon Inspector agent. Then, create the Lambda job that uses the SSM RunShellScript to install the Amazon Inspector agent. Finally, set up Amazon CloudWatch Events to run the Lambda job whenever a new instance enters the Running state.

Here are the details of the three-step process:

Step 1 – Create an IAM role for the Lambda function to use to send commands to EC2 Systems Manager:

  1. Sign in to the AWS Management Console and navigate to the IAM console.
  2. Choose Roles in the navigation pane. Choose Create new role.
  3. Type a name for a role. You should (but are not required to) use a descriptive name such as Inspector-agent-autodeploy-Lambda. Remember the name you choose because you will need it in Step 2.
  4. Choose the AWS Lambda role type.
  5. Attach the policies AWSLambdaBasicExecutionRole and AmazonSSMFullAccess.
  6. Choose Create the role to finish.

Step 2 – Create the Lambda function that will run EC2 Systems Manager commands to install the Amazon Inspector agent:

  1. Sign in to the AWS Management Console in your chosen region and navigate to the Lambda console.
  2. Choose Create a Lambda function.
  3. Skip Select blueprint.
  4. On the Configure triggers page, choose Next. Type a Name and Description for the function. Choose Python 2.7 for Runtime.
  5. Download and save autodeploy.py. Unzip the file, and copy the entire contents of autodeploy.py.
  6. From the Code entry type drop-down list, choose Edit code inline, and replace all the existing text with the text that you just copied from autodeploy.py.
  7. From the Role drop-down list, choose Choose an existing role, and then from the Existing role drop-down list, choose the role that you created in Step 1.
  8. Choose Next and then Create function to finish creating the function.

Step 3 – Set up CloudWatch Events to trigger the function:

  1. In the AWS Management Console in the same region as you used in Step 2, navigate to the CloudWatch console and then choose Events in the navigation pane.
  2. Choose Create rule. From the Select event source drop-down list, choose Amazon EC2.
  3. Choose Specific state(s) and Running. This tells CloudWatch to generate an event when an instance enters the Running state.
  4. Under Targets, choose Add target and then Lambda function.
  5. Choose the function that you created in Step 2.
  6. Click Configure details. Type a name and description for the event, and choose Create rule.

Summary

You have completed the setup! Now, whenever an EC2 instance enters the Running state (either on initial creation or on reboot), CloudWatch Events triggers an event that invokes the Lambda function that you created. The Lambda function then uses EC2 System Manager to install the Amazon Inspector agent on the instance.

In a subsequent AWS Security Blog post, I will show you how to take your security assessment automation a step further by automatically performing remediations for Amazon Inspector findings by using EC2 System Manager and Lambda.

If you have comments about this blog post, submit them in the “Comments” section below. If you have implementation questions, start a new thread on the Amazon Inspector forum.

– Eric

How to Detect and Automatically Remediate Unintended Permissions in Amazon S3 Object ACLs with CloudWatch Events

Post Syndicated from Mustafa Torun original https://aws.amazon.com/blogs/security/how-to-detect-and-automatically-remediate-unintended-permissions-in-amazon-s3-object-acls-with-cloudwatch-events/

Amazon S3 Access Control Lists (ACLs) enable you to specify permissions that grant access to S3 buckets and objects. When S3 receives a request for an object, it verifies whether the requester has the necessary access permissions in the associated ACL. For example, you could set up an ACL for an object so that only the users in your account can access it, or you could make an object public so that it can be accessed by anyone.

If the number of objects and users in your AWS account is large, ensuring that you have attached correctly configured ACLs to your objects can be a challenge. For example, what if a user were to call the PutObjectAcl API call on an object that is supposed to be private and make it public? Or, what if a user were to call the PutObject with the optional Acl parameter set to public-read, therefore uploading a confidential file as publicly readable? In this blog post, I show a solution that uses Amazon CloudWatch Events to detect PutObject and PutObjectAcl API calls in near real time and helps ensure that the objects remain private by making automatic PutObjectAcl calls, when necessary.

Note that this process is a reactive approach, a complement to the proactive approach in which you would use the AWS Identity and Access Management (IAM) policy conditions to force your users to put objects with private access (see Specifying Conditions in a Policy for more information). The reactive approach I present in this post is for “just in case” situations in which the change on the ACL is accidental and must be fixed.

Solution overview

The following diagram illustrates this post’s solution:

  1. An IAM or root user in your account makes a PutObjectAcl or PutObject call.
  2. S3 sends the corresponding API call event to both AWS CloudTrail and CloudWatch Events in near real time.
  3. A CloudWatch Events rule delivers the event to an AWS Lambda function.
  4. If the object is in a bucket in which all the objects need to be private and the object is not private anymore, the Lambda function makes a PutObjectAcl call to S3 to make the object private.

Solution diagram

To detect the PutObjectAcl call and modify the ACL on the object, I:

  1. Turn on object-level logging in CloudTrail for the buckets I want to monitor.
  2. Create an IAM execution role to be used when the Lambda function is being executed so that Lambda can make API calls to S3 on my behalf.
  3. Create a Lambda function that receives the PutObjectAcl API call event, checks whether the call is for a monitored bucket, and, if so, ensures the object is private.
  4. Create a CloudWatch Events rule that matches the PutObjectAcl API call event and invokes the Lambda function created in the previous step.

The remainder of this blog post details the steps of this solution’s deployment and the testing of its setup.

Deploying the solution

In this section, I follow the four solution steps outlined in the previous section to use CloudWatch Events to detect and fix unintended access permissions in S3 object ACLs automatically. I start with turning on object-level logging in CloudTrail for the buckets of interest.

I use the AWS CLI in this section. (To learn more about setting up the AWS CLI, see Getting Set Up with the AWS Command Line Interface.) Before you start, make sure your installed AWS CLI is up to date (specifically, you must use version 1.11.28 or newer). You can check the version of your CLI as follows.

$ aws --version

I run the following AWS CLI command to create an S3 bucket named everything-must-be-private. Remember to replace the placeholder bucket names with your own bucket names, in this instance and throughout the post (bucket names are global in S3).

$ aws s3api create-bucket \
--bucket everything-must-be-private

As the bucket name suggests, I want all the files in this bucket to be private. In the rest of this section, I detail above four steps for deploying the solution.

Step 1: Turn on object-level logging in CloudTrail for the S3 bucket

In this step, I create a CloudTrail trail and turn on object-level logging for the bucket, everything-must-be-private. My goal is to achieve the following setup, which are also illustrated in the following diagram:

  1. A requester makes an API call against this bucket.
  2. I let a CloudTrail trail named my-object-level-s3-trail receive the corresponding events.
  3. I log the events to a bucket named bucket-for-my-object-level-s3-trail and deliver them to CloudWatch Events.

Diagram2-012417-MT

First, I create a file named bucket_policy.json and populate it with the following policy. Remember to replace the placeholder account ID with your own account ID, in this instance and throughout the post.

{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Principal": {
            "Service": "cloudtrail.amazonaws.com"
        },
        "Action": "s3:GetBucketAcl",
        "Resource": "arn:aws:s3:::bucket-for-my-object-level-s3-trail"
    }, {
        "Effect": "Allow",
        "Principal": {
            "Service": "cloudtrail.amazonaws.com"
        },
        "Action": "s3:PutObject",
        "Resource": "arn:aws:s3:::bucket-for-my-object-level-s3-trail/AWSLogs/123456789012/*",
        "Condition": {
            "StringEquals": {
                "s3:x-amz-acl": "bucket-owner-full-control"
            }
        }
    }]
}

Then, I run the following commands to create the bucket for logging with the preceding policy so that CloudTrail can deliver log files to the bucket.

$ aws s3api create-bucket \
--bucket bucket-for-my-object-level-s3-trail

$ aws s3api put-bucket-policy \
--bucket bucket-for-my-object-level-s3-trail \
--policy file://bucket_policy.json

Next, I create a trail and start logging on the trail.

$ aws cloudtrail create-trail \
--name my-object-level-s3-trail \
--s3-bucket-name bucket-for-my-object-level-s3-trail

$ aws cloudtrail start-logging \
--name my-object-level-s3-trail

I then create a file named my_event_selectors.json and populate it with the following content.

[{
        "IncludeManagementEvents": false,
        "DataResources": [{
            "Values": [
                 "arn:aws:s3:::everything-must-be-private/"
            ],
            "Type": "AWS::S3::Object"
        }],
        "ReadWriteType": "All"
}]

By default, S3 object-level operations are not logged in CloudTrail. Only bucket-level operations are logged. As a result, I finish my trail setup by creating the event selector shown previously to have the object-level operations logged in those two buckets. Note that I explicitly set IncludeManagementEvents to false because I want only object-level operations to be logged.

$ aws cloudtrail put-event-selectors \
--trail-name my-object-level-s3-trail \
--event-selectors file://my_event_selectors.json

Step 2: Create the IAM execution role for the Lambda function

In this step, I create an IAM execution role for my Lambda function. The role allows Lambda to perform S3 actions on my behalf while the function is being executed. The role also allows CreateLogGroup, CreateLogStream, and PutLogEvents CloudWatch Logs APIs so that the Lambda function can write logs for debugging.

I start with putting the following trust policy document in a file named trust_policy.json.

{
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Principal": {
                "Service": "lambda.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }]
}

Next, I run the following command to create the IAM execution role.

$ aws iam create-role \
--role-name AllowLogsAndS3ACL \
--assume-role-policy-document file://trust_policy.json

I continue by putting the following access policy document in a file named access_policy.json.

{
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectAcl",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::everything-must-be-private/*"
        }, {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        }]
}

Finally, I run the following command to define the access policy for the IAM execution role I have just created.

$ aws iam put-role-policy \
--role-name AllowLogsAndS3ACL \
--policy-name AllowLogsAndS3ACL \
--policy-document file://access_policy.json

Step 3: Create a Lambda function that processes the PutObjectAcl API call event

In this step, I create the Lambda function that processes the event. It also decides whether the ACL on the object needs to be changed, and, if so, makes a PutObjectAcl call to make it private. I start with adding the following code to a file named lambda_function.py.

from __future__ import print_function

import json
import boto3

print('Loading function')

s3 = boto3.client('s3')

bucket_of_interest = "everything-must-be-private"

# For a PutObjectAcl API Event, gets the bucket and key name from the event
# If the object is not private, then it makes the object private by making a
# PutObjectAcl call.
def lambda_handler(event, context):
    # Get bucket name from the event
    bucket = event['detail']['requestParameters']['bucketName']
    if (bucket != bucket_of_interest):
        print("Doing nothing for bucket = " + bucket)
        return
    
    # Get key name from the event
    key = event['detail']['requestParameters']['key']
    
    # If object is not private then make it private
    if not (is_private(bucket, key)):
        print("Object with key=" + key + " in bucket=" + bucket + " is not private!")
        make_private(bucket, key)
    else:
        print("Object with key=" + key + " in bucket=" + bucket + " is already private.")
    
# Checks an object with given bucket and key is private
def is_private(bucket, key):
    # Get the object ACL from S3
    acl = s3.get_object_acl(Bucket=bucket, Key=key)
    
    # Private object should have only one grant which is the owner of the object
    if (len(acl['Grants']) > 1):
        return False
    
    # If canonical owner and grantee ids do no match, then conclude that the object
    # is not private
    owner_id = acl['Owner']['ID']
    grantee_id = acl['Grants'][0]['Grantee']['ID']
    if (owner_id != grantee_id):
        return False
    return True

# Makes an object with given bucket and key private by calling the PutObjectAcl API.
def make_private(bucket, key):
    s3.put_object_acl(Bucket=bucket, Key=key, ACL="private")
    print("Object with key=" + key + " in bucket=" + bucket + " is marked as private.")

Next, I zip the lambda_function.py file into an archive named CheckAndCorrectObjectACL.zip and run the following command to create the Lambda function.

$ aws lambda create-function \
--function-name CheckAndCorrectObjectACL \
--zip-file fileb://CheckAndCorrectObjectACL.zip \
--role arn:aws:iam::123456789012:role/AllowLogsAndS3ACL \
--handler lambda_function.lambda_handler \
--runtime python2.7

Finally, I run the following command to allow CloudWatch Events to invoke my Lambda function for me.

$ aws lambda add-permission \
--function-name CheckAndCorrectObjectACL \
--statement-id AllowCloudWatchEventsToInvoke \
--action 'lambda:InvokeFunction' \
--principal events.amazonaws.com \
--source-arn arn:aws:events:us-east-1:123456789012:rule/S3ObjectACLAutoRemediate

Step 4: Create the CloudWatch Events rule

Now, I create the CloudWatch Events rule that is triggered when an event is received from S3. I start with defining the event pattern for my rule. I create a file named event_pattern.json and populate it with the following code.

{
	"detail-type": [
		"AWS API Call via CloudTrail"
	],
	"detail": {
		"eventSource": [
			"s3.amazonaws.com"
		],
		"eventName": [
			"PutObjectAcl",
			"PutObject"
		],
		"requestParameters": {
			"bucketName": [
				"everything-must-be-private"
			]
		}
	}
}

With this event pattern, I am configuring my rule so that it is triggered only when a PutObjectAcl or a PutObject API call event from S3 via CloudTrail is delivered for the objects in the bucket I choose. I finish my setup by running the following commands to create the CloudWatch Events rule and adding to it as a target the Lambda function I created in the previous step.

$ aws events put-rule \
--name S3ObjectACLAutoRemediate \
--event-pattern file://event_pattern.json

$ aws events put-targets \
--rule S3ObjectACLAutoRemediate \
--targets Id=1,Arn=arn:aws:lambda:us-east-1:123456789012:function:CheckAndCorrectObjectACL

Test the setup

From now on, whenever a user in my account makes a PutObjectAcl call against the bucket everything-must-be-private, S3 will deliver the corresponding event to CloudWatch Events via CloudTrail. The event must match the CloudWatch Events rule in order to be delivered to the Lambda function. Finally, the function checks if the object ACL is expected. If not, the function makes the object private.

For testing the setup, I create an empty file named MyCreditInfo in the bucket everything-must-be-private, and I check its ACL.

$ aws s3api put-object \
--bucket everything-must-be-private \
--key MyCreditInfo

$ aws s3api get-object-acl \
--bucket everything-must-be-private \
--key MyCreditInfo

In the response to my command, I see only one grantee, which is the owner (me). This means the object is private. Now, I add public read access to this object (which is supposed to stay private).

$ aws s3api put-object-acl \
--bucket everything-must-be-private \
--key MyCreditInfo \
--acl public-read

If I act quickly and describe the ACL on the object again by calling the GetObjectAcl API, I see another grantee that allows everybody to read this object.

{
	"Grantee": {
		"Type": "Group",
		"URI": "http://acs.amazonaws.com/groups/global/AllUsers"
	},
	"Permission": "READ"
}

When I describe the ACL again, I see that the grantee for public read access has been removed. Therefore, the file is private again, as it should be. You can also test the PutObject API call by putting another object in this bucket with public read access.

$ aws s3api put-object \
--bucket everything-must-be-private \
--key MyDNASequence \
--acl public-read

Conclusion

In this post, I showed how you can detect unintended public access permissions in the ACL of an S3 object and how to revoke them automatically with the help of CloudWatch Events. Keep in mind that object-level S3 API call events and Lambda functions are only a small set of the events and targets that are available in CloudWatch Events. To learn more, see Using CloudWatch Events.

If you have comments about this blog post, submit them in the “Comments” section below. If you have questions about this post or how to implement the solution described, please start a new thread on the CloudWatch forum.

– Mustafa

Month in Review: December 2016

Post Syndicated from Derek Young original https://aws.amazon.com/blogs/big-data/month-in-review-december-2016/

Another month of big data solutions on the Big Data Blog.

Take a look at our summaries below and learn, comment, and share. Thank you for reading!

Implementing Authorization and Auditing using Apache Ranger on Amazon EMR
Apache Ranger is a framework to enable, monitor, and manage comprehensive data security across the Hadoop platform. Features include centralized security administration, fine-grained authorization across many Hadoop components (Hadoop, Hive, HBase, Storm, Knox, Solr, Kafka, and YARN) and central auditing. In this post, walk through the steps to enable authorization and audit for Amazon EMR clusters using Apache Ranger.

Amazon Redshift Engineering’s Advanced Table Design Playbook
Amazon Redshift is a fully managed, petabyte scale, massively parallel data warehouse that offers simple operations and high performance. In practice, the best way to improve query performance by orders of magnitude is by tuning Amazon Redshift tables to better meet your workload requirements. This five-part blog series will guide you through applying distribution styles, sort keys, and compression encodings and configuring tables for data durability and recovery purposes.

Interactive Analysis of Genomic Datasets Using Amazon Athena
In this post, learn to prepare genomic data for analysis with Amazon Athena. We’ll demonstrate how Athena is well-adapted to address common genomics query paradigms using the Thousand Genomes dataset hosted on Amazon S3, a seminal genomics study. Although this post is focused on genomic analysis, similar approaches can be applied to any discipline where large-scale, interactive analysis is required.

Joining and Enriching Streaming Data on Amazon Kinesis
In this blog post, learn three approaches for joining and enriching streaming data on Amazon Kinesis Streams by using Amazon Kinesis Analytics, AWS Lambda, and Amazon DynamoDB.

Using SaltStack to Run Commands in Parallel on Amazon EMR
SaltStack is an open source project for automation and configuration management. It started as a remote execution engine designed to scale to many machines while delivering high-speed execution. You can now use the new bootstrap action that installs SaltStack on Amazon EMR. It provides a basic configuration that enables selective targeting of the nodes based on instance roles, instance groups, and other parameters.

Building an Event-Based Analytics Pipeline for Amazon Game Studios’ Breakaway
Amazon Game Studios’ new title Breakaway is an online 4v4 team battle sport that delivers fast action, teamwork, and competition. In this post, learn the technical details of how the Breakaway team uses AWS to collect, process, and analyze gameplay telemetry to answer questions about arena design.

Respond to State Changes on Amazon EMR Clusters with Amazon CloudWatch Events
With new support for Amazon EMR in Amazon CloudWatch Events, you can be notified quickly and programmatically respond to state changes in your EMR clusters. Additionally, these events are also displayed in the Amazon EMR console. CloudWatch Events allows you to create filters and rules to match these events and route them to Amazon SNS topics, AWS Lambda functions, Amazon SQS queues, streams in Amazon Kinesis Streams, or built-in targets.

Run Jupyter Notebook and JupyterHub on Amazon EMR
Data scientists who run Jupyter and JupyterHub on Amazon EMR can use Python, R, Julia, and Scala to process, analyze, and visualize big data stored in Amazon S3. Jupyter notebooks can be saved to S3 automatically, so users can shut down and launch new EMR clusters, as needed. See how EMR makes it easy to spin up clusters with different sizes and CPU/memory configurations to suit different workloads and budgets.

Derive Insights from IoT in Minutes using AWS IoT, Amazon Kinesis Firehose, Amazon Athena, and Amazon QuickSight
In this post, see how you can build a business intelligence capability for streaming IoT device data using AWS serverless and managed services. You can be up and running in minutes―starting small, but able to easily grow to millions of devices and billions of messages.

Serving Real-Time Machine Learning Predictions on Amazon EMR
The typical progression for creating and using a trained model for recommendations falls into two general areas: training the model and hosting the model. Model training has become a well-known standard practice. In this post, we highlight one way to host those recommendations using Amazon EMR with JobServer

Powering Amazon Redshift Analytics with Apache Spark and Amazon Machine Learning
In this post, learn to generate a predictive model for flight delays that can be used to help pick the flight least likely to add to your travel stress. To accomplish this, you’ll use Apache Spark running on Amazon EMR for extracting, transforming, and loading (ETL) the data, Amazon Redshift for analysis, and Amazon Machine Learning for creating predictive models.

FROM THE ARCHIVE

Running sparklyr – RStudio’s R Interface to Spark on Amazon EMR
Sparklyr is an R interface to Spark that allows users to use Spark as the backend for dplyr, one of the most popular data manipulation packages. Sparklyr provides interfaces to Spark packages and also allows users to query data in Spark using SQL and develop extensions for the full Spark API. This short post shows you how to run RStudio and sparklyr on EMR.

Want to learn more about Big Data or Streaming Data? Check out our Big Data and Streaming data educational pages.

Leave a comment below to let us know what big data topics you’d like to see next on the AWS Big Data Blog.

How to Monitor AWS Account Configuration Changes and API Calls to Amazon EC2 Security Groups

Post Syndicated from Jeff Levine original https://aws.amazon.com/blogs/security/how-to-monitor-aws-account-configuration-changes-and-api-calls-to-amazon-ec2-security-groups/

You can use AWS security controls to detect and mitigate risks to your AWS resources. The purpose of each security control is defined by its control objective. For example, the control objective of an Amazon VPC security group is to permit only designated traffic to enter or leave a network interface. Let’s say you have an Internet-facing ecommerce website, and your security administrator has determined that only HTTP (TCP port 80) and HTTPS (TCP 443) traffic should be allowed access to the public subnet. As a result, your administrator configures a security group to meet this control objective.

What if, though, someone were to inadvertently change this security group’s rules and enable FTP or other protocols to access the public subnet from any location on the Internet? That expanded access could weaken the security posture of your assets. Consequently, your administrator might need to monitor the integrity of your company’s security controls so that the controls maintain their desired effectiveness.

In this blog post, I explore two methods for detecting unintended changes to VPC security groups. The two methods address not only control objectives but also control failures.

The two methods and when to use them

In this post, Method 1 uses AWS Config to monitor changes to a security group’s configuration as part of an organization’s overall compliance auditing program. Method 1 views a change to a VPC security group as a compliance risk. Use this method when you want to bolster your company’s compliance management.

Method 2 uses AWS CloudTrail and Amazon CloudWatch Events to identify AWS API calls that could change the configurations of VPC security groups. Method 2 views a change to a VPC security group as a potential security incident that should be identified in near real time. Use this method when you want to support your company’s monitoring of security operations.

Both of these methods can be effective parts of a defense-in-depth approach to control monitoring.

Method 1: Use AWS Config to check the configuration of a security group

The first method uses AWS Config, a fully managed service that provides you with an AWS resource inventory, configuration history, and configuration change notifications to enable security and governance. You can create AWS Config rules that automatically check the configuration of AWS resources that are recorded by AWS Config. For this example, I use a Config rule that is invoked whenever a change is made to a security group. Attach the Config rule to an AWS Lambda function that examines the ingress rules of a security group to see if the group remains in compliance with the rules.

The following Lambda function code defines a list named REQUIRED_PERMISSIONS with elements that represent a protocol, port range, and IP range that together define a security permission. This JSON notation is identical to what you would use when creating a security group with the AWS EC2 authorize-security-group-ingress command.

REQUIRED_PERMISSIONS = [
{
    "IpProtocol" : "tcp",
    "FromPort" : 80,
    "ToPort" : 80,
    "UserIdGroupPairs" : [],
    "IpRanges" : [{"CidrIp" : "0.0.0.0/0"}],
    "PrefixListIds" : []
},
{
    "IpProtocol" : "tcp",
    "FromPort" : 443,
    "ToPort" : 443,
    "UserIdGroupPairs" : [],
    "IpRanges" : [{"CidrIp" : "0.0.0.0/0"}],
    "PrefixListIds" : []
}]

In this function, the first list entry allows TCP port 80 (HTTP) from anywhere on the internet (0.0.0.0/0). The second list entry does the same for TCP port 443 (HTTPS).

The Lambda function then examines the configuration event that was passed to it as a result of the configuration change. The configuration event contains a list variable named ip_permissions that comes from the IpPermissions field of the Config event payload. The list represents the permissions of the security group after the configuration change is made. The function then examines the new permissions to see if they match the permissions listed in the preceding REQUIRED_PERMISSIONS list, as shown in the following code.

authorize_permissions =
    [item for item in REQUIRED_PERMISSIONS
        if item not in ip_permissions]
revoke_permissions =
    [item for item in ip_permissions
        if item not in REQUIRED_PERMISSIONS]

The authorize_permissions list contains the permissions that are in the REQUIRED_PERMISSIONS list that are not in the configuration event’s permission list. Thus, authorize_permissions represents the permissions that must be authorized (added) to the security group to bring it into alignment with the REQUIRED_PERMISSIONS list.

The revoke_permissions list contains the permissions that are in the configuration event’s permission list but not in the REQUIRED_PERMISSIONS list. Therefore, the revoke_permissions list represents the permissions that need to be revoked (removed) from the security group to bring it into alignment with the REQUIRED_PERMISSIONS list.

The function then performs the necessary authorizations and revocations using the authorize_security_group_ingress() and revoke_security_group_ingress() API calls. All of the results are logged by Amazon CloudWatch Logs.

To implement Method 1, follow these steps:

  1. Create a Lambda execution role and an AWS Config role.
  2. Enable AWS Config to record the configuration of security groups so that AWS Config can monitor security groups for changes to their configurations. The following screenshot shows an example of the settings.
    Screenshot showing an example of the configuration settings
  3. Create a security group that enables ingress TCP ports 80, 443, 465, and 993. You will use this security group only for the purposes of this blog post. The group should appear as the following screenshot shows.
    Screenshot of inbound rules of the security group
  4. Create the Python Lambda function using the code from the AWS Config rules library on GitHub.
  5. Create the AWS Config rule using the Lambda function you created in Step 4. For Trigger type, choose Configuration changes. For Scope of changes, choose EC2: SecurityGroup, and then type the ID of the security group you created in Step 3. The following screenshot shows these configuration settings.
    Screenshot of the trigger's configuration settings
  6. Run the Config rule. This will queue the rule for execution, and the rule should run to completion in about 10 minutes.
  7. Check the security group you created in Step 3. You should see that only ingress TCP ports 80 and 443 remain, as shown in the following screenshot.
    Screenshot showing that only ports 80 and 443 remain

The use of AWS Config in Method 1 allows for the configuration of a security group to be tracked along with other AWS resources. Changes to the security group’s configuration are reported during the next Config compliance evaluation, typically within 10 minutes. The notifications of changes to your security groups can be used to support your organization’s compliance management program.

Method 2: Use CloudTrail and CloudWatch Events to monitor API calls

In the first method, I approached the solution from the perspective of the object being changed—in this case, a security group. In the second method, I will focus on the “actors,” namely the API calls that may attempt to change the security group. By using CloudTrail with CloudWatch Events, you can invoke a Lambda function when specific API calls such as authorize_security_group_ingress() and revoke_security_group_ingress() are made. The API call event payload contains an IpPermission list that you can scan, as you did in Method 1.

When you create the Lambda function for CloudWatch Events, you can create an event selector, which functions as a filter for the Lambda function so that the function is only invoked for specific events. This approach has two advantages: it avoids unnecessary calls to Lambda, and you can simplify your Python code to handle only the desired events. Consider the following event selector.

{
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "ec2.amazonaws.com"
    ],
    "eventName": [
      "AuthorizeSecurityGroupIngress",
      "RevokeSecurityGroupIngress"
    ],
    "requestParameters": {
      "groupId": [
        "sg-abc12345"
      ]
    }
  }
}

This selector looks for CloudTrail API events (AWS API Call via CloudTrail) that involve the two API calls we previously discussed with the security group you wish to examine (in this case, sg-abc12345).

The function code for Method 2 differs from that of Method 1 in one important way: the Lambda function for Method 2 does not make any changes to the security group. The reason for this difference is that the code would need to use the same APIs to adjust the security group that triggered the Lambda function in the first place, potentially resulting in recursion.

To implement Method 2, follow these steps:

  1. Create a Lambda execution role and policy.
  2. Create the Python Lambda function using the code from AWS Labs.
  3. Create a security group with no ingress permissions (meaning no TCP or UDP ports). Use this security group only for the purposes of this blog post, and do not attach this group to a resource.
  4. Enable CloudTrail.
  5. Create a CloudWatch Events rule that is triggered by API calls. Use an event selector as described previously in this post as well as the Lambda function you created in Step 2.
  6. Add a rule to the security group you created in Step 3 of Method 2 to allow TCP port 445 inbound. TCP port 445 has no special significance; any TCP ports other than 80 and 443 would work.
    Screenshot showing the inbound rule that allows port 445
  7. After a few minutes, you should see a message in CloudWatch Logs telling you that TCP ports 80 and 443 must be authorized and that TCP port 445 must be revoked, as shown in the following screenshot.
    Screenshot of the CloudWatch Logs message saying which ports must be authorized or revoked

The use of CloudTrail and CloudWatch Events in Method 2 allows for the near real-time detection of API calls that could change the configuration of a VPC security group. The notifications of changes are posted to CloudWatch Logs, providing useful information to your organization’s security operations management program.

Summary

You can layer Config, CloudTrail, and CloudWatch Events on top of Amazon VPC security groups to provide a defense-in-depth approach to security. Though VPC security groups provide critical filtering capabilities, Config rules, CloudTrail, and CloudWatch Events take the protection to a deeper level by monitoring security groups and notifying you of potentially unintended changes.

Whether you use Method 1 or Method 2 depends on your goals. If you are focused on compliance management, Method 1 enables you to incorporate the configuration of security groups into your organization’s compliance management program. If your concern is more about incident detection, Method 2 offers a faster way to detect changes to a security group’s configuration. Both methods can help you add security to your AWS infrastructure.

If you have comments about this blog post, submit them in the “Comments” section below. If you have questions about implementing this post’s two methods, start a new thread on the VPC forum.

– Jeff

Respond to State Changes on Amazon EMR Clusters with Amazon CloudWatch Events

Post Syndicated from Jonathan Fritz original https://aws.amazon.com/blogs/big-data/respond-to-state-changes-on-amazon-emr-clusters-with-amazon-cloudwatch-events/

Jonathan Fritz is a Senior Product Manager for Amazon EMR

Customers can take advantage of the Amazon EMR API to create and terminate EMR clusters, scale clusters using Auto Scaling or manual resizing, and submit and run Apache Spark, Apache Hive, or Apache Pig workloads. These decisions are often triggered from cluster state-related information.

Previously, you could use the “describe” and “list” set of API operations to find the relevant information about your EMR clusters and associated instance groups, steps, and Auto Scaling policies. However, programmatic applications that check resource state changes and post notifications or take actions are forced to poll these API operations, which provides a slower end-to-end reaction time and additional management overhead than if you were able to use an event-driven architecture.

With new support for Amazon EMR in Amazon CloudWatch Events, you can be notified quickly and programmatically respond to state changes in your EMR clusters. Additionally, these events are also displayed in the Amazon EMR console, on the Cluster Details page in the Events section.

There are four new EMR event types:

  • Cluster State Change
  • Instance Group State Change
  • Step State Change
  • Auto Scaling State Change

CloudWatch Events allows you to create filters and rules to match these events and route them to Amazon SNS topics, AWS Lambda functions, Amazon SQS queues, streams in Amazon Kinesis Streams, or built-in targets. You then have the ability to programmatically act on these events, including sending emails and SMS messages, running retry logic in Lambda, or tracking the state of running steps. For more information about the sample events generated for each event type, see the CloudWatch Events documentation.

The following is an example using the CloudWatch Events console to route EMR step failure events to Lambda for automated retry logic and to SNS to push a notification to an email alias:

Cloudwatch_1

You can create rules for EMR event types in the CloudWatch Events console, AWS CLI, or the AWS SDKs using the CloudWatch Events API.

If you have any questions or would like to share an interesting use case about events and notifications with EMR, please leave a comment below.

Related

Dynamically Scale Applications on Amazon EMR with Auto Scaling

AutoScaling_SocialMedia

 

Friday, December 2: Security and Compliance Sessions Today at re:Invent

Post Syndicated from Craig Liebendorfer original https://aws.amazon.com/blogs/security/friday-december-2-security-and-compliance-sessions-today-at-reinvent/

re:Invent stage photo

Today, the following security and compliance sessions will be presented at AWS re:Invent 2016 in Las Vegas (all times local). See the re:Invent Session Catalog for complete information about every session. You can also download the AWS re:Invent 2016 Event App for the latest updates and information.

If you are not attending re:Invent 2016, keep in mind that all videos of and slide decks from these sessions will be made available next week. We will publish a post on the Security Blog next week that links to all videos and slide decks from security and compliance sessions.

9:00 A.M.

9:30 A.M.

10:30 A.M.

11:00 A.M.

12:30 P.M.

– Craig

Blox – New Open Source Scheduler for Amazon EC2 Container Service

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/blox-new-open-source-scheduler-for-amazon-ec2-container-service/

Back in 2014 I talked about Amazon ECS and showed you how it helps you to build, run, and scale Docker-based applications. I talked about the three scheduling options (automated, manual, and custom)  and described how a scheduler works to assign tasks to instances.

At the time  that I wrote that post, your custom scheduler had to call the ListContainerInstances and DescribeContainerInstances functions on a frequent basis in order to discover the current state of the cluster. A few weeks ago we simplified the process of tracking the state of each cluster by adding support for Amazon CloudWatch Events (read Monitor Cluster State with Amazon ECS Event Stream to learn more about how this works).

With this new event stream in place, we want to make it even easier for you to create custom schedulers.

Today we are launching Blox. This new open source project includes a service that consumes the event stream, uses it to track the state of the cluster, and makes the state accessible via a set of REST APIs. The package also includes a daemon scheduler that runs one copy of a task on each container instance in a cluster. This one-per-container model supports workloads that process logs and collect metrics.

Here’s a block diagram (no pun intended):

This is an open source project; we are looking forward to your pull requests and feature proposals.

To learn more, read Introducing Blox From Amazon EC2 Container Service.

— Jeff;

 

Introducing Blox from Amazon EC2 Container Service

Post Syndicated from Chris Barclay original https://aws.amazon.com/blogs/compute/introducing-blox-from-amazon-ec2-container-service/

Today we are announcing Blox, a new open source project from the Amazon ECS team that enables users to build custom schedulers and other tooling on top of ECS. Our goal with Blox is to provide tools that simplify the creation of custom schedulers, dashboards and other extensions, so that customers can meet the needs of their specific use cases.

ECS recently announced the availability of an event stream that delivers ECS container instance and task state changes to Amazon CloudWatch Events. Customers that build scheduling workflows often need to consume the events generated in the ECS cluster, persist this state locally and operate on the local cluster state. Blox includes a cluster-state-service that provides this functionality and offers REST APIs on top of the local cluster state. Blox is targeted at developers that want to build custom schedulers or processes that need the current state of resources in the ECS cluster and developers that want to take action based on cluster events.

Blox also ships with a daemon-scheduler that supports launching one and only one copy of a task across all container instances in an ECS cluster. The scheduler monitors for new container instances joining the cluster and will place the task on them. Blox daemon-scheduler enables tasks like log agents and metric collection agents to run on ECS clusters.

We are excited to release Blox as open source software and plan to build an ecosystem of tools around ECS. If you are interested in using or contributing to Blox, come visit the Blox GitHub repository. We are tracking a number of feature proposals that we are evaluating for the roadmap. We invite you to come participate in our GitHub repository and help identify and prioritize improvements.

Deploying Blox

Blox Deployment on a Local Environment

Our recommended way for getting started with Blox is to deploy the framework on your local Docker installation. Blox offers a Docker Compose file that enables deployment in local environments. This allows you to get started with building custom schedulers using the cluster-state-service.

Here is the Blox architecture when run locally:

  • ECS pushes the cluster state changes as CloudWatch events.
  • CloudWatch events is configured to send to the SQS Queue.
  • Blox cluster-state-service consumes these events and recreates and stores the cluster state locally and offers REST APIs.
  • Blox daemon-scheduler uses the cluster-state-service APIs to track container instances in ECS cluster and launch tasks on them.

Step 1: Create SQS Queue and Configure CloudWatch Events to send ECS events to the SQS Queue

Blox depends on an ECS event stream that is delivered via CloudWatch events. In order to use Blox, you need to create an SQS queue and configure CloudWatch to deliver the ECS events to this SQS queue. Blox provides a pre-built AWS CloudFormation template that will deploy and configure the required Amazon AWS components. Once you have pulled the CloudFormation template from the Blox repository, run the following command using the AWS CLI:

$ aws --region  cloudformation create-stack --stack-name BloxLocal --template-body file://cloudformation_template.json

In a few minutes, the CloudFormation template will finish setting up the CloudWatch event and the SQS queue and you will be ready to deploy Blox.

Step 2: Launch Blox

Next, download the Docker Compose file from the Blox repo. Before launching Blox, you will first need to update docker-compose.yml with the following changes:

  • Update the AWS_REGION value with the region of your ECS and SQS resources.
  • Update the AWS_PROFILE value with your profile name in ~/.aws/credentials. You can skip this step if you are using the default profile.

After you have updated docker-compose.yml, you can use the following commands to launch the Blox containers on your local Docker environment.

# From the folder where you downloaded docker-compose.yml
$ docker-compose up –d
$ docker-compose ps

You will see output that shows the Blox cluster-state-service, daemon-scheduler, and etcd storage:

Name             Command                          State   Ports
-----------------------------------------------------------------------------
etcd_1        /usr/local/bin/etcd --data ...   Up      2379/tcp, 2380/tcp
scheduler_1   --bind 0.0.0.0:2000 --css- ...   Up      0.0.0.0:2000->2000/tcp
css_1         --bind 0.0.0.0:3000 --etcd ...   Up      3000/tcp

You have now completed the local installation of Blox. You can begin consuming the Scheduler API at http://localhost:2000/.

Using the daemon-scheduler

The daemon-scheduler uses the following concepts:

  • An environment represents the configuration for desired state of the tasks to be maintained. For daemon-scheduler, the environment indicates the task definition to launch in a specific cluster.
  • A deployment is the operation that brings the environment into existence. A deployment indicates to the scheduler that the desired configuration state in the environment should be established in the cluster.

Step 1: Create an ECS cluster

If you don’t have an ECS cluster, follow our Create Cluster guide.

Step 2: Register Task Definition

In order to launch tasks in ECS cluster, you need to register a task definition with ECS. Here is a Task definition you can use, if you don’t have one already.

$ cat > /tmp/nginx.json << EOF
 {
    "family": "nginx",
    "containerDefinitions": [{
        "name": "nginx",
        "image": "nginx",
        "cpu": 1024,
        "memory": 128
    } ]
}
EOF

$ aws ecs register-task-definition --cli-input-json file:///tmp/nginx.json

Query the ARN for the nginx task definition. You need this for the next step.

$ aws ecs list-task-definitions
{
   "taskDefinitionArns": [
        "arn:aws:ecs:us-west-2:<your-account-id>:task-definition/nginx:1"
    ]
}

Launch Daemon workloads using the daemon-scheduler

For this exercise, we will be using the demo-cli that Blox provides to interact with the scheduler. Please consult the Blox GitHub repository regarding the APIs that the daemon-scheduler exposes.

Step 3: Create an environment
Create an environment by replacing the cluster name and task definition ARN in the following command:

./blox-create-environment.py --environment TestEnvironment --cluster <MyClusterName> --task-definition <task-def-arn>

Sample output:

{
  "items": [
    {
      "deploymentToken": "17fb6b8b-abf3-4e7b-b9f4-fdb431d53887",
      "health": "healthy",
      "name": "releaseenvironment",
      "instanceGroup": {
        "cluster": "arn:aws:ecs:us-west-2:203719379804:cluster/BloxTestCluster-1123-2"
      }
    }
  ]
}

Upon successful creation of the environment, the daemon-scheduler response will have a deploymentToken that will be used in our next step.

Step 4: Create a Deployment

In order to bring this environment into existence in your ECS cluster, you need to perform a deployment operation:

./blox-create-deployment.py --environment TestEnvironment --deploymentToken <deploymentToken>

Creating a deployment will result in the scheduler launching the task definition attached to the environment across all the container instances in your cluster. You can now go to the ECS console and check out the tasks running on your container instances. You have now successfully used the daemon-scheduler to launch daemon workloads in your ECS cluster.

Blox Deployment on AWS

Blox can also be deployed on AWS. Use the Blox CloudFormation template to create:

  • A new ECS cluster with one container instance on which the Blox components are setup as an ECS service with cluster-state-service, daemon-scheduler and etcd containers making up a single task.
  • An Application Load Balancer is created in front of the daemon-scheduler endpoint.
  • An API Gateway is set up as the public facing frontend to Blox and provides the authentication mechanism. This API Gateway can be used to reach the scheduler and manage tasks on the ECS cluster.
  • A Lambda function that acts as a simple proxy enables the public facing API Gateway endpoint to forward requests onto the ALB listener in the VPC.

This Blox deployment can then be used to manage ECS clusters associated with the account.

See the instructions in our GitHub repo for the steps to configure this option.

Available now

Blox is available now. To learn more, see the Blox documentation in our GitHub repo.

Thursday, December 1: Security and Compliance Sessions Today at re:Invent

Post Syndicated from Craig Liebendorfer original https://aws.amazon.com/blogs/security/thursday-december-1-security-and-compliance-sessions-today-at-reinvent/

re:Invent stage photo

Today, the following security and compliance sessions will be presented at AWS re:Invent 2016 in Las Vegas (all times local). See the re:Invent Session Catalog for complete information about every session. You can also download the AWS re:Invent 2016 Event App for the latest updates and information.

If you are not attending re:Invent 2016, keep in mind that all videos of and slide decks from these sessions will be made available next week. We will publish a post on the Security Blog next week that links to all videos and slide decks from security and compliance sessions.

11:00 A.M.

11:30 A.M.

12:30 P.M.

1:00 P.M.

2:00 P.M.

2:30 P.M.

3:30 P.M.

4:00 P.M.

– Craig