Tag Archives: AWS CloudFormation

Continue Rolling Back an Update for AWS CloudFormation stacks in the UPDATE_ROLLBACK_FAILED state

Post Syndicated from Anil Kumar original http://blogs.aws.amazon.com/application-management/post/Tx11YT4MHFDZMK6/Continue-Rolling-Back-an-Update-for-AWS-CloudFormation-stacks-in-the-UPDATE-ROLL

AWS CloudFormation allows developers and systems administrators to create and manage a collection of related AWS resources (called a stack) by provisioning and updating them in an orderly and predictable way.

In this blog post, we will describe a new feature in AWS CloudFormation that will allow continue rolling back an update in a self-service manner for stacks in UPDATE_ROLLBACK_FAILED state.

A stack’s state is set to UPDATE_ROLLBACK_FAILED when CloudFormation cannot roll back all changes during an update. For example, if you manually deleted a resource outside of CloudFormation, the service’s attempt to roll back to the original state will fail and the stack’s state will be set to UPDATE_ROLLBACK_FAILED. Because CloudFormation does not know a change (in this case, the deletion of a resource) was made, it assumes the resource still exists and attempts to roll back to it, causing the update rollback to fail. There are other reasons an update rollback might fail, including an AWS service limit violation or because a dependent resource did not stabilize. For more information about common errors, see the AWS CloudFormation User Guide.

Until recently, after a stack was in the UPDATE_ROLLBACK_FAILED state, you had only two options: delete the stack or contact AWS Support to return the stack to a working state. In many cases (for example, if it is running production workloads), deleting the stack is not an acceptable option.

AWS CloudFormation now offers a third option: continue update rollback, which you can initiate from the AWS CloudFormation console or with the continue-update-rollback command in the AWS Command Line Interface (CLI). This functionality is enabled for all the stacks in the UPDATE_ROLLBACK_FAILED state.

For example, if the stack is in the UPDATE_ROLLBACK_FAILED state due to a service limit violation, you can request a limit increase or delete resources to stay within the limit and then use the continue update rollback functionality to reinitiate the rollback and bring the stack to the UPDATE_ROLLBACK_COMPLETE state.

In the AWS CloudFormation console, select the stack, choose Actions, and then choose Continue Update Rollback.

The use of the continue update rollback functionality means you do not need to contact AWS Support. In situations where you can find and fix the error, the continue update rollback process will let you recover the stack to the UPDATE_ROLLBACK_COMPLETE state so you can run updates again. You can however still contact the support if this option does not recover the stack or if you need any further assistance.  

For more information about update rollback failures, see the AWS CloudFormation User Guide.

We hope you find this blog post helpful. Feel free to leave feedback in the comments or in our user forum.

Continue Rolling Back an Update for AWS CloudFormation stacks in the UPDATE_ROLLBACK_FAILED state

Post Syndicated from Anil Kumar original http://blogs.aws.amazon.com/application-management/post/Tx11YT4MHFDZMK6/Continue-Rolling-Back-an-Update-for-AWS-CloudFormation-stacks-in-the-UPDATE-ROLL

AWS CloudFormation allows developers and systems administrators to create and manage a collection of related AWS resources (called a stack) by provisioning and updating them in an orderly and predictable way.

In this blog post, we will describe a new feature in AWS CloudFormation that will allow continue rolling back an update in a self-service manner for stacks in UPDATE_ROLLBACK_FAILED state.

A stack’s state is set to UPDATE_ROLLBACK_FAILED when CloudFormation cannot roll back all changes during an update. For example, if you manually deleted a resource outside of CloudFormation, the service’s attempt to roll back to the original state will fail and the stack’s state will be set to UPDATE_ROLLBACK_FAILED. Because CloudFormation does not know a change (in this case, the deletion of a resource) was made, it assumes the resource still exists and attempts to roll back to it, causing the update rollback to fail. There are other reasons an update rollback might fail, including an AWS service limit violation or because a dependent resource did not stabilize. For more information about common errors, see the AWS CloudFormation User Guide.

Until recently, after a stack was in the UPDATE_ROLLBACK_FAILED state, you had only two options: delete the stack or contact AWS Support to return the stack to a working state. In many cases (for example, if it is running production workloads), deleting the stack is not an acceptable option.

AWS CloudFormation now offers a third option: continue update rollback, which you can initiate from the AWS CloudFormation console or with the continue-update-rollback command in the AWS Command Line Interface (CLI). This functionality is enabled for all the stacks in the UPDATE_ROLLBACK_FAILED state.

For example, if the stack is in the UPDATE_ROLLBACK_FAILED state due to a service limit violation, you can request a limit increase or delete resources to stay within the limit and then use the continue update rollback functionality to reinitiate the rollback and bring the stack to the UPDATE_ROLLBACK_COMPLETE state.

In the AWS CloudFormation console, select the stack, choose Actions, and then choose Continue Update Rollback.

The use of the continue update rollback functionality means you do not need to contact AWS Support. In situations where you can find and fix the error, the continue update rollback process will let you recover the stack to the UPDATE_ROLLBACK_COMPLETE state so you can run updates again. You can however still contact the support if this option does not recover the stack or if you need any further assistance.  

For more information about update rollback failures, see the AWS CloudFormation User Guide.

We hope you find this blog post helpful. Feel free to leave feedback in the comments or in our user forum.

Optimize AWS CloudFormation Templates

Post Syndicated from Elliot Yamaguchi original http://blogs.aws.amazon.com/application-management/post/Tx3NFBN5FT8NA1X/Optimize-AWS-CloudFormation-Templates

The following post is by guest blogger Julien Lépine, Solutions Architect at AWS. He explains how to optimize templates so that AWS CloudFormation quickly deploys your environments.

______________________________________________________________________________________

Customers sometimes ask me if there’s a way to optimize large AWS CloudFormation templates, which can take several minutes to deploy a stack. Often stack creation is slow because one resource depends on the availability of another resource before it can be provisioned. Examples include:

A front-end web server that has a dependency on an application server

A service that waits for another remote service to be available

In this post, I describe how to speed up stack creation when resources have dependencies on other resources.

Note: I show how to launch Windows instances with Windows PowerShell, but you can apply the same concepts to Linux instances launched with shell scripts.

How CloudFormation Creates Stacks

When CloudFormation provisions two instances, it provisions them randomly. Defining one resource before another in a template doesn’t guarantee that CloudFormation will provision that resource first. You need to explicitly tell CloudFormation the right order for instance provisioning.

To demonstrate how to do this, I’ll start with the following CloudFormation template:

{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description": "This is a demonstration AWS CloudFormation template containing two instances",
"Parameters": {
"ImageId" : {
"Description": "Identifier of the base Amazon Machine Image (AMI) for the instances in this sample (please use Microsoft Windows Server 2012 R2 Base)",
"Type" : "AWS::EC2::Image::Id"
},
"InstanceType" : {
"Description": "EC2 instance type to use for the instances in this sample",
"Type" : "String"
},
},
"Resources" : {
"Instance1": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": { "Ref" : "ImageId" },
"InstanceType": { "Ref": "InstanceType" },
}
},

"Instance2": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": { "Ref" : "ImageId" },
"InstanceType": { "Ref": "InstanceType" },
}
}
}
}

CloudFormation would likely create the stack in the following sequence:

This is fast, but if Instance2 is dependent on Instance1, you would ordinarily need to hard code or script the provisioning sequence to ensure that Instance1 is provisioned first.

Specifying Dependencies

When you need CloudFormation to wait to provision one resource until another one has been provisioned, you can use the DependsOn attribute.

"Instance2": {
"DependsOn": ["Instance1"]
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": { "Ref" : "ImageId" },
"InstanceType": { "Ref": "InstanceType" }
}
}

You can also introduce references between elements by using either the { "Ref": "MyResource" } or the { "Fn::GetAtt" : [ "MyResource" , "MyAttribute" ] } functions. When you use one of these functions, CloudFormation behaves as if you’ve added a DependsOn attribute to the resource. In the following example, the identifier of Instance1 is used in a tag for Instance2.

"Instance2": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": { "Ref" : "ImageId" },
"InstanceType": { "Ref": "InstanceType" },
"Tags": [ { "Key" : "Dependency", "Value" : { "Ref": "Instance1" } } ]
}
}

Both methods of specifying dependencies result in the same sequence:

Now, CloudFormation waits for Instance1 to be provisioned before provisioning Instance2. But I’m not guaranteed that services hosted on Instance1 will be available, so I will have to address that in the template.

Note that instances are provisioned quickly in CloudFormation. In fact, it happens in the time it takes to call the RunInstances Amazon Elastic Compute Cloud (EC2) API. But it takes longer for an instance to fully boot than it does to provision the instance.

Using Creation Policies to Wait for On-Instance Configurations

In addition to provisioning the instances in the right order, I want to ensure that a specific setup milestone has been achieved inside Instance1 before contacting it. To do this, I use a CreationPolicy attribute. A CreationPolicy is an attribute you can add to an instance to prevent it from being marked CREATE_COMPLETE until it has been fully initialized.

In addition to adding the CreationPolicy attribute, I want to ask Instance1 to notify CloudFormation after it’s done initializing. I can do this in the instance’s UserData section. On Windows instances, I can use this section to execute code in batch files or in Windows PowerShell in a process called bootstrapping.

I’ll execute a batch script, then tell CloudFormation that the creation process is done by sending a signal specifying that Instance1 is ready. Here’s the code with a CreationPolicy attribute and a UserData section that includes a script that invokes cfn-signal.exe:

    "Instance1": {
      "Type": "AWS::EC2::Instance",
      "CreationPolicy" : {
        "ResourceSignal" : {
          "Timeout": "PT15M",
          "Count"  : "1"
        }
      },
      "Properties": {
        "ImageId": { "Ref" : "ImageId" },
        "InstanceType": { "Ref": "InstanceType" },
        "UserData": {
          "Fn::Base64": {
            "Fn::Join": [ "n", [
                "<script>",

                "REM …Do any instance configuration steps deemed necessary…",

                { "Fn::Join": ["", [ "cfn-signal.exe -e 0 –stack "", { "Ref": "AWS::StackName" }, "" –resource "Instance1" –region "", { "Ref" : "AWS::Region" }, """ ] ] },
                "</script>"
            ] ]
          }
        }
      }
    }

I don’t need to change the definition of Instance2 because it’s already coded to wait for Instance1. I now know that Instance1 will be completely set up before Instance2 is provisioned. The sequence looks like this:

Optimizing the Process with Parallel Provisioning

It takes only a few seconds to provision an instance in CloudFormation, but it can take several minutes for an instance to boot and be ready because it must wait for the complete OS boot sequence, activation and the execution of the UserData scripts. As we saw in the figures, the time it takes to create the complete CloudFormation stack is about twice the boot and initialization time for a resource. Depending on the complexity of our processes, booting can take up to 10 minutes.

I can reduce waiting time by running instance creation in parallel and waiting only when necessary – before the application is configured. I can do this by splitting instance preparation into two steps: booting and initialization. Booting happens in parallel for both instances, but initialization for Instance2 starts only when Instance1 is completely ready.

This is the new sequence:

Because I’m doing some tasks in parallel, it takes much less time for Instance2 to become available.

The only problem is that CloudFormation has no built-in construct to enter a dependency in the middle of the booting process of another resource. Let’s devise a solution for this.

Using Wait Conditions

Creation policies also provide a notification mechanism. I can decouple notification for the creation of an instance from the notification that the instance is fully ready by using a wait condition.

"Instance1WaitCondition" : {
"Type" : "AWS::CloudFormation::WaitCondition",
"DependsOn" : ["Instance1"],
"CreationPolicy" : {
"ResourceSignal" : {
"Timeout": "PT15M",
"Count" : "1"
}
}
}

Then I need to ask Instance1 to notify the wait condition after it’s done processing, instead of notifying itself. I’ll use the UserData section of the instance to do this.

    "Instance1": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "ImageId": { "Ref" : "ImageId" },
        "InstanceType": { "Ref": "InstanceType" },
        "UserData": {
          "Fn::Base64": {
            "Fn::Join": [ "n", [
                "<script>",

                "REM …Do any instance configuration steps deemed necessary…",

                { "Fn::Join": ["", [ "cfn-signal.exe -e 0 –stack "", { "Ref": "AWS::StackName" }, "" –resource "Instance1WaitCondition" –region "", { "Ref" : "AWS::Region" }, """ ] ] },
                "</script>"
            ] ]
          }
        }
      }
    }

Note that CreationPolicy is now defined inside Instance1WaitCondition, and the call to cfn-signal.exe notifies Instance1WaitCondition instead of Instance1.

We now have two resources that signal two different states of Instance1:

Instance1 is marked as created as soon as it is provisioned.

Instance1WaitCondition is marked as created only when Instance1 is fully initialized.

Let’s see how we can use this technique to optimize the booting process.

PowerShell to the Rescue

The DependsOn attribute is only available at the top level of resources, but I want to wait for Instance1 after the boot of Instance2. To allow that I need  a way to check the status of resources from within the instance’s initialization script so that I can see when resource creation for Instance1WaitCondition is complete. Let’s use Windows PowerShell to provide some automation.

To check resource status from within an instance’s initialization script, I’ll use AWS Tools for Windows PowerShell, a package that is installed by default on every Microsoft Windows Server image provided by Amazon Web Services. The package includes more than 1,100 cmdlets, giving us access to all of the APIs available on the AWS cloud.

The Get-CFNStackResources cmdlet allows me to see whether resource creation for Instance1WaitCondition is complete. This PowerShell script loops until a resource is created:

$region = ""
$stack = ""
$resource = "Instance1WaitCondition"
$output = (Get-CFNStackResources -StackName $stack -LogicalResourceId $resource -Region $region)
while (($output -eq $null) -or ($output.ResourceStatus -ne "CREATE_COMPLETE") -and ($output.ResourceStatus -ne "UPDATE_COMPLETE"))
{
Start-Sleep 10
$output = (Get-CFNStackResource -StackName $stack -LogicalResourceId $resource -Region $region)
}

Securing Access to the Resources

When calling an AWS API, I need to be authenticated and authorized. I can do this by providing an access key and a secret key to each API call, but there’s a much better way. I can simply create an AWS Identity and Access Management (IAM) role for the instance. When an instance has an IAM role, code that runs on the instance (including our PowerShell code in UserData) is authorized to make calls to the AWS APIs that are granted in the role.

When creating this role in IAM, I specify only the required actions, and limit these actions to only the current CloudFormation stack.

"DescribeRole": {
"Type" : "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version" : "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": [ "ec2.amazonaws.com" ] },
"Action": [ "sts:AssumeRole" ]
}
]
},
"Path": "/",
"Policies": [
{
"PolicyName" : "DescribeStack",
"PolicyDocument": {
"Version" : "2012-10-17",
"Statement": [
{
"Effect" : "Allow",
"Action" : ["cloudformation:DescribeStackResource", "cloudformation:DescribeStackResources"],
"Resource" : [ { "Ref" : "AWS::StackId" } ]
}
]
}
}
]
}
},
"DescribeInstanceProfile": {
"Type" : "AWS::IAM::InstanceProfile",
"Properties": {
"Path" : "/",
"Roles": [ { "Ref": "DescribeRole" } ]
}
}

Creating the Resources

The description for Instance1WaitCondition and Instance1 is fine, but I need to update Instance2 to add the IAM Role and include the PowerShell wait script. In the UserData section, I will add a scripted reference to Instance1WaitCondition. This "soft" reference doesn’t introduce any dependency in CloudFormation as this is just a simple string. In the UserData section, I will also add a GetAtt reference to Instance1 so that these instances will be provisioned quickly, one after another, without having to wait for the full instance to boot. I also need to secure my API calls by specifying the IAM role we have created as an IamInstanceProfile.

"Instance2": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": { "Ref" : "ImageId" },
"InstanceType": { "Ref": "InstanceType" },
"IamInstanceProfile": { "Ref": "DescribeInstanceProfile" },
"UserData": {
"Fn::Base64": {
"Fn::Join": [ "n", [
"",
"$resource = "Instance1WaitCondition"",
{ "Fn::Join": ["", [ "$region = ‘", { "Ref" : "AWS::Region" }, "’" ] ] },
{ "Fn::Join": ["", [ "$stack = ‘", { "Ref" : "AWS::StackId" }, "’" ] ] },

"#…Wait for instance 1 to be fully available…",

"$output = (Get-CFNStackResources -StackName $stack -LogicalResourceId $resource -Region $region)",
"while (($output -eq $null) -or ($output.ResourceStatus -ne "CREATE_COMPLETE") -and ($output.ResourceStatus -ne "UPDATE_COMPLETE")) {",
" Start-Sleep 10",
" $output = (Get-CFNStackResources -StackName $stack -LogicalResourceId $resource -Region $region)",
"}",

"#…Do any instance configuration steps you deem necessary…",

{ "Fn::Join": ["", [ "$instance1Ip = ‘", { "Fn::GetAtt" : [ "Instance1" , "PrivateIp" ] }, "’" ] ] },

"#…You can use the private IP address from Instance1 in your configuration scripts…",

""
] ]
}
}
}
}

Now, CloudFormation provisions Instance2 just after Instance1, saving a lot of time because Instance2 boots while Instance1 is booting, but Instance2 then waits for Instance1 to be fully operational before finishing its configuration.

During new environment creation, when a stack contains numerous resources, some with cascading dependencies, this technique can save a lot of time. And when you really need to get an environment up and running quickly, for example, when you’re performing disaster recovery, that’s important.

More Optimization Options

If you want a more reliable way to execute multiple scripts on an instance in CloudFormation, check out AWS::CloudFormation::cfn-init, which provides a flexible and powerful way to configure an instance when it’s started. To automate and simplify scripting your instances and reap the benefits of automatic domain joining for instances, see Amazon EC2 Simple Systems Manager (SSM). To operate your Windows instances in a full DevOps environment, consider using AWS OpsWorks.

Optimize AWS CloudFormation Templates

Post Syndicated from Elliot Yamaguchi original http://blogs.aws.amazon.com/application-management/post/Tx3NFBN5FT8NA1X/Optimize-AWS-CloudFormation-Templates

The following post is by guest blogger Julien Lépine, Solutions Architect at AWS. He explains how to optimize templates so that AWS CloudFormation quickly deploys your environments.

______________________________________________________________________________________

Customers sometimes ask me if there’s a way to optimize large AWS CloudFormation templates, which can take several minutes to deploy a stack. Often stack creation is slow because one resource depends on the availability of another resource before it can be provisioned. Examples include:

A front-end web server that has a dependency on an application server

A service that waits for another remote service to be available

In this post, I describe how to speed up stack creation when resources have dependencies on other resources.

Note: I show how to launch Windows instances with Windows PowerShell, but you can apply the same concepts to Linux instances launched with shell scripts.

How CloudFormation Creates Stacks

When CloudFormation provisions two instances, it provisions them randomly. Defining one resource before another in a template doesn’t guarantee that CloudFormation will provision that resource first. You need to explicitly tell CloudFormation the right order for instance provisioning.

To demonstrate how to do this, I’ll start with the following CloudFormation template:

{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description": "This is a demonstration AWS CloudFormation template containing two instances",
"Parameters": {
"ImageId" : {
"Description": "Identifier of the base Amazon Machine Image (AMI) for the instances in this sample (please use Microsoft Windows Server 2012 R2 Base)",
"Type" : "AWS::EC2::Image::Id"
},
"InstanceType" : {
"Description": "EC2 instance type to use for the instances in this sample",
"Type" : "String"
},
},
"Resources" : {
"Instance1": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": { "Ref" : "ImageId" },
"InstanceType": { "Ref": "InstanceType" },
}
},

"Instance2": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": { "Ref" : "ImageId" },
"InstanceType": { "Ref": "InstanceType" },
}
}
}
}

CloudFormation would likely create the stack in the following sequence:

This is fast, but if Instance2 is dependent on Instance1, you would ordinarily need to hard code or script the provisioning sequence to ensure that Instance1 is provisioned first.

Specifying Dependencies

When you need CloudFormation to wait to provision one resource until another one has been provisioned, you can use the DependsOn attribute.

"Instance2": {
"DependsOn": ["Instance1"]
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": { "Ref" : "ImageId" },
"InstanceType": { "Ref": "InstanceType" }
}
}

You can also introduce references between elements by using either the { "Ref": "MyResource" } or the { "Fn::GetAtt" : [ "MyResource" , "MyAttribute" ] } functions. When you use one of these functions, CloudFormation behaves as if you’ve added a DependsOn attribute to the resource. In the following example, the identifier of Instance1 is used in a tag for Instance2.

"Instance2": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": { "Ref" : "ImageId" },
"InstanceType": { "Ref": "InstanceType" },
"Tags": [ { "Key" : "Dependency", "Value" : { "Ref": "Instance1" } } ]
}
}

Both methods of specifying dependencies result in the same sequence:

Now, CloudFormation waits for Instance1 to be provisioned before provisioning Instance2. But I’m not guaranteed that services hosted on Instance1 will be available, so I will have to address that in the template.

Note that instances are provisioned quickly in CloudFormation. In fact, it happens in the time it takes to call the RunInstances Amazon Elastic Compute Cloud (EC2) API. But it takes longer for an instance to fully boot than it does to provision the instance.

Using Creation Policies to Wait for On-Instance Configurations

In addition to provisioning the instances in the right order, I want to ensure that a specific setup milestone has been achieved inside Instance1 before contacting it. To do this, I use a CreationPolicy attribute. A CreationPolicy is an attribute you can add to an instance to prevent it from being marked CREATE_COMPLETE until it has been fully initialized.

In addition to adding the CreationPolicy attribute, I want to ask Instance1 to notify CloudFormation after it’s done initializing. I can do this in the instance’s UserData section. On Windows instances, I can use this section to execute code in batch files or in Windows PowerShell in a process called bootstrapping.

I’ll execute a batch script, then tell CloudFormation that the creation process is done by sending a signal specifying that Instance1 is ready. Here’s the code with a CreationPolicy attribute and a UserData section that includes a script that invokes cfn-signal.exe:

    "Instance1": {
      "Type": "AWS::EC2::Instance",
      "CreationPolicy" : {
        "ResourceSignal" : {
          "Timeout": "PT15M",
          "Count"  : "1"
        }
      },
      "Properties": {
        "ImageId": { "Ref" : "ImageId" },
        "InstanceType": { "Ref": "InstanceType" },
        "UserData": {
          "Fn::Base64": {
            "Fn::Join": [ "n", [
                "<script>",

                "REM …Do any instance configuration steps deemed necessary…",

                { "Fn::Join": ["", [ "cfn-signal.exe -e 0 –stack "", { "Ref": "AWS::StackName" }, "" –resource "Instance1" –region "", { "Ref" : "AWS::Region" }, """ ] ] },
                "</script>"
            ] ]
          }
        }
      }
    }

I don’t need to change the definition of Instance2 because it’s already coded to wait for Instance1. I now know that Instance1 will be completely set up before Instance2 is provisioned. The sequence looks like this:

Optimizing the Process with Parallel Provisioning

It takes only a few seconds to provision an instance in CloudFormation, but it can take several minutes for an instance to boot and be ready because it must wait for the complete OS boot sequence, activation and the execution of the UserData scripts. As we saw in the figures, the time it takes to create the complete CloudFormation stack is about twice the boot and initialization time for a resource. Depending on the complexity of our processes, booting can take up to 10 minutes.

I can reduce waiting time by running instance creation in parallel and waiting only when necessary – before the application is configured. I can do this by splitting instance preparation into two steps: booting and initialization. Booting happens in parallel for both instances, but initialization for Instance2 starts only when Instance1 is completely ready.

This is the new sequence:

Because I’m doing some tasks in parallel, it takes much less time for Instance2 to become available.

The only problem is that CloudFormation has no built-in construct to enter a dependency in the middle of the booting process of another resource. Let’s devise a solution for this.

Using Wait Conditions

Creation policies also provide a notification mechanism. I can decouple notification for the creation of an instance from the notification that the instance is fully ready by using a wait condition.

"Instance1WaitCondition" : {
"Type" : "AWS::CloudFormation::WaitCondition",
"DependsOn" : ["Instance1"],
"CreationPolicy" : {
"ResourceSignal" : {
"Timeout": "PT15M",
"Count" : "1"
}
}
}

Then I need to ask Instance1 to notify the wait condition after it’s done processing, instead of notifying itself. I’ll use the UserData section of the instance to do this.

    "Instance1": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "ImageId": { "Ref" : "ImageId" },
        "InstanceType": { "Ref": "InstanceType" },
        "UserData": {
          "Fn::Base64": {
            "Fn::Join": [ "n", [
                "<script>",

                "REM …Do any instance configuration steps deemed necessary…",

                { "Fn::Join": ["", [ "cfn-signal.exe -e 0 –stack "", { "Ref": "AWS::StackName" }, "" –resource "Instance1WaitCondition" –region "", { "Ref" : "AWS::Region" }, """ ] ] },
                "</script>"
            ] ]
          }
        }
      }
    }

Note that CreationPolicy is now defined inside Instance1WaitCondition, and the call to cfn-signal.exe notifies Instance1WaitCondition instead of Instance1.

We now have two resources that signal two different states of Instance1:

Instance1 is marked as created as soon as it is provisioned.

Instance1WaitCondition is marked as created only when Instance1 is fully initialized.

Let’s see how we can use this technique to optimize the booting process.

PowerShell to the Rescue

The DependsOn attribute is only available at the top level of resources, but I want to wait for Instance1 after the boot of Instance2. To allow that I need  a way to check the status of resources from within the instance’s initialization script so that I can see when resource creation for Instance1WaitCondition is complete. Let’s use Windows PowerShell to provide some automation.

To check resource status from within an instance’s initialization script, I’ll use AWS Tools for Windows PowerShell, a package that is installed by default on every Microsoft Windows Server image provided by Amazon Web Services. The package includes more than 1,100 cmdlets, giving us access to all of the APIs available on the AWS cloud.

The Get-CFNStackResources cmdlet allows me to see whether resource creation for Instance1WaitCondition is complete. This PowerShell script loops until a resource is created:

$region = ""
$stack = ""
$resource = "Instance1WaitCondition"
$output = (Get-CFNStackResources -StackName $stack -LogicalResourceId $resource -Region $region)
while (($output -eq $null) -or ($output.ResourceStatus -ne "CREATE_COMPLETE") -and ($output.ResourceStatus -ne "UPDATE_COMPLETE"))
{
Start-Sleep 10
$output = (Get-CFNStackResource -StackName $stack -LogicalResourceId $resource -Region $region)
}

Securing Access to the Resources

When calling an AWS API, I need to be authenticated and authorized. I can do this by providing an access key and a secret key to each API call, but there’s a much better way. I can simply create an AWS Identity and Access Management (IAM) role for the instance. When an instance has an IAM role, code that runs on the instance (including our PowerShell code in UserData) is authorized to make calls to the AWS APIs that are granted in the role.

When creating this role in IAM, I specify only the required actions, and limit these actions to only the current CloudFormation stack.

"DescribeRole": {
"Type" : "AWS::IAM::Role",
"Properties": {
"AssumeRolePolicyDocument": {
"Version" : "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": { "Service": [ "ec2.amazonaws.com" ] },
"Action": [ "sts:AssumeRole" ]
}
]
},
"Path": "/",
"Policies": [
{
"PolicyName" : "DescribeStack",
"PolicyDocument": {
"Version" : "2012-10-17",
"Statement": [
{
"Effect" : "Allow",
"Action" : ["cloudformation:DescribeStackResource", "cloudformation:DescribeStackResources"],
"Resource" : [ { "Ref" : "AWS::StackId" } ]
}
]
}
}
]
}
},
"DescribeInstanceProfile": {
"Type" : "AWS::IAM::InstanceProfile",
"Properties": {
"Path" : "/",
"Roles": [ { "Ref": "DescribeRole" } ]
}
}

Creating the Resources

The description for Instance1WaitCondition and Instance1 is fine, but I need to update Instance2 to add the IAM Role and include the PowerShell wait script. In the UserData section, I will add a scripted reference to Instance1WaitCondition. This "soft" reference doesn’t introduce any dependency in CloudFormation as this is just a simple string. In the UserData section, I will also add a GetAtt reference to Instance1 so that these instances will be provisioned quickly, one after another, without having to wait for the full instance to boot. I also need to secure my API calls by specifying the IAM role we have created as an IamInstanceProfile.

"Instance2": {
"Type": "AWS::EC2::Instance",
"Properties": {
"ImageId": { "Ref" : "ImageId" },
"InstanceType": { "Ref": "InstanceType" },
"IamInstanceProfile": { "Ref": "DescribeInstanceProfile" },
"UserData": {
"Fn::Base64": {
"Fn::Join": [ "n", [
"",
"$resource = "Instance1WaitCondition"",
{ "Fn::Join": ["", [ "$region = ‘", { "Ref" : "AWS::Region" }, "’" ] ] },
{ "Fn::Join": ["", [ "$stack = ‘", { "Ref" : "AWS::StackId" }, "’" ] ] },

"#…Wait for instance 1 to be fully available…",

"$output = (Get-CFNStackResources -StackName $stack -LogicalResourceId $resource -Region $region)",
"while (($output -eq $null) -or ($output.ResourceStatus -ne "CREATE_COMPLETE") -and ($output.ResourceStatus -ne "UPDATE_COMPLETE")) {",
" Start-Sleep 10",
" $output = (Get-CFNStackResources -StackName $stack -LogicalResourceId $resource -Region $region)",
"}",

"#…Do any instance configuration steps you deem necessary…",

{ "Fn::Join": ["", [ "$instance1Ip = ‘", { "Fn::GetAtt" : [ "Instance1" , "PrivateIp" ] }, "’" ] ] },

"#…You can use the private IP address from Instance1 in your configuration scripts…",

""
] ]
}
}
}
}

Now, CloudFormation provisions Instance2 just after Instance1, saving a lot of time because Instance2 boots while Instance1 is booting, but Instance2 then waits for Instance1 to be fully operational before finishing its configuration.

During new environment creation, when a stack contains numerous resources, some with cascading dependencies, this technique can save a lot of time. And when you really need to get an environment up and running quickly, for example, when you’re performing disaster recovery, that’s important.

More Optimization Options

If you want a more reliable way to execute multiple scripts on an instance in CloudFormation, check out AWS::CloudFormation::cfn-init, which provides a flexible and powerful way to configure an instance when it’s started. To automate and simplify scripting your instances and reap the benefits of automatic domain joining for instances, see Amazon EC2 Simple Systems Manager (SSM). To operate your Windows instances in a full DevOps environment, consider using AWS OpsWorks.

How to Add DNS Filtering to Your NAT Instance with Squid

Post Syndicated from Nicolas Malaval original https://blogs.aws.amazon.com/security/post/TxFRX7UFUIT2GD/How-to-Add-DNS-Filtering-to-Your-NAT-Instance-with-Squid

Amazon Virtual Private Cloud (Amazon VPC) enables you to launch AWS resources on a virtual private network that you’ve defined. On an Amazon VPC, network address translation (NAT) instances, and more recently NAT gateways, are commonly used to enable instances in a private subnet to initiate outbound traffic to the Internet, but prevent the instances from receiving inbound traffic initiated by someone on the Internet.

For security and compliance purposes, you might have to filter the requests initiated by these instances. Using iptables rules, you could restrict outbound traffic with your NAT instance based on a predefined destination port or IP address. However, you may need to enforce more complex policies, such as allowing requests to AWS endpoints only, which cannot be achieved easily by using iptables rules.

In this post, I discuss and give an example of how Squid, a leading open-source proxy, can restrict both HTTP and HTTPS outbound traffic to a given set of Internet domains, while being fully transparent for instances in the private subnet. First, I explain briefly how to create the infrastructure resources required for this approach. Then, I provide step-by-step instructions to install, configure, and test Squid as a transparent proxy.

Keep in mind, that a possible alternative solution could be to deploy a proxy (also known as a forward proxy) in your Amazon VPC. However, a major drawback is that the proxy must be explicitly configured on every instance in the private subnet. This could cause connectivity issues that would be difficult to troubleshoot, if not impossible to remediate, when the application does not support proxy usage.

Deploying the example

The following steps are for manually creating and configuring the required resources. Alternatively, you could use AWS CloudFormation to automate this procedure. Click Create a Stack to open the CloudFormation console and create an AWS CloudFormation stack from the template I developed. Follow the on-screen instructions and go directly to the “Testing the deployment” section later in this blog post when the stack creation has completed (it can take up to 20 minutes).

Note: If you need to allow requests to S3 buckets in the same region as your VPC, you could use a VPC endpoint for Amazon S3 instead (see VPC Endpoints). It enables you to create a private connection between your VPC and another AWS service without requiring access over the Internet. It also has a policy that controls the use of the endpoint to access Amazon S3 resources.

Set up a VPC and create 2 Amazon EC2 instances

The following steps take you through the manual creation and configuration of a VPC and two EC2 instances: one in a public subnet for deploying Squid, and another in a private subnet for testing the configuration. See the NAT Instances documentation for further details, because the prerequisites are similar.

To create and configure a VPC and 2 EC2 instances:

Create a VPC (see Creating a VPC, if you need help with this step).

Create two subnets (see Creating a Subnet): one called “Public Subnet” and another called “Private Subnet.”

Create and attach an Internet Gateway to the VPC (see Attaching an Internet Gateway).

Add a rule to the default route table that sends traffic destined outside the VPC (0.0.0.0/0) to the Internet Gateway (see Adding and Removing Routes from a Route Table).

Add a rule to the VPC default security group that allows ingress SSH traffic (TCP 22) from your 0.0.0.0/0 (see Adding Rules to a Security Group).

Launch an Amazon Linux t2.micro instance called “Squid Instance” in the Public Subnet (make sure to use the Amazon Linux AMI, not the NAT instance AMI). Enable Auto-assign Public IP, choose the VPC default security group as the security group to attach, and select a valid key pair. Leave all other parameters as default (see Launching an Instance).

After the instance starts running, disable the source/destination check (see Disable Source/Destination Checks).

Create a new route table in the VPC. Add a rule that sends traffic destined outside the VPC (0.0.0.0/0) to the Squid instance and associate this new route table with the Private Subnet.

Create an instance role called “Testing Instance Role” and attach the managed policy AmazonEC2ReadOnlyAccess (see detailed instructions).

Finally, launch another Amazon Linux t2.micro instance called “Testing Instance” in the Private Subnet (make sure to use the Amazon Linux AMI, not the NAT instance AMI). Select Testing Instance Role as the instance role, attach the VPC default security group, and select the same key pair. Leave all other parameters as default.

The following diagram illustrates how the components in this process interact with each other. Squid Instance intercepts HTTP/S requests sent by Testing Instance. Squid Instance then initiates a connection with the destination host on behalf of Testing Instance, which goes through the Internet gateway.

Installing Squid

Squid intercepts the requested domain before applying the filtering policy:

For HTTP, Squid retrieves the Host header field included in all HTTP/1.1 request messages, which specifies the Internet host being requested.

For HTTPS, the HTTP traffic is encapsulated in a TLS connection between the instance in the private subnet and the remote host. Squid cannot retrieve the Host header field because the header is encrypted. A feature called SslBump would allow Squid to decrypt the traffic, but this would not be transparent for the client because the certificate would be considered invalid in most cases. The feature we use instead, called SslPeekAndSplice, retrieves the Server Name Indication (SNI) from the TLS initiation, which contains the requested Internet host. As a result, Squid can make filtering decisions without unencrypting the HTTPS traffic.

Note: Some older client-side software, stacks do not support SNI. These are the minimum versions of some important stacks and programming languages that support SNI: Python 2.7.9 and 3.2, Java 7 JSSE, wget 1.14, OpenSSL 0.9.8j, cURL 7.18.1

The feature SslPeekAndSplice was introduced in Squid 3.5. However, when this post was written, the Amazon Linux repository included Squid 3.1.10 (you can check whether Squid 3.5 is available using the command, yum info squid). For the purpose of this post, I will compile and install Squid 3.5 from the official source code.

Note: This Squid installation includes the minimum features required for this example and is not intended for production purposes. You may prefer to adapt it to your own needs and install Squid from your own Red Hat Package Manager (RPM) package or from unofficial RPM packages for CentOS 6.

Squid installation instructions

To manually compile, install, and configure Squid on the Squid instance:

Connect to your Squid instance using SSH with the user ec2-user.

Install the prerequisite packages.

sudo yum update -y
sudo yum install -y perl gcc autoconf automake make sudo wget gcc-c++ libxml2-devel libcap-devel libtool libtool-ltdl-devel openssl openssl-devel

Go to this Squid page and retrieve the link to the tar.gz source code archive for the latest release (3.5.13 was the last release when this post was written). Use this link to download and extract the archive on the Squid instance.

SQUID_ARCHIVE=http://www.squid-cache.org/Versions/v3/3.5/squid-3.5.13.tar.gz
cd /tmp
wget $SQUID_ARCHIVE
tar xvf squid*.tar.gz
cd $(basename squid*.tar.gz .tar.gz)

Compile and install Squid with the minimum required options. This may take up to 15 minutes.

sudo ./configure –prefix=/usr –exec-prefix=/usr –libexecdir=/usr/lib64/squid –sysconfdir=/etc/squid –sharedstatedir=/var/lib –localstatedir=/var –libdir=/usr/lib64 –datadir=/usr/share/squid –with-logdir=/var/log/squid –with-pidfile=/var/run/squid.pid –with-default-user=squid –disable-dependency-tracking –enable-linux-netfilter –with-openssl –without-nettle

sudo make
sudo make install

Complete the Squid installation.

sudo adduser -M squid
sudo chown -R squid:squid /var/log/squid /var/cache/squid
sudo chmod 750 /var/log/squid /var/cache/squid
sudo touch /etc/squid/squid.conf
sudo chown -R root:squid /etc/squid/squid.conf
sudo chmod 640 /etc/squid/squid.conf
cat | sudo tee /etc/init.d/squid <<‘EOF’
#!/bin/sh
# chkconfig: – 90 25
echo -n ‘Squid service’
case "$1" in
start)
/usr/sbin/squid
;;
stop)
/usr/sbin/squid -k shutdown
;;
reload)
/usr/sbin/squid -k reconfigure
;;
*)
echo "Usage: `basename $0` {start|stop|reload}"
;;
esac
EOF
sudo chmod +x /etc/init.d/squid
sudo chkconfig squid on

Note: If you have installed Squid from a RPM package, you are not required to follow the previous instructions for installing Squid before proceeding to the next steps, because your Squid instance already has the required configuration.

Configuring and starting Squid

The SslPeekAndSplice feature is implemented in the same Squid module as SslBump. To enable this module, Squid requires that we provide a certificate, though it will not be used to decode HTTPS traffic. I create a certificate using OpenSSL.

sudo mkdir /etc/squid/ssl
cd /etc/squid/ssl
sudo openssl genrsa -out squid.key 2048
sudo openssl req -new -key squid.key -out squid.csr -subj "/C=XX/ST=XX/L=squid/O=squid/CN=squid"
sudo openssl x509 -req -days 3650 -in squid.csr -signkey squid.key -out squid.crt
sudo cat squid.key squid.crt | sudo tee squid.pem

Next, configure Squid to allow requests to *.amazonaws.com, which corresponds to AWS endpoints. Note that you can restrict access to a defined set of AWS services only. See AWS Regions and Endpoints for a detailed list of endpoints.

For HTTPS traffic, note the ssl_bump directives instructing Squid to “peek” (retrieve the SNI) and then “splice” (become a TCP tunnel without decoding) or “terminate” the connection depending on the requested host.

cat | sudo tee /etc/squid/squid.conf <<EOF
visible_hostname squid

#Handling HTTP requests
http_port 3129 intercept
acl allowed_http_sites dstdomain .amazonaws.com
#acl allowed_http_sites dstdomain [you can add other domains to permit]
http_access allow allowed_http_sites

#Handling HTTPS requests
https_port 3130 cert=/etc/squid/ssl/squid.pem ssl-bump intercept
acl SSL_port port 443
http_access allow SSL_port
acl allowed_https_sites ssl::server_name .amazonaws.com
#acl allowed_https_sites ssl::server_name [you can add other domains to permit]
acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
ssl_bump peek step1 all
ssl_bump peek step2 allowed_https_sites
ssl_bump splice step3 allowed_https_sites
ssl_bump terminate step2 all

http_access deny all
EOF

You may have noticed that Squid listens on port 3129 for HTTP traffic and 3130 for HTTPS. Because Squid cannot directly listen to 80 and 443, we have to redirect the incoming requests from instances in the private subnet to the Squid ports using iptables. You do not have to enable IP Forwarding or add any FORWARD rule, as you would do with a standard NAT instance.

sudo iptables -t nat -A PREROUTING -p tcp –dport 80 -j REDIRECT –to-port 3129
sudo iptables -t nat -A PREROUTING -p tcp –dport 443 -j REDIRECT –to-port 3130

You can now start Squid.

sudo service squid start

Testing the deployment

The Testing Instance that was launched earlier can be used to test the configuration. Because this instance is not accessible from the Internet, you have to jump onto the Squid instance to log on to Testing Instance.

Because both instances were launched with the same key pair, you can connect to Squid Instance using the -A argument in order to forward the SSH key to Testing Instance.

ssh-add [key]
ssh -A [email protected][public IP of the Squid instance] –i [key]
ssh [email protected][private IP of the client instance] –i [key]

You can test the transparent proxy instance with the following commands. Only the last three requests should return a valid response, because Squid allows traffic to *.amazonaws.com only.

curl http://www.amazon.com
curl https://www.amazon.com
curl http://calculator.s3.amazonaws.com/index.html
curl https://calculator.s3.amazonaws.com/index.html
aws ec2 describe-regions –region us-east-1

You can now clean up the resources you just created.

Summary

In this blog post, I have shown how you can use Squid to filter outgoing traffic to the Internet and help meet your security and compliance needs, while being fully transparent for the back-end instances in your VPC.

I invite you to adapt this example to your own requirements. For example, you may implement a high-availability solution, similar to the solution described in High Availability for Amazon VPC NAT Instances: An Example; centralize Squid metrics and access logs, similar to the solution described in Using Squid Proxy Instances for Web Service Access in Amazon VPC: Another Example with AWS CodeDeploy and Amazon CloudWatch; or leverage other Squid features (logging, caching, etc.) for further visibility and control over outbound traffic.

If you have any questions or suggestions, please leave a comment below or on the IAM forum.

– Nicolas

Agile Analytics with Amazon Redshift

Post Syndicated from Nick Corbett original https://blogs.aws.amazon.com/bigdata/post/Tx3HHDIXCZFDGTN/Agile-Analytics-with-Amazon-Redshift

Nick Corbett is a Big Data Consultant for AWS Professional Services

What makes outstanding business intelligence (BI)? It needs to be accurate and up-to-date, but this alone won’t differentiate a solution. Perhaps a better measure is to consider the reaction you get when your latest report or metric is released to the business. Good BI excites:  it prompts new ways of thinking and new ideas that, more often than not, require change to support. Businesses are constantly looking to evolve, to use their BI to gain insight and competitive advantage. Truly outstanding BI is agile enough to keep pace with this demand.

In this post, I show you how your Amazon Redshift data warehouse can be agile. To do this, you need to adopt a continuous delivery (CD) approach that draws on many of the tools and techniques already successfully used by software engineers. CD focuses on automating the release process, allowing quick and frequent deployments to production whilst ensuring quality is maintained. In return, you will enjoy many benefits; for example, you can identify issues quickly, and those that you do find are likely to be less complex.  By shortening your release cycle, you can respond to requests from your business more quickly.

A simple CD process, or pipeline, is shown below.  This uses a combination of AWS fully-managed services and the extensible, open-source, continuous integration server Jenkins. You can follow the instructions at the AWS Big Data Blog repository on GitHub to build your own sample environment and learn how to configure these components. The repository includes a CloudFormation script to set up and configure a Jenkins server in a virtual private cloud (VPC) and instructions for setting up AWS CodeCommit and AWS CodePipeline.  Note that starting this environment incurs a charge in your account, although all the AWS services used are eligible for the AWS Free Tier.

Build a Deployment Package

One of the key assumptions of CD is that no one directly interacts with the production system; all deployments and updates are fully automated. The logical conclusion of this assumption is that everything required to build, maintain, update and test your data warehouse should be scripted and under source control. The sample environment uses AWS CodeCommit as a source control repository. AWS CodeCommit is a fully-managed source control service that makes it easy for companies to host secure and highly scalable private Git repositories.

The first stage of your CD pipeline is to build a deployment package. This is a snapshot of a branch of your source code repository and is immutable. After it’s built, the same package is used for all your testing and finally deployed to your production system.

Each time you build a new package you should aim to answer one question: can I deploy this to my production system?  You can answer ‘yes’ when you are satisfied that the new features in the deployment work as expected and that don’t break anything that’s already part of your production system. The faster you can answer this question, the better. Testing each package costs you, both in infrastructure costs and people resources if manual testing is needed. The more testing you do on a package, the more expensive it becomes. You should look to build a process that fails fast by performing simple, automated tests first and only spend time manually testing builds that you know are of good quality.

The environment uses a Jenkins job to build the deployment package. Jenkins is configured to scan the repository in AWS CodeCommit periodically and trigger a job when changes are detected. The sample pipeline scans the master branch, but you can configure Jenkins to scan any branch, perhaps to test new features as they are being developed. The sample job simply zips all the files in the repository before uploading to an Amazon S3 bucket that has versioning enabled. With fail-fast methodology, you could use this opportunity to verify that all files in the package conform to your coding standards; for example, do they have the correct header and have naming standards been followed?  At this point, you don’t have a Amazon Redshift database to test things in, but you can do quick and easy tests on the files, such as using grep commands to check the contents.

Test the deployment package

The remaining stages in the pipeline are managed by AWS CodePipeline, a continuous delivery service for fast and reliable updates. AWS CodePipeline builds, tests, and deploys your code every time there is a change, based on the release process models that you define. This enables you to deliver features and updates rapidly and reliably.

The CD pipeline shown above uses AWS CodePipeline to run tests in two environments before deploying the package to the production system. The tests have been split into a set of functional tests, run against a small set of data, and non-functional tests run against a production-sized Amazon Redshift cluster. You may choose to organize your CD pipeline differently, but your process will probably involve running one or more test stages followed by a deployment to production. Each stage of the pipeline is implemented by a Jenkins job that carries out four main tasks.

1. Build a test environment

To run tests, you need an Amazon Redshift database. The sample job uses the AWS CloudFormation Plugin to create a VPC with a public facing, single-node Amazon Redshift database. The configuration of the plugin is shown below:

As you can see from the screenshot, the job is configured to build the CloudFormation template found in ./cloudformation/redshiftvpc.json within your deployment package. The definition of your test environment is held in the same source control as the other project assets.

The sample job creates an empty Amazon Redshift data warehouse and creates tables using SQL scripts. However, you can use the snapshotidentifer property of the Amazon Redshift object in the CloudFormation template to create a data warehouse from a point-in-time backup of a cluster. You may have a pre-prepared snapshot that has the same structure as your production system but contains test data. Alternatively, you might first use the AWS Command Line Interface (CLI) to call describe-cluster-snapshots and find the ID of the latest production snapshot.  Creating a test environment that mirrors production is a good choice for non-functional testing, especially performance tests.

Any outputs that are returned by your CloudFormation stack are automatically added as variables in your build environment and can be used in subsequent steps.  The variables are named <stack name>_<output name>.  In the sample job, you can see that the endpoint of the new Amazon Redshift database is used by subsequent steps.

2. Apply the update in the deployment package

The sample job takes the latest folder in the ./updates folder of the deployment package and runs all the .sql files against the data warehouse.  This assumes that you’ve asked your developers to provide scripts for all the changes they want to make to the system. If any of the .sql files fails to run, then the overall Jenkins job fails. If this occurs, you should ask your development team to fix the issue and restart the pipeline with a new deployment package.

3. Run the tests

You now have a populated Amazon Redshift data warehouse that you’ve successfully updated with the deployment package. You should run a series of tests to ensure that the new features work as expected and that the deployment hasn’t broken any existing functionality.

For example, imagine that your data warehouse has a customer table with a name column and you’ve asked your developers to refactor this to first_name and last_name. You would expect your developers to provide an ALTER TABLE  statement to change the definition of your customer table and an UPDATE statement to populate the new columns from the existing name column.

After the update scripts have run, the most basic check would be to confirm that the customer table contains first_name and last_name columns. However, you should also test the script that populates these new columns.  What happened if the name column contained three words, such as John Robert Smith? What if it contained initials or suffixes, such as JJ Smith Esq? What if it were null or contained a single very long name that might be truncated due to its length? You should also look at how your customer names are used. Perhaps you have a view used by a report that shows your best customers – does the name of each customer still appear correctly now that it is a concatenation of two fields?

Over time, you can build a library of tests that are run whenever you have a new deployment package. This ensures that you not only test new features but don’t regress in any other areas.

You may also choose to run non-functional tests. For example, is the query plan for the view that drives your best customer report still the same? Does the query still execute in the same amount of time? Is access to use the view restricted to the correct groups?

The sample job contains the code to run a set of SQL statements and check the result against what is expected. The pseudo-code is shown below:

for each .sql file in ./tests
result = run contents of .sql file against Amazon Redshift
expected = contents of .result file
if [ Ignore Case and White Space ] Result != Expected
// test fail
exit 1
end for
exit 0

The test loops over all .sql files in the ./tests folder and executes each one. It then compares the result of the query with the contents of the .result file with the same name. For example, if message_test1.sql is executed, the value returned is compared with the contents of message_test1.sql.result. This comparison is made after removing all whitespace. If the result is not what was expected, the Jenkins job ends in failure. You can adapt this bash script to include performance statistics by monitoring the execution duration.

At the end of your testing, you can decide whether the deployment meets your quality bar. It may be that you still need to do manual testing before promoting to production, but this should only be performed after all automated tests have been passed. You may want to avoid involving your test team and the extra expense unless there’s a high chance that you will take this deployment forward to production.

4. Delete your test environment

After you have finished your testing, you can delete your test environment.  This can be done by your Jenkins server, meaning that the entire process of creating an environment, running the tests, and deleting the environment can be automated. You can save money with a transient environment that’s created only when you need to run tests and then immediately deleted. If you do need to perform manual testing, you can configure your Jenkins job to not delete the cluster.

Deploy to production

You are now ready to deploy the changes to your production environment. Because the same set of update scripts in the same deployment package have already been successfully run and tested in pre-production, you can be confident that this change will be easy to roll out. In AWS CodePipeline, the transition between the PreProduction and Production stages has been disabled:

When you re-enable a disabled transition, the latest revision runs through the remaining stages of the pipeline. At the end of the automated process, the decision to release to production is manual: it can be initiated by your release manager either using either the AWS Management Console or the API.

Post-deployment tasks

After you’ve deployed to production, you may want to have a final stage in your pipeline that sets up for the next round of testing. If you are using an environment containing test data for your functional testing, then a new snapshot is required for your next release. You could also implement some of the Top 10 Performance Tuning Tasks to make sure that your Amazon Redshift data warehouse is in good health following the latest update.

Summary

In this post, I have shown you how to build a CD pipeline for your Amazon Redshift data warehouse. The pipeline presented here is fairly simple but hopefully it’s easy to see how the concept can be expanded with more stages or more sophisticated testing.

However, remember that a simple pipeline is better than no pipeline at all. You should take a holistic view:  your ‘system’ is the Amazon Redshift data warehouse and the CD pipeline needed to maintain it. Start simple, iterate with each development sprint, and build complexity as you go. Work towards an agile data warehouse that can keep pace with your business, leading change rather than reacting to it. 

If you have any questions or suggestions, please leave a comment below.

————————–

Related:

Top 10 Performance Tuning Techniques for Amazon Redshift

 

 

Agile Analytics with Amazon Redshift

Post Syndicated from Nick Corbett original https://blogs.aws.amazon.com/bigdata/post/Tx3HHDIXCZFDGTN/Agile-Analytics-with-Amazon-Redshift

Nick Corbett is a Big Data Consultant for AWS Professional Services

What makes outstanding business intelligence (BI)? It needs to be accurate and up-to-date, but this alone won’t differentiate a solution. Perhaps a better measure is to consider the reaction you get when your latest report or metric is released to the business. Good BI excites:  it prompts new ways of thinking and new ideas that, more often than not, require change to support. Businesses are constantly looking to evolve, to use their BI to gain insight and competitive advantage. Truly outstanding BI is agile enough to keep pace with this demand.

In this post, I show you how your Amazon Redshift data warehouse can be agile. To do this, you need to adopt a continuous delivery (CD) approach that draws on many of the tools and techniques already successfully used by software engineers. CD focuses on automating the release process, allowing quick and frequent deployments to production whilst ensuring quality is maintained. In return, you will enjoy many benefits; for example, you can identify issues quickly, and those that you do find are likely to be less complex.  By shortening your release cycle, you can respond to requests from your business more quickly.

A simple CD process, or pipeline, is shown below.  This uses a combination of AWS fully-managed services and the extensible, open-source, continuous integration server Jenkins. You can follow the instructions at the AWS Big Data Blog repository on GitHub to build your own sample environment and learn how to configure these components. The repository includes a CloudFormation script to set up and configure a Jenkins server in a virtual private cloud (VPC) and instructions for setting up AWS CodeCommit and AWS CodePipeline.  Note that starting this environment incurs a charge in your account, although all the AWS services used are eligible for the AWS Free Tier.

Build a Deployment Package

One of the key assumptions of CD is that no one directly interacts with the production system; all deployments and updates are fully automated. The logical conclusion of this assumption is that everything required to build, maintain, update and test your data warehouse should be scripted and under source control. The sample environment uses AWS CodeCommit as a source control repository. AWS CodeCommit is a fully-managed source control service that makes it easy for companies to host secure and highly scalable private Git repositories.

The first stage of your CD pipeline is to build a deployment package. This is a snapshot of a branch of your source code repository and is immutable. After it’s built, the same package is used for all your testing and finally deployed to your production system.

Each time you build a new package you should aim to answer one question: can I deploy this to my production system?  You can answer ‘yes’ when you are satisfied that the new features in the deployment work as expected and that don’t break anything that’s already part of your production system. The faster you can answer this question, the better. Testing each package costs you, both in infrastructure costs and people resources if manual testing is needed. The more testing you do on a package, the more expensive it becomes. You should look to build a process that fails fast by performing simple, automated tests first and only spend time manually testing builds that you know are of good quality.

The environment uses a Jenkins job to build the deployment package. Jenkins is configured to scan the repository in AWS CodeCommit periodically and trigger a job when changes are detected. The sample pipeline scans the master branch, but you can configure Jenkins to scan any branch, perhaps to test new features as they are being developed. The sample job simply zips all the files in the repository before uploading to an Amazon S3 bucket that has versioning enabled. With fail-fast methodology, you could use this opportunity to verify that all files in the package conform to your coding standards; for example, do they have the correct header and have naming standards been followed?  At this point, you don’t have a Amazon Redshift database to test things in, but you can do quick and easy tests on the files, such as using grep commands to check the contents.

Test the deployment package

The remaining stages in the pipeline are managed by AWS CodePipeline, a continuous delivery service for fast and reliable updates. AWS CodePipeline builds, tests, and deploys your code every time there is a change, based on the release process models that you define. This enables you to deliver features and updates rapidly and reliably.

The CD pipeline shown above uses AWS CodePipeline to run tests in two environments before deploying the package to the production system. The tests have been split into a set of functional tests, run against a small set of data, and non-functional tests run against a production-sized Amazon Redshift cluster. You may choose to organize your CD pipeline differently, but your process will probably involve running one or more test stages followed by a deployment to production. Each stage of the pipeline is implemented by a Jenkins job that carries out four main tasks.

1. Build a test environment

To run tests, you need an Amazon Redshift database. The sample job uses the AWS CloudFormation Plugin to create a VPC with a public facing, single-node Amazon Redshift database. The configuration of the plugin is shown below:

As you can see from the screenshot, the job is configured to build the CloudFormation template found in ./cloudformation/redshiftvpc.json within your deployment package. The definition of your test environment is held in the same source control as the other project assets.

The sample job creates an empty Amazon Redshift data warehouse and creates tables using SQL scripts. However, you can use the snapshotidentifer property of the Amazon Redshift object in the CloudFormation template to create a data warehouse from a point-in-time backup of a cluster. You may have a pre-prepared snapshot that has the same structure as your production system but contains test data. Alternatively, you might first use the AWS Command Line Interface (CLI) to call describe-cluster-snapshots and find the ID of the latest production snapshot.  Creating a test environment that mirrors production is a good choice for non-functional testing, especially performance tests.

Any outputs that are returned by your CloudFormation stack are automatically added as variables in your build environment and can be used in subsequent steps.  The variables are named <stack name>_<output name>.  In the sample job, you can see that the endpoint of the new Amazon Redshift database is used by subsequent steps.

2. Apply the update in the deployment package

The sample job takes the latest folder in the ./updates folder of the deployment package and runs all the .sql files against the data warehouse.  This assumes that you’ve asked your developers to provide scripts for all the changes they want to make to the system. If any of the .sql files fails to run, then the overall Jenkins job fails. If this occurs, you should ask your development team to fix the issue and restart the pipeline with a new deployment package.

3. Run the tests

You now have a populated Amazon Redshift data warehouse that you’ve successfully updated with the deployment package. You should run a series of tests to ensure that the new features work as expected and that the deployment hasn’t broken any existing functionality.

For example, imagine that your data warehouse has a customer table with a name column and you’ve asked your developers to refactor this to first_name and last_name. You would expect your developers to provide an ALTER TABLE  statement to change the definition of your customer table and an UPDATE statement to populate the new columns from the existing name column.

After the update scripts have run, the most basic check would be to confirm that the customer table contains first_name and last_name columns. However, you should also test the script that populates these new columns.  What happened if the name column contained three words, such as John Robert Smith? What if it contained initials or suffixes, such as JJ Smith Esq? What if it were null or contained a single very long name that might be truncated due to its length? You should also look at how your customer names are used. Perhaps you have a view used by a report that shows your best customers – does the name of each customer still appear correctly now that it is a concatenation of two fields?

Over time, you can build a library of tests that are run whenever you have a new deployment package. This ensures that you not only test new features but don’t regress in any other areas.

You may also choose to run non-functional tests. For example, is the query plan for the view that drives your best customer report still the same? Does the query still execute in the same amount of time? Is access to use the view restricted to the correct groups?

The sample job contains the code to run a set of SQL statements and check the result against what is expected. The pseudo-code is shown below:

for each .sql file in ./tests
result = run contents of .sql file against Amazon Redshift
expected = contents of .result file
if [ Ignore Case and White Space ] Result != Expected
// test fail
exit 1
end for
exit 0

The test loops over all .sql files in the ./tests folder and executes each one. It then compares the result of the query with the contents of the .result file with the same name. For example, if message_test1.sql is executed, the value returned is compared with the contents of message_test1.sql.result. This comparison is made after removing all whitespace. If the result is not what was expected, the Jenkins job ends in failure. You can adapt this bash script to include performance statistics by monitoring the execution duration.

At the end of your testing, you can decide whether the deployment meets your quality bar. It may be that you still need to do manual testing before promoting to production, but this should only be performed after all automated tests have been passed. You may want to avoid involving your test team and the extra expense unless there’s a high chance that you will take this deployment forward to production.

4. Delete your test environment

After you have finished your testing, you can delete your test environment.  This can be done by your Jenkins server, meaning that the entire process of creating an environment, running the tests, and deleting the environment can be automated. You can save money with a transient environment that’s created only when you need to run tests and then immediately deleted. If you do need to perform manual testing, you can configure your Jenkins job to not delete the cluster.

Deploy to production

You are now ready to deploy the changes to your production environment. Because the same set of update scripts in the same deployment package have already been successfully run and tested in pre-production, you can be confident that this change will be easy to roll out. In AWS CodePipeline, the transition between the PreProduction and Production stages has been disabled:

When you re-enable a disabled transition, the latest revision runs through the remaining stages of the pipeline. At the end of the automated process, the decision to release to production is manual: it can be initiated by your release manager either using either the AWS Management Console or the API.

Post-deployment tasks

After you’ve deployed to production, you may want to have a final stage in your pipeline that sets up for the next round of testing. If you are using an environment containing test data for your functional testing, then a new snapshot is required for your next release. You could also implement some of the Top 10 Performance Tuning Tasks to make sure that your Amazon Redshift data warehouse is in good health following the latest update.

Summary

In this post, I have shown you how to build a CD pipeline for your Amazon Redshift data warehouse. The pipeline presented here is fairly simple but hopefully it’s easy to see how the concept can be expanded with more stages or more sophisticated testing.

However, remember that a simple pipeline is better than no pipeline at all. You should take a holistic view:  your ‘system’ is the Amazon Redshift data warehouse and the CD pipeline needed to maintain it. Start simple, iterate with each development sprint, and build complexity as you go. Work towards an agile data warehouse that can keep pace with your business, leading change rather than reacting to it. 

If you have any questions or suggestions, please leave a comment below.

————————–

Related:

Top 10 Performance Tuning Techniques for Amazon Redshift

 

 

Speed Template Sharing with an AWS CloudFormation Designer URL

Post Syndicated from Elliot Yamaguchi original http://blogs.aws.amazon.com/application-management/post/Tx3GOECCYX4F7QT/Speed-Template-Sharing-with-an-AWS-CloudFormation-Designer-URL

AWS CloudFormation Designer (Designer) is a graphic tool for creating, viewing, and modifying AWS CloudFormation templates. With Designer, you can construct templates by adding resources using a drag-and-drop interface, and then edit resource details by using the integrated JSON editor.

A Designer URL is a web link that you can share with anyone that has access to your template so that they can directly view and edit a copy of the template in Designer. This helps template users save time because they don’t need to open Designer, and then open the template. At AWS, we use Designer URLs with our sample templates so that our customers can quickly view and edit templates. In this post, I show how to construct a Designer URL.

I construct a Designer URL similar to launch a stack URLs. First, I save the template in an Amazon S3 bucket and grant AWS users who should have access to it Open/Download permissions. After I save the template, I retrieve its URL by using the Amazon Simple Storage Service (Amazon S3) console.

In my example, the template URL is https://s3.amazonaws.com/samplecfntemplates/mytemplate.template. Although I’ve granted Everyone permissions to open and download the template, I haven’t granted them permissions to edit it. Users can still edit and save a copy of the template, but they cannot modify the original.

To construct the Designer URL, I use the following URL syntax:

https://console.aws.amazon.com/cloudformation/designer/home?region=region&templateURL=template_location

I use the region parameter to specify which region Designer will be used in. If I don’t specify a region, users are directed to the region they last used. With the template_location parameter, I specify the URL of the template file.

For example, the following Designer URL opens Designer with the mytemplate.template in the us-west-2 region:

https://console.aws.amazon.com/cloudformation/designer/home?region=us-west-2&templateURL=https://s3-us-west-2.amazonaws.com/sample/mytemplate.template

Anyone who uses that URL is directed to Designer with the mytemplate.template loaded. Users who are not already authenticated are directed to the AWS Management Console sign-in page, and then are redirected to Designer. They can then make changes or create a stack using the template. If they want to save their changes, they have to save the template to a new file.

For more information about creating templates, see Working with Templates in the AWS CloudFormation User Guide.

Speed Template Sharing with an AWS CloudFormation Designer URL

Post Syndicated from Elliot Yamaguchi original http://blogs.aws.amazon.com/application-management/post/Tx3GOECCYX4F7QT/Speed-Template-Sharing-with-an-AWS-CloudFormation-Designer-URL

AWS CloudFormation Designer (Designer) is a graphic tool for creating, viewing, and modifying AWS CloudFormation templates. With Designer, you can construct templates by adding resources using a drag-and-drop interface, and then edit resource details by using the integrated JSON editor.

A Designer URL is a web link that you can share with anyone that has access to your template so that they can directly view and edit a copy of the template in Designer. This helps template users save time because they don’t need to open Designer, and then open the template. At AWS, we use Designer URLs with our sample templates so that our customers can quickly view and edit templates. In this post, I show how to construct a Designer URL.

I construct a Designer URL similar to launch a stack URLs. First, I save the template in an Amazon S3 bucket and grant AWS users who should have access to it Open/Download permissions. After I save the template, I retrieve its URL by using the Amazon Simple Storage Service (Amazon S3) console.

In my example, the template URL is https://s3.amazonaws.com/samplecfntemplates/mytemplate.template. Although I’ve granted Everyone permissions to open and download the template, I haven’t granted them permissions to edit it. Users can still edit and save a copy of the template, but they cannot modify the original.

To construct the Designer URL, I use the following URL syntax:

https://console.aws.amazon.com/cloudformation/designer/home?region=region&templateURL=template_location

I use the region parameter to specify which region Designer will be used in. If I don’t specify a region, users are directed to the region they last used. With the template_location parameter, I specify the URL of the template file.

For example, the following Designer URL opens Designer with the mytemplate.template in the us-west-2 region:

https://console.aws.amazon.com/cloudformation/designer/home?region=us-west-2&templateURL=https://s3-us-west-2.amazonaws.com/sample/mytemplate.template

Anyone who uses that URL is directed to Designer with the mytemplate.template loaded. Users who are not already authenticated are directed to the AWS Management Console sign-in page, and then are redirected to Designer. They can then make changes or create a stack using the template. If they want to save their changes, they have to save the template to a new file.

For more information about creating templates, see Working with Templates in the AWS CloudFormation User Guide.

Setting Up the Jenkins Plugin for AWS CodeDeploy

Post Syndicated from Shankar Sivadasan original http://blogs.aws.amazon.com/application-management/post/TxMJROUIFQZ4HS/Setting-Up-the-Jenkins-Plugin-for-AWS-CodeDeploy

The following is a guest post by Maitreya Ranganath, Solutions Architect.

In this post, we’ll show you how to use the Jenkins plugin to automatically deploy your builds with AWS CodeDeploy. We’ll walk through the steps for creating an AWS CodeCommit repository, installing Jenkins and the Jenkins plugin, adding files to the CodeCommit repository, and configuring the plugin to create a deployment when changes are committed to an AWS CodeCommit repository.

Create an AWS CodeCommit Repository

First, we will create an AWS CodeCommit repository to store our sample code files.

1. Sign in to the AWS Management Console and open the AWS CodeCommit console in the us-east-1 (N. Virginia) Region.  Choose Get Started or Create Repository.

2. For Repository Name, type a name for your repository (for example, DemoRepository). For Description, type Repository for Jenkins Code Deploy.

3. Choose the Create repository button.

4. Choose the repository you just created to view its details.

5. Choose the Clone URL button, and then choose HTTPS. Copy the URL displayed into a clipboard. You’ll need it later to configure Jenkins.

 

Now that you have created an AWS CodeCommit repository, we’ll create a Jenkins server and AWS CodeDeploy environment.

Create a Jenkins Server and AWS CodeDeploy Environment

In this step, we’ll launch a CloudFormation template that will create the following resources:

An Amazon S3 bucket that will be used to store deployment files.

JenkinsRole, an IAM role and instance profile for the Amazon EC2 instance that will run Jenkins. This role allows Jenkins on the EC2 instance to assume the CodeDeployRole and access repositories in CodeCommit.

CodeDeployRole, an IAM role assumed by the CodeDeploy Jenkins plugin. This role has permissions to write files to the S3 bucket created by this template and to create deployments in CodeDeploy.

Jenkins server, an EC2 instance running Jenkins.

An Auto Scaling group of EC2 instances running Apache and the CodeDeploy agent fronted by an Elastic Load Balancing load balancer.

To create the CloudFormation stack, choose the link that corresponds to the AWS region where you want to work:

For the us-east-1 region:

or use the link below:

https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=JenkinsCodeDeploy&templateURL=https://s3.amazonaws.com/aws-codedeploy-us-east-1/templates/latest/CodeDeploy_SampleCF_Jenkins_Integration.json

For the us-west-2 region:

or use the link below:

https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=JenkinsCodeDeploy&templateURL=https://s3.amazonaws.com/aws-codedeploy-us-east-1/templates/latest/CodeDeploy_SampleCF_Jenkins_Integration.json

6. Choose Next and specify the following values:

For InstanceCount, accept the default of 3. (Three EC2 instances will be launched for CodeDeploy.)

For InstanceType, accept the default of t2.medium.

For KeyName, choose an existing EC2 key pair. You will use it to connect by using SSH to the Jenkins server. Ensure that you have access to the private key of this key pair.

For PublicSubnet1, choose a public subnet where the load balancer, Jenkins server, and CodeDeploy web servers will be launched.

For PublicSubnet2, choose a public subnet where the load balancers and CodeDeploy web servers will be launched.

For VpcId, choose the VPC for the public subnets you used in PublicSubnet1 and PublicSubnet2.

For YourIPRange, type the CIDR block of the network from where you will connect to the Jenkins server using HTTP and SSH. If your local machine has a static public IP address, find it by going to https://www.whatismyip.com/ and then entering it followed by a ‘/32’. If you do not have a static IP address (or aren’t sure if you have one), you may enter ‘0.0.0.0/0’ in this field then any address can reach your Jenkins server.

7. On the Review page, select the I acknowledge that this template might cause AWS CloudFormation to create IAM resources check box, and then choose Create.

8. Wait for the CloudFormation stack status to change to CREATE_COMPLETE. This will take approximately 6-10 minutes.

 

9. Note the values displayed on the Outputs tab. You’ll need them later.

10. Point your browser to the ELBDNSName from the Outputs tab and verify that you can see the Sample Application page.

Secure Jenkins

Point your browser to the JenkinsServerDNSName (for example, ec2-54-163-4-211.compute-1.amazonaws.com) from the Outputs tab. You should be able to see the Jenkins home page:

The Jenkins installation is currently accessible through the Internet without any form of authentication. Before proceeding to the next step, let’s secure Jenkins. On the Jenkins home page, choose Manage Jenkins. Choose Configure Global Security, and then to enable Jenkins security, select the Enable security check box.

Under Security Realm, choose Jenkins’s own user database and select the Allow users to sign up check box. Under Authorization, choose Matrix-based security. Add a user (for example, admin) and give this user all privileges. Save your changes.

Now you will be asked to provide a user name and password for the user. Choose Create an account, provide the user name (for example, admin), a strong password, and then complete the user details. Now you will be able to sign in securely to Jenkins.

Create a Project and Configure the CodeDeploy Jenkins Plugin

Now we’ll create a project in Jenkins and configure the Jenkins plugin to poll for code updates from the AWS CodeCommit repository.

1. Sign in to Jenkins with the user name and password you created earlier.

2. Choose New Item, and then choose Freestyle project. Type a name for the project (for example, CodeDeployApp), and then choose OK.

3. On the project configuration page, under Source Code Management, choose Git. Paste the URL you noted when you created the AWS CodeCommit repository (step 5).

4. In Build Triggers, select the Poll SCM check box. In the Schedule text field, type H/2 * * * *. This tells Jenkins to poll CodeCommit every two minutes for updates. (This may be too frequent for production use, but it works well for testing because it returns results frequently.)

5. Under Post-build Actions, choose Add post-build actions, and then select the Deploy an application to AWS CodeDeploy check box.

6. Paste the values you noted on the Outputs tab when you created the CloudFormation stack (step 9):

For AWS CodeDeploy Application Name, paste the value of CodeDeployApplicationName.

For AWS CodeDeploy Deployment Group, paste the value of CodeDeployDeploymentGroup.

For AWS CodeDeploy Deployment Config, type CodeDeployDefault.OneAtATime.

For AWS Region, choose the region where you created the CodeDeploy environment.

For S3 Bucket, paste the value of S3BucketName.

Leave the other settings at their default (blank).

7. Choose Use temporary credentials, and then paste the value of JenkinsCodeDeployRoleArn that appeared in the CloudFormation output.

Note the External ID field displayed on this page. This is a unique random ID generated by the CodeDeploy Jenkins plugin. This ID can be used to add a condition to the IAM role to ensure that only the plugin can assume this role. To keep things simple, we will not use the External ID as a condition, but we strongly recommend you use it for added protection in a production scenario, especially when you are using cross-account IAM roles.

 

8. Choose Test Connection.

 

9. Confirm the text “Connection test passed” appears, and then choose Save to save your settings.

Add Files to the CodeCommit Repository

Now, we’ll use the git command-line tool to clone the AWS CodeCommit repository and then add files to it. These steps show you how to use SSH to connect to the Jenkins server. If you are more comfortable with Git integrated in your IDE, follow the steps in the CodeCommit documentation to clone the repository and add files to it.

1. Use SSH to connect to the public DNS name of the EC2 instance for Jenkins (JenkinsServerDNSName from the Outputs tab) and sign in as the ec2-user. Run the following commands to configure git. Replace the values enclosed in quotes with your name and email address.

$ aws configure set region us-east-1
$ aws configure set output json
$ git config –global credential.helper ‘!aws codecommit credential-helper [email protected]
$ git config –global credential.useHttpPath true
$ git config –global user.name "YOUR NAME"
$ git config –global user.email "[email protected]"

2. Clone the repository you created in the previous step.

$ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/DemoRepository
Cloning into ‘DemoRepository’…
warning: You appear to have cloned an empty repository.
Checking connectivity… done.

3. Switch to the DemoRepository directory:

$ cd DemoRepository/

4. Now we’ll download the source for the Sample CodeDeploy application.

$ curl -OLs http://aws-codedeploy-us-east-1.s3.amazonaws.com/samples/latest/SampleApp_Linux.zip

5. Unzip the downloaded file:

$ unzip SampleApp_Linux.zip
Archive: SampleApp_Linux.zip
extracting: scripts/install_dependencies
extracting: scripts/start_server
inflating: scripts/stop_server
inflating: appspec.yml
inflating: index.html
inflating: LICENSE.txt

6. Delete the ZIP file:

$ rm SampleApp_Linux.zip

7. Use a text editor to edit the index.html file:

$ vi index.html

8. Scroll down to the body tag and add the highlighted text:

9. Save the file and close the editor.

10. Add the files to git and commit them with a comment:

$ git add appspec.yml index.html LICENSE.txt scripts/*
$ git commit -m "Initial versions of files"

11. Now push these updates to CodeCommit:

$ git push

12. If your updates have been successfully pushed to CodeCommit, you should see something like the following:

Counting objects: 9, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (9/9), 5.05 KiB | 0 bytes/s, done.
Total 9 (delta 0), reused 0 (delta 0)
remote:
To https://git-codecommit.us-east-1.amazonaws.com/v1/repos/DemoRepository
* [new branch] master -> master

13. On the Jenkins dashboard, choose the CodeDeployApp project.

14. Choose Git Polling Log to see the results of polling git for updates. There may be a few failed polls from earlier when the repository was empty.

15. Within two minutes of pushing updates, a new build with a Build ID (for example, #2 or #3) should appear in the build history.

16. Choose the most recent build. On the Build details page, choose Console Output to view output from the build.

17. At the bottom of the output, check that the status of the build is SUCCESS.

18. In the CodeDeploy console, choose AWS CodeDeploy, and then choose Deployments.

 

19. Confirm that there are two deployments: the initial deployment created by CloudFormation and a recent deployment of the latest code from AWS CodeCommit. Confirm that the status of the recent deployment is Succeeded.

 

20. Point your browser to the ELBDNSName from the Outputs tab of CloudFormation. Confirm that the text “This version was deployed by Jenkins” appears on the page.

Congratulations, you have now successfully set up the CodeDeploy Jenkins plugin and used it to automatically deploy a revision to CodeDeploy when code updates are pushed to AWS CodeCommit.

You can experiment by committing more changes to the code and then pushing them to deploy the updates automatically.

Cleaning Up

In this section, we’ll delete the resources we’ve created so that you will not be charged for them going forward.

1. Sign in to the Amazon S3 console and choose the S3 bucket you created earlier. The bucket name will start with “jenkinscodedeploy-codedeploybucket.” Choose all files in the bucket, and from Actions, choose Delete.

2. Choose OK to confirm the deletion.

3. In the CloudFormation console, choose the stack named “JenkinsCodeDeploy,” and from Actions, choose Delete Stack. Refresh the Events tab of the stack until the stack disappears from the stack list.

Setting Up the Jenkins Plugin for AWS CodeDeploy

Post Syndicated from Shankar Sivadasan original http://blogs.aws.amazon.com/application-management/post/TxMJROUIFQZ4HS/Setting-Up-the-Jenkins-Plugin-for-AWS-CodeDeploy

The following is a guest post by Maitreya Ranganath, Solutions Architect.

In this post, we’ll show you how to use the Jenkins plugin to automatically deploy your builds with AWS CodeDeploy. We’ll walk through the steps for creating an AWS CodeCommit repository, installing Jenkins and the Jenkins plugin, adding files to the CodeCommit repository, and configuring the plugin to create a deployment when changes are committed to an AWS CodeCommit repository.

Create an AWS CodeCommit Repository

First, we will create an AWS CodeCommit repository to store our sample code files.

1. Sign in to the AWS Management Console and open the AWS CodeCommit console in the us-east-1 (N. Virginia) Region.  Choose Get Started or Create Repository.

2. For Repository Name, type a name for your repository (for example, DemoRepository). For Description, type Repository for Jenkins Code Deploy.

3. Choose the Create repository button.

4. Choose the repository you just created to view its details.

5. Choose the Clone URL button, and then choose HTTPS. Copy the URL displayed into a clipboard. You’ll need it later to configure Jenkins.

 

Now that you have created an AWS CodeCommit repository, we’ll create a Jenkins server and AWS CodeDeploy environment.

Create a Jenkins Server and AWS CodeDeploy Environment

In this step, we’ll launch a CloudFormation template that will create the following resources:

An Amazon S3 bucket that will be used to store deployment files.

JenkinsRole, an IAM role and instance profile for the Amazon EC2 instance that will run Jenkins. This role allows Jenkins on the EC2 instance to assume the CodeDeployRole and access repositories in CodeCommit.

CodeDeployRole, an IAM role assumed by the CodeDeploy Jenkins plugin. This role has permissions to write files to the S3 bucket created by this template and to create deployments in CodeDeploy.

Jenkins server, an EC2 instance running Jenkins.

An Auto Scaling group of EC2 instances running Apache and the CodeDeploy agent fronted by an Elastic Load Balancing load balancer.

To create the CloudFormation stack, choose the link that corresponds to the AWS region where you want to work:

For the us-east-1 region:

or use the link below:

https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=JenkinsCodeDeploy&templateURL=https://s3.amazonaws.com/aws-codedeploy-us-east-1/templates/latest/CodeDeploy_SampleCF_Jenkins_Integration.json

For the us-west-2 region:

or use the link below:

https://console.aws.amazon.com/cloudformation/home?region=us-west-2#/stacks/new?stackName=JenkinsCodeDeploy&templateURL=https://s3.amazonaws.com/aws-codedeploy-us-east-1/templates/latest/CodeDeploy_SampleCF_Jenkins_Integration.json

6. Choose Next and specify the following values:

For InstanceCount, accept the default of 3. (Three EC2 instances will be launched for CodeDeploy.)

For InstanceType, accept the default of t2.medium.

For KeyName, choose an existing EC2 key pair. You will use it to connect by using SSH to the Jenkins server. Ensure that you have access to the private key of this key pair.

For PublicSubnet1, choose a public subnet where the load balancer, Jenkins server, and CodeDeploy web servers will be launched.

For PublicSubnet2, choose a public subnet where the load balancers and CodeDeploy web servers will be launched.

For VpcId, choose the VPC for the public subnets you used in PublicSubnet1 and PublicSubnet2.

For YourIPRange, type the CIDR block of the network from where you will connect to the Jenkins server using HTTP and SSH. If your local machine has a static public IP address, find it by going to https://www.whatismyip.com/ and then entering it followed by a ‘/32’. If you do not have a static IP address (or aren’t sure if you have one), you may enter ‘0.0.0.0/0’ in this field then any address can reach your Jenkins server.

7. On the Review page, select the I acknowledge that this template might cause AWS CloudFormation to create IAM resources check box, and then choose Create.

8. Wait for the CloudFormation stack status to change to CREATE_COMPLETE. This will take approximately 6-10 minutes.

 

9. Note the values displayed on the Outputs tab. You’ll need them later.

10. Point your browser to the ELBDNSName from the Outputs tab and verify that you can see the Sample Application page.

Secure Jenkins

Point your browser to the JenkinsServerDNSName (for example, ec2-54-163-4-211.compute-1.amazonaws.com) from the Outputs tab. You should be able to see the Jenkins home page:

The Jenkins installation is currently accessible through the Internet without any form of authentication. Before proceeding to the next step, let’s secure Jenkins. On the Jenkins home page, choose Manage Jenkins. Choose Configure Global Security, and then to enable Jenkins security, select the Enable security check box.

Under Security Realm, choose Jenkins’s own user database and select the Allow users to sign up check box. Under Authorization, choose Matrix-based security. Add a user (for example, admin) and give this user all privileges. Save your changes.

Now you will be asked to provide a user name and password for the user. Choose Create an account, provide the user name (for example, admin), a strong password, and then complete the user details. Now you will be able to sign in securely to Jenkins.

Create a Project and Configure the CodeDeploy Jenkins Plugin

Now we’ll create a project in Jenkins and configure the Jenkins plugin to poll for code updates from the AWS CodeCommit repository.

1. Sign in to Jenkins with the user name and password you created earlier.

2. Choose New Item, and then choose Freestyle project. Type a name for the project (for example, CodeDeployApp), and then choose OK.

3. On the project configuration page, under Source Code Management, choose Git. Paste the URL you noted when you created the AWS CodeCommit repository (step 5).

4. In Build Triggers, select the Poll SCM check box. In the Schedule text field, type H/2 * * * *. This tells Jenkins to poll CodeCommit every two minutes for updates. (This may be too frequent for production use, but it works well for testing because it returns results frequently.)

5. Under Post-build Actions, choose Add post-build actions, and then select the Deploy an application to AWS CodeDeploy check box.

6. Paste the values you noted on the Outputs tab when you created the CloudFormation stack (step 9):

For AWS CodeDeploy Application Name, paste the value of CodeDeployApplicationName.

For AWS CodeDeploy Deployment Group, paste the value of CodeDeployDeploymentGroup.

For AWS CodeDeploy Deployment Config, type CodeDeployDefault.OneAtATime.

For AWS Region, choose the region where you created the CodeDeploy environment.

For S3 Bucket, paste the value of S3BucketName.

Leave the other settings at their default (blank).

7. Choose Use temporary credentials, and then paste the value of JenkinsCodeDeployRoleArn that appeared in the CloudFormation output.

Note the External ID field displayed on this page. This is a unique random ID generated by the CodeDeploy Jenkins plugin. This ID can be used to add a condition to the IAM role to ensure that only the plugin can assume this role. To keep things simple, we will not use the External ID as a condition, but we strongly recommend you use it for added protection in a production scenario, especially when you are using cross-account IAM roles.

 

8. Choose Test Connection.

 

9. Confirm the text “Connection test passed” appears, and then choose Save to save your settings.

Add Files to the CodeCommit Repository

Now, we’ll use the git command-line tool to clone the AWS CodeCommit repository and then add files to it. These steps show you how to use SSH to connect to the Jenkins server. If you are more comfortable with Git integrated in your IDE, follow the steps in the CodeCommit documentation to clone the repository and add files to it.

1. Use SSH to connect to the public DNS name of the EC2 instance for Jenkins (JenkinsServerDNSName from the Outputs tab) and sign in as the ec2-user. Run the following commands to configure git. Replace the values enclosed in quotes with your name and email address.

$ aws configure set region us-east-1
$ aws configure set output json
$ git config –global credential.helper ‘!aws codecommit credential-helper [email protected]
$ git config –global credential.useHttpPath true
$ git config –global user.name "YOUR NAME"
$ git config –global user.email "[email protected]"

2. Clone the repository you created in the previous step.

$ git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/DemoRepository
Cloning into ‘DemoRepository’…
warning: You appear to have cloned an empty repository.
Checking connectivity… done.

3. Switch to the DemoRepository directory:

$ cd DemoRepository/

4. Now we’ll download the source for the Sample CodeDeploy application.

$ curl -OLs http://aws-codedeploy-us-east-1.s3.amazonaws.com/samples/latest/SampleApp_Linux.zip

5. Unzip the downloaded file:

$ unzip SampleApp_Linux.zip
Archive: SampleApp_Linux.zip
extracting: scripts/install_dependencies
extracting: scripts/start_server
inflating: scripts/stop_server
inflating: appspec.yml
inflating: index.html
inflating: LICENSE.txt

6. Delete the ZIP file:

$ rm SampleApp_Linux.zip

7. Use a text editor to edit the index.html file:

$ vi index.html

8. Scroll down to the body tag and add the highlighted text:

9. Save the file and close the editor.

10. Add the files to git and commit them with a comment:

$ git add appspec.yml index.html LICENSE.txt scripts/*
$ git commit -m "Initial versions of files"

11. Now push these updates to CodeCommit:

$ git push

12. If your updates have been successfully pushed to CodeCommit, you should see something like the following:

Counting objects: 9, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (7/7), done.
Writing objects: 100% (9/9), 5.05 KiB | 0 bytes/s, done.
Total 9 (delta 0), reused 0 (delta 0)
remote:
To https://git-codecommit.us-east-1.amazonaws.com/v1/repos/DemoRepository
* [new branch] master -> master

13. On the Jenkins dashboard, choose the CodeDeployApp project.

14. Choose Git Polling Log to see the results of polling git for updates. There may be a few failed polls from earlier when the repository was empty.

15. Within two minutes of pushing updates, a new build with a Build ID (for example, #2 or #3) should appear in the build history.

16. Choose the most recent build. On the Build details page, choose Console Output to view output from the build.

17. At the bottom of the output, check that the status of the build is SUCCESS.

18. In the CodeDeploy console, choose AWS CodeDeploy, and then choose Deployments.

 

19. Confirm that there are two deployments: the initial deployment created by CloudFormation and a recent deployment of the latest code from AWS CodeCommit. Confirm that the status of the recent deployment is Succeeded.

 

20. Point your browser to the ELBDNSName from the Outputs tab of CloudFormation. Confirm that the text “This version was deployed by Jenkins” appears on the page.

Congratulations, you have now successfully set up the CodeDeploy Jenkins plugin and used it to automatically deploy a revision to CodeDeploy when code updates are pushed to AWS CodeCommit.

You can experiment by committing more changes to the code and then pushing them to deploy the updates automatically.

Cleaning Up

In this section, we’ll delete the resources we’ve created so that you will not be charged for them going forward.

1. Sign in to the Amazon S3 console and choose the S3 bucket you created earlier. The bucket name will start with “jenkinscodedeploy-codedeploybucket.” Choose all files in the bucket, and from Actions, choose Delete.

2. Choose OK to confirm the deletion.

3. In the CloudFormation console, choose the stack named “JenkinsCodeDeploy,” and from Actions, choose Delete Stack. Refresh the Events tab of the stack until the stack disappears from the stack list.

How to Set Up SSO to the AWS Management Console for Multiple Accounts by Using AD FS and SAML 2.0

Post Syndicated from Alessandro Martini original https://blogs.aws.amazon.com/security/post/Tx2989L4392V75K/How-to-Set-Up-SSO-to-the-AWS-Management-Console-for-Multiple-Accounts-by-Using-A

AWS supports Security Assertion Markup Language (SAML) 2.0, an open standard for identity federation used by many identity providers (IdPs). SAML enables federated single sign-on (SSO), which enables your users to sign in to the AWS Management Console or to make programmatic calls to AWS APIs by using assertions from a SAML-compliant IdP. Many of you maintain multiple AWS accounts (for example, production, development, and test accounts), and have asked how to use SAML to enable identity federation to those accounts. Therefore, in this blog post I will demonstrate how you can enable federated users to access the AWS Management Console with multiple AWS accounts and SAML.

If you use Microsoft Active Directory for corporate directories, you may already be familiar with how Active Directory and AD FS work together to enable federation, as described in the AWS Security Blog post, Enabling Federation to AWS Using Windows Active Directory, AD FS, and SAML 2.0. As a result, I decided to use Active Directory with AD FS as the example IdP in this post.

To automate both the installation and configuration of AD FS and Active Directory, I will use Windows PowerShell in this post. By leveraging Windows PowerShell, you eliminate the manual installation and configuration steps, and allow yourself to focus on the high-level process.

If you want to manage access to all your AWS accounts with Active Directory and AD FS, you’ve come to the right place!

Background

To set up your Windows Active Directory domain, you have many options. You can use an Amazon EC2 instance and set up your own domain with dcpromo or by installing the Active Directory role (if using Windows Server 2012 and later). You can automate this process by using an AWS CloudFormation template that creates a Windows instance and sets up a domain for you. Alternatively, you may want to create a Simple AD with AWS Directory Service. Information about how to manage these directories, join EC2 instances to the domain, and create users and groups is in our documentation.

First things first

With SAML federation, AWS requires the IdP to issue a SAML assertion with some mandatory attributes (known as claims). This AWS documentation explains how to configure the SAML assertion. In short, you need the assertion to contain:

An attribute of name https://aws.amazon.com/SAML/Attributes/Role (note this is not a URL to a resource, but a custom attribute for our AWS Security Token Service [STS]). Its value must be at least one role/provider pair as a comma-separated list of their Amazon Resource Names (ARNs). Because the ARNs are unique per AWS account, this information tells AWS to which account you want to federate.

An attribute of name https://aws.amazon.com/SAML/Attributes/RoleSessionName (again, this is just a definition of type, not an actual URL) with a string value. This is the federated user’s friendly name in AWS.

A name identifier (NameId) that is used to identify the subject of a SAML assertion.

AWS has recently published troubleshooting steps in our documentation about how to debug a SAML response from your IdP. In my experience, the problem usually is related to the three attributes mentioned above: they are either missing, misspelled (remember the names are cAsE sEnSitiVe!), or they don’t contain the expected values. If you are struggling with SAML federation, you should always start by first collecting a copy of the SAML response you are sending to AWS.

Don’t know how to collect a copy of the SAML response? Check our documentation, and then decode and troubleshoot the response.

Use case

A company, Example Corp., wants:

Federated identity access for specific groups of users in its organization.

To manage federation across multiple AWS accounts.

To deal with three populations of users:

Users that will access 1 account with 1 role (1:1).

Users that will access multiple accounts with 1 role (N:1).

Users that will access multiple accounts with multiple roles (N:M).

Example Corp. is using Active Directory, and they want to use AD FS to manage federation centrally. Example Corp. wants to federate to these two AWS accounts: 123456789012 and 111122223333.

Prepare your environment

The blog post, Enabling Federation to AWS Using Windows Active Directory, AD FS, and SAML 2.0, shows how to prepare Active Directory and install AD FS 2.0 on Windows Server 2008 R2. In this blog post, we will install AD FS 3.0 on Windows Server 2012 R2. AD FS 3.0 cannot be installed on Windows Server 2008 R2 and earlier, so make sure you pick the right version of Windows Server. The AD FS 3.0 and AD FS 2.0 installations and configurations are very similar; therefore, I decided to use this blog post as a chance to show how to do the same with Windows PowerShell. I will report the steps from the GUI as well, but more as additional reading at the end of the blog post.

I like how Windows PowerShell can make the configuration steps easy. To make things even easier for you here, I have kept the same naming convention from Jeff Wierer’s blog post for the claim rules, Active Directory groups, and IAM entities.

The .zip file with the collection of related scripts contains:

Two folders: Logs (where log files are stored) and Utilities (where this PowerShell script is saved).

The following scripts:

00-Configure-AD.ps1 – It simplifies Active Directory group and user creation as well as the configuration required to leverage the federation solution explained in this post.

01-Install-ADFS.ps1 – It installs AD FS 3.0 on Windows Server 2012 R2 and downloads the federation metadata.

02-Configure-IAM.ps1 – It creates an identity provider and two IAM roles in the AWS account you choose.

03-Configure-ADFS.ps1 – It creates a relying party trust to AWS by using the following templates:

auth.txt

claims.txt

Extract the file on the Windows Server 2012 R2 computer you designated for the AD FS 3.0 installation. Also, install AWS Tools for Windows PowerShell on that computer, because this is required to complete the IAM configuration from the command line. You don’t need to configure a credential profile at this time.

General workflow

These are the steps of the general workflow:

The user goes to the AD FS sign-in page to authenticate.

AD FS authenticates the user against Active Directory.

Active Directory returns the user’s information.

AD FS dynamically builds ARNs by using Active Directory group memberships for the IAM roles and user attributes for the AWS account IDs, and sends a signed assertion to AWS STS.

The user gets to the AWS role selection page, where he can choose which account to access and which role to assume.

AWS STS is the single point of access for all SAML-federated access. The ARNs in the SAML response are used to identify your SAML provider and IAM role in your destination account. The following section explains how to simplify administration for the ARNs in your AD FS server, providing custom claim rule code examples.

The solution in action

I want to start from the end to show you how the user experience will look.

I have a user called Bob who is a member of two Active Directory groups:

AWS-Dev

AWS-Production

Note: You need to enable View > Advanced Features in Active Directory Users and Computers to see the Attribute editor tab.

This user has two AWS account IDs in the url attribute, as shown in the following images.

Bob then connects to https://adfs.example.com/adfs/ls/idpinitiatedosignon.aspx, where he can pick Amazon Web Services as the destination application after he has authenticated, as shown in the following image.

When Bob gets to the AWS role selection page, he gets 4 possible choices (2 choices for each of the 2 accounts displayed) as the combination of the groups he belongs to and the AWS account IDs from the url attribute, as shown in the following image. Thanks to the new role selection for SAML-based single sign-on, it is easier for the user to understand the destination account he would access.

This workflow is summarized in the following diagram.

Let’s now see how to use my Windows PowerShell scripts to set up this solution.

Active Directory configuration (Windows PowerShell: 00-Configure-AD.ps1)

The first script, 00-Configure-AD.ps1, can be used to create two Active Directory groups (AWS-Production and AWS-Dev) and a user with a password of your choice. The script asks you many questions so that you can either create new users or assign permissions to already existing users. Let’s see how it works in more detail.

To run the scripts, I launch Windows PowerShell with administrative privileges (see the following screenshot) from the server where I will install AD FS 3.0. The machine is already joined to the example.com domain, so I connect using my Domain Administrator user, Alessandro.

I download the scripts to my desktop, and after unzipping the file, I launch the script located in my AD FS folder on my desktop:

PS C:UsersalessandroDesktopADFS> .0-Configure-AD.ps1

The script will ask some questions about what you want to do. In order, it will ask for (based on your answers):

Active Directory AWS groups creation.

Do you want to create two AD groups called AWS-Production and AWS-Dev?

AD FS service account creation.

Do you want to create an AD FS service account? A user name and password will be requested.

How many new Active Directory users do you want to create?

List the AWS account IDs you want this user to access (for example, 123456789012, 111122223333).

Active Directory group membership for AWS access.

What level of access do you want to grant?

How many existing Active Directory users do you want to grant access to AWS?

Type the user name of the user you want to manage.

AWS account association.

Do you want to keep the existing AWS account associations?

Check the current Active Directory group membership for AWS access.

Do you want to keep [GROUP MEMBERSHIP]?

Active Directory group membership for AWS access.

What level of access do you want to grant?

The following screenshot shows the workflow for the creation of user Bob, which is assigned to the AWS accounts 123456789012 and 111122223333; he also is a member of AWS-Production.

My answers to the questions of the script are in red:

Active Directory AWS groups creation

Do you want to create two AD groups called AWS-Production and AWS-Dev? Y

AD FS service account creation

Do you want to create an AD FS service account? User name and password will be requested. Y

A credential request window allows me to type the user name and password for the user creation.

How many new Active Directory users do you want to create? 1

A credential request window allows me to type the user name and password for the user creation.

List the AWS account IDs you want this user to access (such as 123456789012,111122223333)  123456789012,111122223333

Active Directory group membership for AWS access

What level of access do you want to grant? P

How many existing Active Directory users do you want to grant access to AWS? 0

If you don’t need to create the Active Directory groups AWS-Production and AWS-Dev, you can simply type N when asked. You can do the same thing for the AD FS service account.

The provided script and the steps just outlined do the following in your domain:

Create two Active Directory Groups named AWS-Production and AWS-Dev.

Create the AD FS service account ADFSSVC. This account will be used as the AD FS service account later on. This account is not associated with any Active Directory group because this is a service account.

Create a user named Bob.

Give Bob an email address ([email protected]). This is automatically done by the script by combining the user name and the domain name.

Associate Bob with two AWS account IDs: 123456789012 and  111122223333

Add Bob to the AWS-Production group.

If you have an existing user you want to manage, you can run the script again and tell the script how many existing users you want to manage: How many existing Active Directory users do you want to grant access to AWS?

In the following example, I don’t need to create the Active Directory groups and AD FS service account again, but I manage Bob (which now already exists) and add him to the AWS-Dev group.

These are my answers to the questions:

Active Directory AWS groups creation

Do you want to create two AD groups called AWS-Production and AWS-Dev? N

AD FS service account creation

Do you want to create an AD FS service account? User name and password will be requested. N

How many new Active Directory users do you want to create? 0

How many existing Active Directory users do you want to grant access to AWS? 1

Enter the user name of the user you want to manage. Bob

AWS account association.

Do you want to keep the existing AWS account associations? Y

Check the current Active Directory group membership for AWS access.

Do you want to keep [GROUP MEMBERSHIP]? Y

Active Directory group membership for AWS access.

What level of access do you want to grant? D

The code is available to you, and it can be adjusted to run in noninteractive mode and to accept parameters to run in a batch script.

AD FS installation (Windows PowerShell: 01-Install-ADFS.ps1)

The script I will use now is 01-Install-ADFS.ps1. This script will install the AD FS 3.0 Windows role, create a self-signed certificate, and configure AD FS for the first use. The configuration will ask for the credentials of the service account you have created before (ADFSSVC). Please note you should provide the user with the NETBIOS (for example, EXAMPLEadfssvc).

Before you can move to the next step and create a SAML provider in IAM, you need the federation metadata document for your AD FS federation server, which you can download from https://<yourservername>/FederationMetadata/2007-06/FederationMetadata.xml. The federation metadata is automatically downloaded in the same folder as the script (the downloaded file is called federationmetadata.xml).

The warning about the SPN is a known issue that can be fixed by running the following command at the command line (make sure you run the command line as an administrator):

setspn -a host/localhost adfssvc

Note that adfssvc is the name of the service account I used.

If the command is successful, you will see output like this:

Registering ServicePrincipalNames for
CN=ADFSSVC,CN=Users, DC=example,DC=com    
host/localhost

IAM configuration (Windows PowerShell: 02-Configure-IAM.ps1)

The next script, 02-Configure-IAM.ps1, will create an identity provider and 2 IAM roles in a specified AWS account. The SAML provider name that is created is ADFS, and the IAM roles are called ADFS-Production and ADFS-Dev. The roles trust the SAML provider ADFS.

The script will first ask how many AWS accounts you want to configure. Here is the question and my answer (in red): How many AWS accounts do you want to configure? 1

You will now be asked for the IAM access key and secret access keys for each of the AWS accounts you want to configure. The IAM user is used to create the required IAM entities, and it must have enough permissions to create an identity provider and a role.

Note: The script creates a SAML provider called ADFS and 2 IAM roles called ADFS-Dev and ADFS-Production. If you are creating these IAM objects manually, remember that you need to use the same names that I use in this blog post. This solution assumes the SAML provider and the IAM role names are the same across all the AWS accounts.

Here is the output that I got, which includes the IAM roles information.

I can then run the script again for the other AWS account, 111122223333.

AD FS configuration (Windows PowerShell: 03-Configure-ADFS.ps1)

The last script (03-Configure-ADFS.ps1) configures AD FS by creating the AWS relying party trust. All the required claim rules are added. You can see the rules in the .txt files that are in the .zip file you downloaded at the beginning of this process). If you are interested about the logic behind the code and how I came up with it, see the “Under the hood” section near the end of this blog post

auth.txt

@RuleTemplate = "AllowAllAuthzRule"
=> issue(Type = "http://schemas.microsoft.com/authorization/claims/permit", Value =
"true");

claims.txt

@RuleTemplate = "MapClaims"
@RuleName = "NameId"
c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/primarysid"]
 => issue(Type = "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier", Issuer = c.Issuer, OriginalIssuer = c.OriginalIssuer, Value = c.Value, ValueType = c.ValueType, Properties["http://schemas.xmlsoap.org/ws/2005/05/identity/claimproperties/format"] = "urn:oasis:names:tc:SAML:2.0:nameid-format:persistent");
 
@RuleTemplate = "LdapClaims"
@RuleName = "RoleSessionName"
c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Issuer == "AD AUTHORITY"]
 => issue(store = "Active Directory", types = ("https://aws.amazon.com/SAML/Attributes/RoleSessionName"), query = ";mail;{0}", param = c.Value);
 
@RuleName = "Get AD Groups"
c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Issuer == "AD AUTHORITY"]
 => add(store = "Active Directory", types = ("http://temp/variable"), query = ";tokenGroups;{0}", param = c.Value);
 
@RuleName = "Get AWS Accounts from User attributes"
c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname"]
 => add(store = "Active Directory", types = ("http://temp/AWSAccountsFromUser"), query = ";url;{0}", param = c.Value);
 
@RuleName = "Dynamic ARN – Adding AWS Accounts"
c:[Type == "http://temp/AWSAccountsFromUser"]
 => add(Type = "http://temp/AWSAccountsFromUser2", Value = RegExReplace("arn:aws:iam::AWSACCOUNT:saml-provider/ADFS,arn:aws:iam::AWSACCOUNT:role/ADFS-", "AWSACCOUNT", c.Value));
 
@RuleName = "Dynamic ARN – Adding Roles"
c1:[Type == "http://temp/AWSAccountsFromUser2"]
 && c2:[Type == "http://temp/variable", Value =~ "(?i)^AWS-"]
 => issue(Type = "https://aws.amazon.com/SAML/Attributes/Role", Value = RegExReplace(c2.Value, "AWS-", c1.Value)); 

Run the script in a Windows PowerShell window launched as administrator, as shown in the following image.

The browser should be automatically launched with the AD FS sign-in page.

You will notice that an application is already set up (Amazon Web Services). You can now authenticate with the user Bob (or whoever you have previously configured with the Windows PowerShell script 00-Configure-AD.ps1), and you should be able to federate to AWS.

If you can’t access your AWS accounts, start troubleshooting by first collecting a copy of the SAML response you are sending to AWS (this is explained in our documentation). You can then decode and troubleshoot it.

How to handle exceptions

The solution presented so far works well if you have users with the same permissions across different accounts. However, what if a user must have Production access in one account and only Dev access in a second account? I will now show a few additional claim rules you can add at the end of the claim rules chain. Based on your needs, you can pick the claim rule code for the exception that you need. Because this is a case-by-case choice, I will show how to manage the exceptions in the UI. First, you need to open the AD FS Microsoft Management Console on your AD FS server.

Expand Trust Relationships, click Relying Party Trusts, right-click the relying party trust Amazon Web Services, and then click Edit Claim Rules.

You then should see the 6 rules shown in the following image.

Each of the following paragraphs will explain how to add a seventh rule. You can add as many rules as needed to manage multiple exceptions at the same time.

Exception—Static ARNs for DOMAINuser

With the following custom claim rule, we check the Windows account name of the authenticated user. If the user matches our condition, we issue specific ARNs for him.

You can place this additional custom claim rule after all the other claim rules you have already created, as shown in the following image.

The next rule—Exception – Static ARNs for DOMAINuser—is another custom claim rule. Follow these steps to create it.

In the Edit Claim Rules for Amazon Web Services dialog box, click Add Rule.

In the Claim rule template list, select Send Claims Using a Custom Rule, and then click Next.

For Claim rule name, type Exception – Static ARNs for DOMAINuser, and then in Custom rule, enter the following:

c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Value == "DOMAINusername"]
 => issue(Type = "https://aws.amazon.com/SAML/Attributes/Role", Value = "arn:aws:iam::YOURACCOUNTID:saml-provider/ADFS,arn:aws:iam::YOURACCOUNTID:role/ADFS-Dev");

Code explanation

If the user is DOMAINusername, issue a claim of type https://aws.amazon.com/SAML/Attributes/Role with value:

"arn:aws:iam::YOURACCOUNTID:saml-provider/ADFS,arn:aws:iam::YOURACCOUNTID:role/ADFS-Dev”

In this example, user EXAMPLEBob would be granted access to ADFS-Dev in the specified account.

Now, Bob will be able to pick ADFS-Dev from account 444455556666 as well.

This workflow is summarized in the following diagram.

EXAMPLEBob goes to the AD FS sign-in page to authenticate.

AD FS authenticates the user against Active Directory.

Active Directory returns the user’s information. EXAMPLEBob belongs to two AD groups (AWS-Production and AWS-Dev) and his user object attribute refers to two AWS accounts (123456789012 and 111122223333).

AD FS dynamically builds four ARNs by using Active Directory group memberships for the IAM roles and user attributes for the AWS account IDs. Additionally, AD FS adds the ARNs for a third AWS account (444455556666) with a single IAM role (ADFS-Dev) and sends a signed assertion to STS.

EXAMPLEBob gets to the AWS role selection page, where he can choose between accounts 123456789012, 111122223333, and 444455556666. In the first two accounts, he can choose between ADFS-Production and ADFS-Dev. For account 444455556666, only ADFS-Dev is available. All of these are IAM roles created in the specific accounts.

Exception—Static ARNs for anyone

This exception is to grant access to certain AWS accounts and IAM roles to any authenticated user. Again, you can place this exception at the end of the claim rules chain you have defined so far.

Follow these steps to create this customer claim rule:

In the Edit Claim Rules for Amazon Web Services dialog box, click Add Rule.

In the Claim rule template list, select Send Claims Using a Custom Rule, and then click Next.

For Claim rule name, type Exception – Static ARNs for anyone, and then in Custom rule, enter the following code. Make sure to change the parts in red to your AWS account ID and the required IAM role name.

c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname"]
 => issue(Type = "https://aws.amazon.com/SAML/Attributes/Role", Value = "arn:aws:iam::YOURACCOUNTID:saml-provider/ADFS,arn:aws:iam::YOURACCOUNTID:role/ADFS-Dev");

You can iterate the same logic for as many accounts as you need. The result is that anyone has been granted access to the specified account (444455556666) with the specified IAM role (ADFS-Dev).

Code explanation

If there is an incoming claim for an authenticated user (http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname), you issue a claim of type https://aws.amazon.com/SAML/Attributes/Role with value  "arn:aws:iam::YOURACCOUNTID:saml-provider/ADFS,arn:aws:iam::YOURACCOUNTID:role/ADFS-Dev”. In this example, any authenticated user will be granted access to ADFS-Dev in the specified account.

This workflow is summarized in the following diagram.

A domain user goes to the AD FS login page to authenticate.

AD FS authenticates the user against Active Directory.

Active Directory returns the user’s information. Let’s assume this domain user does not belong to any Active Directory group that starts with AWS- (in other words, AWS-Production, AWS-Dev), and his user object attribute contains no AWS account.

AD FS has no information to build dynamic ARNs from Active Directory group memberships and user attributes. AD FS has a rule to generate static ARNs for any authenticated user for a specific AWS account (444455556666) with a single IAM role (ADFS-Dev), and sends a signed assertion to AWS STS.

The domain user can only assume one role in one AWS account. Therefore, the AWS role selection page is automatically skipped. The domain user will get access to account 444455556666 with the ADFS-Dev role, which must be created in the AWS account before any access is attempted.

Handling exceptions with an ad hoc attribute store (Microsoft SQL Server)

AD FS can retrieve information from Active Directory. Additionally, AD FS provides built-in capabilities to read information from a SQL database. For convenience, I am using Microsoft SQL Server.

I launched an EC2 instance with SQL Server installed, and I joined it to my Active Directory Domain as SQL.example.com. Because I need a domain-joined SQL server, I am not using Amazon RDS, but you can decide to use License Mobility, or use an Amazon-provided Amazon Machine Image.

The idea is to create a new database that AD FS can access to read information from a specific table. I store, for each Active Directory user that I want to grant access to AWS, the values for the role attribute. This attribute must be the comma-separated list of the SAML provider and IAM role ARNs, like: arn:aws:iam::YOURACCOUNTID:saml-provider/SAMLPROVIDERNAME,arn:aws:iam::YOURACCOUNTID:role/ROLENAME". You can then customize the parts in red and store the resulting strings in the database so that you can retrieve them when the user authenticates.

The downside of using SQL Server is that you are using a third system that you need to manage for your federation (Active Directory, AD FS, and SQL Server). High availability and fault tolerance for a SQL database are challenges for DBAs. On the other hand, you won’t have to change any claim rules in AD FS in case a new exception needs to be defined for a user. An update in the DB table would only be required because AD FS queries it during the claim rules chain evaluation.

SQL configuration

On the database EC2 instance, I opened the SQL Server Management Studio and connected with a user that has enough privileges to create a new database there. Then I:

Created a new database named ADFS.

Created a new table named AWS with columns UserId and RoleARN. Here is a query example to create this table:

CREATE TABLE AWS
( UserId varchar(100) NOT NULL,
  RoleARN varchar(200) NOT NULL,
  CONSTRAINT AWS_pk PRIMARY KEY (UserId,RoleARN)
); 

Added values to the new table. For example, if I want to give Bob access to the IAM role ADFS-Dev to two other AWS accounts (777788889999 and 444455556666), here are the values to add:

UserId: EXAMPLEBob
RoleARN: arn:aws:iam::777788889999:saml-provider/ADFS,arn:aws:iam::777788889999:role/ADFS-Dev

UserId: EXAMPLEBob
RoleARN: arn:aws:iam::444455556666:saml-provider/ADFS,arn:aws:iam::444455556666:role/ADFS-Dev

Note: The primary key of the table is the UserId and the RoleARN together, so you can define multiple RoleARNs for the same user. Please change the parts in red to your AWS account information.

Make sure the AD FS account has read access to the SQL database and table.

AD FS configuration

To configure a new attribute store, you first need to open the AD FS Microsoft Management Console.

Under Trust Relationships, right-click Attribute Stores, and then click Add Attribute Store.

Type the following values:

Display name: SQL

Attribute store type: SQL

Connection string: Server=SQL;Database=ADFS;Integrated Security=True

Click OK. Right-click the relying party Amazon Web Services, and then click Edit Claim Rules.

This rule—Exception – ARNs from SQL—is again a custom claim rule.

In order to create this rule, follow these steps.

In the Edit Claim Rules for Amazon Web Services dialog box, click Add Rule.

In the Claim rule template list, select Send Claims Using a Custom Rule, and then click Next.

For Claim rule name, type Exception – ARNs from SQL, and then in Custom rule, enter the following:

c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname"]
 => issue(store = "SQL", types = ("https://aws.amazon.com/SAML/Attributes/Role"), query = "SELECT RoleARN from dbo.AWS where UserId= {0}", param = c.Value);

Code explanation

If there is an incoming claim that shows you are authenticated (http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname), you then issue a claim of type https://aws.amazon.com/SAML/Attributes/Role. This is the result of the SQL query on the AWS table in the database AD FS where the column UserId is equal to the Windows account name of the current user. Because RoleARN in the database already contains the comma-separated list of SAML providers and IAM role ARNs, the returned value is already a valid role claim.

The entire workflow is summarized in the following diagram.

EXAMPLEBob goes to the AD FS login page to authenticate.

AD FS authenticates the user against Active Directory.

Active Directory returns the user’s information. EXAMPLEBob belongs to two AD groups (AWS-Production and AWS-Dev), and his user object attribute refers to two AWS accounts (123456789012 and 111122223333).

AD FS queries the SQL server to get possible exceptions defined for EXAMPLEBob.

SQL returns two Role attributes that refer to account 777788889999, IAM role ADFS-Dev, and account 444455556666, IAM role ADFS-Dev.

AD FS dynamically builds four ARNs by using Active Directory group memberships for the IAM roles and user attributes for the AWS account IDs. Additionally, AD FS adds the ARNs for two additional AWS accounts (444455556666 and 777788889999) with a single IAM role (ADFS-Dev) and sends a signed assertion to AWS STS.

EXAMPLEBob gets to the AWS role selection page, where he can choose among accounts 123456789012, 111122223333, 444455556666, and 777788889999. In the first two accounts, he can choose between ADFS-Production and ADFS-Dev. For accounts 444455556666 and 777788889999, only ADFS-Dev is available. All of these are IAM roles created in each specific accounts.

Under the hood—AD FS claim rule explanation (from the GUI)

I will start from the initial AD FS configuration so that you can understand exactly what the provided Windows PowerShell scripts do.

In these steps, I will add the claim rules so that the elements AWS requires and AD FS doesn’t provide by default (NameId, RoleSessionName, and Roles) are added to the SAML authentication response. When you’re ready, open the AD FS Microsoft Management Console (MMC).

Under Trust Relationships, click Relying Party Trusts, right-click the relying party (in this case Amazon Web Services), and then click Edit Claim Rules (see the following screenshot).

Follow the subsequent procedures to create the claim rules for NameId, RoleSessionName, and Roles, which are three mandatory attributes for the SAML response that AD FS will send to AWS STS.

Adding NameId

A name identifier, represented by the NameID element in SAML 2.0, is generally used to identify the subject of a SAML assertion. One reason for including an identifier is to enable the relying party to refer to the subject later, such as in a query or a sign-out request. You will set this attribute of the Windows account name of the user as follows.

In the Edit Claim Rules for Amazon Web Services dialog box, click Add Rule.

Select Transform an Incoming Claim, and then click Next (see the following screenshot).

Use the following settings:

Claim rule name: NameId

Incoming claim type: Windows account name

Outgoing claim type: Name ID

Outgoing name ID format: Persistent Identifier

Pass through all claim values: Select this option

Then click Finish.

Adding a RoleSessionName

You will use the email address of an authenticated user as the RoleSessionName. You can query Active Directory for this attribute as follows.

In the Edit Claim Rules for Amazon Web Services dialog box, click Add Rule.

In the Claim rule template list, select Send LDAP Attributes as Claims (as shown in the following image).

Use the following settings:

Claim rule name: RoleSessionName

Attribute store: Active Directory

LDAP Attribute: E-Mail-Addresses

Outgoing Claim Type: https://aws.amazon.com/SAML/Attributes/RoleSessionName 

Then click Finish.

Adding Roles

Unlike the two previous claims, here I use custom rules to send role attributes. The role must be a comma-separated list of two ARNs: the SAML provider and the IAM role you want to assume. Generate this string by retrieving all the authenticated user’s Active Directory groups and then matching the groups that start with IAM roles of a similar name. I used the names of these groups to create ARNs of IAM roles in the Example Corp. AWS accounts (those that start with AWS-). To know if the user can access one or more of my accounts, I query a user attribute. With few custom claim rules, you can use regular expressions to identify these special Active Directory groups, get the user attribute, and build these ARN strings.

Sending role attributes requires four custom rules. The first rule retrieves all the authenticated user’s Active Directory group memberships; the second rule retrieves the AWS accounts of the user; the third and fourth perform the transformation to the role’s claim.

In the Edit Claim Rules for Amazon Web Services dialog box, click Add Rule.

In the Claim rule template list, select Send Claims Using a Custom Rule, and then click Next.

For Claim rule name, type Get AD Groups, and then in Custom rule, enter the following:

c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname", Issuer == "AD AUTHORITY"]
 => add(store = "Active Directory", types = ("http://temp/variable"), query = ";tokenGroups;{0}", param = c.Value);

Click Finish.

This custom rule uses a script in the claim rule language that retrieves all the groups the authenticated user is a member of and places them into a temporary claim named http://temp/variable (a variable you can access later). I use this in the next rule to transform the groups into IAM role ARNs.

Dynamically generate multi-account role attributes

These AD FS claim rules that you will have at the end of the configuration in the AD FS MMC—NameId, RoleSessionName, and Get AD Groups—are the ones just defined (see the following screenshot). Get AWS Accounts from User attributes, Dynamic ARN – Adding AWS Accounts, and Dynamic ARN – Adding Roles are custom claim rules, and I will explain them in this section.

Get AWS accounts from user attributes

We now define a claim rule to get the AWS accounts a user can access from his Active Directory user object attributes. We will use the Active Directory user attribute url, because this is an attribute defined by default in Active Directory. No Active Directory schema extension is required then. You can use a different user attribute instead, if url is already in use in your organization.

In the Edit Claim Rules for Amazon Web Services dialog box, click Add Rule.

In the Claim rule template list, select Send Claims Using a Custom Rule, and then click Next.

For Claim rule name, type Get AWS Accounts from User attributes, and then in Custom rule, enter the following:

c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname"]
 => add(store = "Active Directory", types = ("http://temp/AWSAccountsFromUser"), query = ";url;{0}", param = c.Value);

Code explanation

Let’s analyze the code in this example:

The “if statement” condition:

c:[Type == "http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname"]

The special operator:

=>

The add statement:

add(store = "Active Directory", types = ("http://temp/AWSAccountsFromUser"), query = ";url;{0}", param = c.Value);

For each rule defined, AD FS checks the input claims, evaluates them against the condition, and applies the statement to the claim if the condition is true. The variable c in the syntax is an incoming claim that you can check conditions against and use values from it in the following statement. In this example, you will check to see if there is an incoming claim that has a type that is http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname.

Then, add a claim. Using the add statement instead of the issue statement will add a claim to the incoming claim set. This will not add the claim to the outgoing token. It is like setting a temporary variable you can use in subsequent rules. In this example, add a claim of type http://temp/AWSAccountsFromUser. Its value is the result of the query = ";url;{0}" on the incoming claim http://schemas.microsoft.com/ws/2008/06/identity/claims/windowsaccountname, which basically means, “Get the url attribute for the incoming user object.”

Note: You can read the AD FS 2.0 Claims Rule Language Primer to learn more about claims rule language.

Dynamic ARN—Adding AWS Accounts

Using a template for the ARN string, first replace the placeholder AWS account IDs with the AWS account IDs the incoming users have been granted access to.

In the Edit Claim Rules for Amazon Web Services dialog box, click Add Rule.

In the Claim rule template list, select Send Claims Using a Custom Rule, and then click Next.

For Claim rule name, type Dynamic ARN – Adding AWS Accounts, and then in Custom rule, enter the following:

c:[Type == "http://temp/AWSAccountsFromUser"]
 => add(Type = "http://temp/AWSAccountsFromUser2", Value = RegExReplace("arn:aws:iam::AWSACCOUNT:saml-provider/ADFS,arn:aws:iam::AWSACCOUNT:role/ADFS-", "AWSACCOUNT", c.Value));

Note: Copy the code as it is and make no changes. In this case, AWSACCOUNT is a placeholder that will be automatically replaced by a real AWS account.

Code explanation

If there is an incoming claim of type “http://temp/AWSAccountsFromUser”, add another claim of type “http://temp/AWSAccountsFromUser2” as the result of the replacement of the string AWSACCOUNT in the string template “arn:aws:iam::AWSACCOUNT:saml-provider/ADFS,arn:aws:iam::AWSACCOUNT:role/ADFS-” with the values contained in the incoming claim. The incoming claim contains all the possible AWS account IDs the user can access. The output looks like the following (you would have your own AWS account IDs in place of the fictitious AWS account IDs):

“arn:aws:iam::123456789012:saml-provider/ADFS,arn:aws:iam::123456789012:role/ADFS-”

“arn:aws:iam:: 111122223333:saml-provider/ADFS,arn:aws:iam:: 111122223333:role/ADFS-”

Note: Because we are adding and not issuing a claim, you won’t actually see any http://temp/AWSAccountsFromUser2 claim in your SAML response.

Dynamic ARN—Adding Roles

I will now replace the IAM role name placeholder based on the Active Directory group membership of the user.

In the Edit Claim Rules for Amazon Web Services dialog box, click Add Rule.

In the Claim rule template list, select Send Claims Using a Custom Rule, and then click Next.

For Claim rule name, type Dynamic ARN – Adding Roles, and then in Custom rule, enter the following:

c1:[Type == "http://temp/AWSAccountsFromUser2"]
 && c2:[Type == "http://temp/variable", Value =~ "(?i)^AWS-"]
 => issue(Type = "https://aws.amazon.com/SAML/Attributes/Role", Value = RegExReplace(c2.Value, "AWS-", c1.Value));

Code explanation

If there is an incoming claim of type “http://temp/AWSAccountsFromUser2” and an incoming claim of type “http://temp/variable” with a value that starts with “AWS-”, issue an outgoing claim of type “https://aws.amazon.com/SAML/Attributes/Role” as the result of the replacement of the string “AWS-” inside the value of the second condition claim (c2) with the value of the first condition claim (c1).

Claim c2 contains the groups that start with “AWS-” that the user belongs to, and claim c1 contains as many strings as the AWS account ID the user has access to in the following form (please note these are fictitious AWS account IDs):

“arn:aws:iam::12345678912:saml-provider/ADFS,arn:aws:iam::123456789012:role/ADFS-”

“arn:aws:iam:: 111122223333:saml-provider/ADFS,arn:aws:iam:: 111122223333:role/ADFS-”

The claim rule replaces the substring “AWS-” in the Active Directory group name and amends the aforementioned strings with ARNs. For example, with the groups mentioned at the beginning of this blog post—AWS-Production and AWS-Dev—the resulting ARNs would be:

“arn:aws:iam::123456789012:saml-provider/ADFS,arn:aws:iam::123456789012:role/ADFS-Production”
“arn:aws:iam::123456789012:saml-provider/ADFS,arn:aws:iam::123456789012:role/ADFS-Dev”

“arn:aws:iam:: 111122223333:saml-provider/ADFS,arn:aws:iam:: 111122223333:role/ADFS-Production”
“arn:aws:iam:: 111122223333:saml-provider/ADFS,arn:aws:iam:: 111122223333:role/ADFS-Dev”

Summing up

Your Active Directory users now can access multiple AWS accounts with their Active Directory credentials. They first log in by using the provided AD FS login page. After each user is authenticated, AD FS is configured to get the following information related to the user from Active Directory:

The user object attribute url, which contains the AWS account the user can access.

The user group membership, which contains the IAM roles the user can access in each account.

Bob can get production access to all the accounts defined in his user attribute url by belonging to the Active Directory group AWS-Production. To grant Bob access to a new AWS account, you can update Bob’s url attribute with the new AWS account, and AD FS will automatically combine this additional account with the Active Directory groups to which Bob belongs.

To change Bob’s access to his AWS accounts, Bob shouldn’t assume the Production role anymore, but the Dev one instead. You simply need to remove Bob from the AWS-Production Active Directory group and have him belong to the AWS-Dev Active Directory group. This change will propagate to all his AWS accounts.

Each AWS account you have in this configuration needs to be configured in the same way. You need to create a SAML provider called ADFS with your AD FS metadata and create IAM roles that trust this provider. The IAM roles must comply with the naming convention you have defined with AD FS (in other words, AWS-Production for the Active Directory group name and ADFS-Production as the related IAM role). To make the configuration easier, I have provided this Windows PowerShell script collection that will help you configure Active Directory, AD FS, and IAM.

When you don’t want a change in either the user attribute or group membership to affect multiple accounts at the same time, you must create an exception. You can define exceptions directly in the claim rule code in AD FS, or in an attribute store as a SQL database. The latter approach introduces a new system to manage and is therefore more complex, but if you need to add an exception for a specific user, you don’t need to change any claim rules in AD FS. AD FS can query the database to retrieve any exception for a specific user.

Note that the AWS Security Blog earlier this year published How to Implement Federated API and CLI Access Using SAML 2.0 and AD FS. By combining that blog post with this one, you can achieve multi-account federated API and CLI access!

I hope you found this post useful. If you have questions, please post them on the IAM forum or in the comments area below.

– Alessandro

What’s New in AWS Key Management Service: AWS CloudFormation Support and Integration with More AWS Services

Post Syndicated from Sreekumar Pisharody original https://blogs.aws.amazon.com/security/post/TxHY6YJA60MTUL/What-s-New-in-AWS-Key-Management-Service-AWS-CloudFormation-Support-and-Integrat

We’re happy to make two announcements about what’s new in AWS Key Management Service (KMS).

First, AWS CloudFormation has added a template for KMS that lets you quickly create KMS customer master keys (CMK) and set their properties. Starting today, you can use the AWS::KMS::Key resource to create a CMK in KMS. To get started, you can use AWS CloudFormation Designer to drag-and-drop a KMS key resource type into your template, as shown in the following image.

To learn more about using KMS with CloudFormation, see the “AWS::KMS::Key” section of the AWS CloudFormation User Guide.

Second, AWS Import/Export Snowball, AWS CloudTrail, Amazon SES, Amazon WorkSpaces, and Amazon Kinesis Firehose now support encryption of data within those services using keys in KMS. As with other KMS-integrated services, you can use CloudTrail to audit the use of your KMS key to encrypt or decrypt your data in SES, Amazon WorkSpaces, CloudTrail, Import/Export Snowball, and Amazon Kinesis Firehose. To see the complete list of AWS services integrated with KMS, see KMS Product Details. For more details about how these services encrypt your data with KMS, see the How AWS Services Use AWS KMS documentation pages.

If you have questions or comments, please add them in the “Comments” section below or on the KMS forum.

– Sree

AWS CloudFormation Security Best Practices

Post Syndicated from George Huang original http://blogs.aws.amazon.com/application-management/post/Tx2UMVHOX7UP4V7/AWS-CloudFormation-Security-Best-Practices

The following is a guest post by Hubert Cheung, Solutions Architect.

AWS CloudFormation makes it easy for developers and systems administrators to create and manage a collection of related AWS resources by provisioning and updating them in an orderly and predictable way. Many of our customers use CloudFormation to control all of the resources in their AWS environments so that they can succinctly capture changes, perform version control, and manage costs in their infrastructure, among other activities.

Customers often ask us how to control permissions for CloudFormation stacks. In this post, we share some of the best security practices for CloudFormation, which include using AWS Identity and Access Management (IAM) policies, CloudFormation-specific IAM conditions, and CloudFormation stack policies. Because most CloudFormation deployments are executed from the AWS command line interface (CLI) and SDK, we focus on using the AWS CLI and SDK to show you how to implement the best practices.

Limiting Access to CloudFormation Stacks with IAM

With IAM, you can securely control access to AWS services and resources by using policies and users or roles. CloudFormation leverages IAM to provide fine-grained access control.

As a best practice, we recommend that you limit service and resource access through IAM policies by applying the principle of least privilege. The simplest way to do this is to limit specific API calls to CloudFormation. For example, you may not want specific IAM users or roles to update or delete CloudFormation stacks. The following sample policy allows all CloudFormation APIs access, but denies UpdateStack and DeleteStack APIs access on your production stack:

{
"Version":"2012-10-17",
"Statement":[{
"Effect":"Allow",
"Action":[
"cloudformation:*"
],
"Resource":"*"
},
{
"Effect":"Deny",
"Action":[
"cloudformation:UpdateStack",
"cloudformation:DeleteStack"
],
"Resource":"arn:aws:cloudformation:us-east-1:123456789012:stack/MyProductionStack/*"
}]
}

We know that IAM policies often need to allow the creation of particular resources, but you may not want them to be created as part of CloudFormation. This is where CloudFormation’s support for IAM conditions comes in.

IAM Conditions for CloudFormation

There are three CloudFormation-specific IAM conditions that you can add to your IAM policies:

cloudformation:TemplateURL

cloudformation:ResourceTypes

cloudformation:StackPolicyURL

With these three conditions, you can ensure that API calls for stack actions, such as create or update, use a specific template or are limited to specific resources, and that your stacks use a stack policy, which prevents stack resources from unintentionally being updated or deleted during stack updates.

Condition: TemplateURL

The first condition, cloudformation:TemplateURL, lets you specify where the CloudFormation template for a stack action, such as create or update, resides and enforce that it be used. In an IAM policy, it would look like this:

{
"Version":"2012-10-17",
"Statement":[{
"Effect": "Deny",
"Action": [
"cloudformation:CreateStack",
“cloudformation:UpdateStack”
],
"Resource": "*",
"Condition": {
"StringNotEquals": {
"cloudformation:TemplateURL": [
"https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template"
]
}
}
},
{
"Effect": "Deny",
"Action": [
"cloudformation:CreateStack",
"cloudformation:UpdateStack"
],
"Resource": "*",
"Condition": {
"Null": {
"cloudformation:TemplateURL": "true"
}
}
}]
}

The first statement ensures that for all CreateStack or UpdateStack API calls, users must use the specified template. The second ensures that all CreateStack or UpdateStack API calls must include the TemplateURL parameter. From the CLI, your calls need to include the –template-url parameter:

aws cloudformation create-stack –stack-name cloudformation-demo –template-url https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template

Condition: ResourceTypes

CloudFormation also allows you to control the types of resources that are created or updated in templates with an IAM policy. The CloudFormation API accepts a ResourceTypes parameter. In your API call, you specify which types of resources can be created or updated. However, to use the new ResourceTypes parameter, you need to modify your IAM policies to enforce the use of this particular parameter by adding in conditions like this:

{
"Version":"2012-10-17",
"Statement":[{
"Effect": "Deny",
"Action": [
"cloudformation:CreateStack",
"cloudformation:UpdateStack"
],
"Resource": "*",
"Condition": {
"ForAllValues:StringLike": {
"cloudformation:ResourceTypes": [
"AWS::IAM::*"
]
}
}
},
{
"Effect": "Deny",
"Action": [
"cloudformation:CreateStack",
"cloudformation:UpdateStack"
],
"Resource": "*",
"Condition": {
"Null": {
"cloudformation:ResourceTypes": "true"
}
}
}]
}

From the CLI, your calls need to include a –resource-types parameter. A call to update your stack will look like this:

aws cloudformation create-stack –stack-name cloudformation-demo –template-url https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template –resource-types=”[AWS::IAM::Group, AWS::IAM::User]”

Depending on the shell, the command might need to be enclosed in quotation marks as follow; otherwise, you’ll get a “No JSON object could be decoded” error:

aws cloudformation create-stack –stack-name cloudformation-demo –template-url https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template –resource-types=’[“AWS::IAM::Group”, “AWS::IAM::User”]’

The ResourceTypes conditions ensure that CloudFormation creates or updates the right resource types and templates with your CLI or API calls. In the first example, our IAM policy would have blocked the API calls because the example included AWS::IAM resources. If our template included only AWS::EC2::Instance resources, the CLI command would look like this and would succeed:

aws cloudformation create-stack –stack-name cloudformation-demo –template-url https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template –resource-types=’[“AWS::EC2::Instance”]’

The third condition is the StackPolicyURL condition. Before we explain how that works, we need to provide some additional context about stack policies.

Stack Policies

Often, the worst disruptions are caused by unintentional changes to resources. To help in mitigating this risk, CloudFormation provides stack policies, which prevent stack resources from unintentionally being updated or deleted during stack updates. When used in conjunction with IAM, stack policies provide a second layer of defense against both unintentional and malicious changes to your stack resources.

The CloudFormation stack policy is a JSON document that defines what can be updated as part of a stack update operation. To set or update the policy, your IAM users or roles must first have the ability to call the cloudformation:SetStackPolicy action.

You apply the stack policy directly to the stack. Note that this is not an IAM policy. By default, setting a stack policy protects all stack resources with a Deny to deny any updates unless you specify an explicit Allow. This means that if you want to restrict only a few resources, you must explicitly allow all updates by including an Allow on the resource "*" and a Deny for specific resources. 

For example, stack policies are often used to protect a production database because it contains data that will go live. Depending on the field that’s changing, there are times when the entire database could be replaced during an update. In the following example, the stack policy explicitly denies attempts to update your production database:

{
"Statement" : [
{
"Effect" : "Deny",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "LogicalResourceId/ProductionDB_logical_ID"
},
{
"Effect" : "Allow",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "*"
}
]
}

You can generalize your stack policy to include all RDS DB instances or any given ResourceType. To achieve this, you use conditions. However, note that because we used a wildcard in our example, the condition must use the "StringLike" condition and not "StringEquals":

{
"Statement" : [
{
"Effect" : "Deny",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "*",
"Condition" : {
"StringLike" : {
"ResourceType" : ["AWS::RDS::DBInstance", "AWS::AutoScaling::*"]
}
}
},
{
"Effect" : "Allow",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "*"
}
]
}

For more information about stack policies, see Prevent Updates to Stack Resources.

Finally, let’s ensure that all of your stacks have an appropriate pre-defined stack policy. To address this, we return to  IAM policies.

Condition:StackPolicyURL

From within your IAM policy, you can ensure that every CloudFormation stack has a stack policy associated with it upon creation with the StackPolicyURL condition:

{
"Version":"2012-10-17",
"Statement":[
{
"Effect": "Deny",
"Action": [
"cloudformation:SetStackPolicy"
],
"Resource": "*",
"Condition": {
"ForAnyValue:StringNotEquals": {
"cloudformation:StackPolicyUrl": [
"https://s3.amazonaws.com/samplebucket/sampleallowpolicy.json"
]
}
}
},
{
"Effect": "Deny",
"Action": [
"cloudformation:CreateStack",
"cloudformation:UpdateStack"
],
"Resource": "*",
"Condition": {
"ForAnyValue:StringNotEquals": {
"cloudformation:StackPolicyUrl": [
“https://s3.amazonaws.com/samplebucket/sampledenypolicy.json”
]
}
}
},
{
"Effect": "Deny",
"Action": [
"cloudformation:CreateStack",
"cloudformation:UpdateStack",
“cloudformation:SetStackPolicy”
],
"Resource": "*",
"Condition": {
"Null": {
"cloudformation:StackPolicyUrl": "true"
}
}
}]
}

This policy ensures that there must be a specific stack policy URL any time SetStackPolicy is called. In this case, the URL is https://s3.amazonaws.com/samplebucket/sampleallowpolicy.json. Similarly, for any create and update stack operation, this policy ensures that the StackPolicyURL is set to the sampledenypolicy.json document in S3 and that a StackPolicyURL is always specified. From the CLI, a create-stack command would look like this:

aws cloudformation create-stack –stack-name cloudformation-demo –parameters ParameterKey=Password,ParameterValue=CloudFormationDemo –capabilities CAPABILITY_IAM –template-url https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template –stack-policy-url https://s3-us-east-1.amazonaws.com/samplebucket/sampledenypolicy.json

Note that if you specify a new stack policy on a stack update, CloudFormation uses the existing stack policy: it uses the new policy only for subsequent updates. For example, if your current policy is set to deny all updates, you must run a SetStackPolicy command to change the stack policy to the one that allows updates. Then you can run an update command against the stack. To update the stack we just created, you can run this:

aws cloudformation set-stack-policy –stack-name cloudformation-demo –stack-policy-url https://s3-us-east-1.amazonaws.com/samplebucket/sampleallowpolicy.json

Then you can run the update:

aws cloudformation update-stack –stack-name cloudformation-demo –parameters ParameterKey=Password,ParameterValue=NewPassword –capabilities CAPABILITY_IAM –template-url https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template –stack-policy-url https://s3-us-west-2.amazonaws.com/awshubfiles/sampledenypolicy.json

The IAM policy that we used ensures that a specific stack policy is applied to the stack any time a stack is updated or created.

Conclusion

CloudFormation provides a repeatable way to create and manage related AWS resources. By using a combination of IAM policies, users, and roles, CloudFormation-specific IAM conditions, and stack policies, you can ensure that your CloudFormation stacks are used as intended and minimize accidental resource updates or deletions.

You can learn more about this topic and other CloudFormation best practices in the recording of our re:Invent 2015 session, (DVO304) AWS CloudFormation Best Practices, and in our documentation.

AWS CloudFormation Security Best Practices

Post Syndicated from George Huang original http://blogs.aws.amazon.com/application-management/post/Tx2UMVHOX7UP4V7/AWS-CloudFormation-Security-Best-Practices

The following is a guest post by Hubert Cheung, Solutions Architect.

AWS CloudFormation makes it easy for developers and systems administrators to create and manage a collection of related AWS resources by provisioning and updating them in an orderly and predictable way. Many of our customers use CloudFormation to control all of the resources in their AWS environments so that they can succinctly capture changes, perform version control, and manage costs in their infrastructure, among other activities.

Customers often ask us how to control permissions for CloudFormation stacks. In this post, we share some of the best security practices for CloudFormation, which include using AWS Identity and Access Management (IAM) policies, CloudFormation-specific IAM conditions, and CloudFormation stack policies. Because most CloudFormation deployments are executed from the AWS command line interface (CLI) and SDK, we focus on using the AWS CLI and SDK to show you how to implement the best practices.

Limiting Access to CloudFormation Stacks with IAM

With IAM, you can securely control access to AWS services and resources by using policies and users or roles. CloudFormation leverages IAM to provide fine-grained access control.

As a best practice, we recommend that you limit service and resource access through IAM policies by applying the principle of least privilege. The simplest way to do this is to limit specific API calls to CloudFormation. For example, you may not want specific IAM users or roles to update or delete CloudFormation stacks. The following sample policy allows all CloudFormation APIs access, but denies UpdateStack and DeleteStack APIs access on your production stack:

{
"Version":"2012-10-17",
"Statement":[{
"Effect":"Allow",
"Action":[
"cloudformation:*"
],
"Resource":"*"
},
{
"Effect":"Deny",
"Action":[
"cloudformation:UpdateStack",
"cloudformation:DeleteStack"
],
"Resource":"arn:aws:cloudformation:us-east-1:123456789012:stack/MyProductionStack/*"
}]
}

We know that IAM policies often need to allow the creation of particular resources, but you may not want them to be created as part of CloudFormation. This is where CloudFormation’s support for IAM conditions comes in.

IAM Conditions for CloudFormation

There are three CloudFormation-specific IAM conditions that you can add to your IAM policies:

cloudformation:TemplateURL

cloudformation:ResourceTypes

cloudformation:StackPolicyURL

With these three conditions, you can ensure that API calls for stack actions, such as create or update, use a specific template or are limited to specific resources, and that your stacks use a stack policy, which prevents stack resources from unintentionally being updated or deleted during stack updates.

Condition: TemplateURL

The first condition, cloudformation:TemplateURL, lets you specify where the CloudFormation template for a stack action, such as create or update, resides and enforce that it be used. In an IAM policy, it would look like this:

{
"Version":"2012-10-17",
"Statement":[{
"Effect": "Deny",
"Action": [
"cloudformation:CreateStack",
“cloudformation:UpdateStack”
],
"Resource": "*",
"Condition": {
"StringNotEquals": {
"cloudformation:TemplateURL": [
"https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template"
]
}
}
},
{
"Effect": "Deny",
"Action": [
"cloudformation:CreateStack",
"cloudformation:UpdateStack"
],
"Resource": "*",
"Condition": {
"Null": {
"cloudformation:TemplateURL": "true"
}
}
}]
}

The first statement ensures that for all CreateStack or UpdateStack API calls, users must use the specified template. The second ensures that all CreateStack or UpdateStack API calls must include the TemplateURL parameter. From the CLI, your calls need to include the –template-url parameter:

aws cloudformation create-stack –stack-name cloudformation-demo –template-url https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template

Condition: ResourceTypes

CloudFormation also allows you to control the types of resources that are created or updated in templates with an IAM policy. The CloudFormation API accepts a ResourceTypes parameter. In your API call, you specify which types of resources can be created or updated. However, to use the new ResourceTypes parameter, you need to modify your IAM policies to enforce the use of this particular parameter by adding in conditions like this:

{
"Version":"2012-10-17",
"Statement":[{
"Effect": "Deny",
"Action": [
"cloudformation:CreateStack",
"cloudformation:UpdateStack"
],
"Resource": "*",
"Condition": {
"ForAllValues:StringLike": {
"cloudformation:ResourceTypes": [
"AWS::IAM::*"
]
}
}
},
{
"Effect": "Deny",
"Action": [
"cloudformation:CreateStack",
"cloudformation:UpdateStack"
],
"Resource": "*",
"Condition": {
"Null": {
"cloudformation:ResourceTypes": "true"
}
}
}]
}

From the CLI, your calls need to include a –resource-types parameter. A call to update your stack will look like this:

aws cloudformation create-stack –stack-name cloudformation-demo –template-url https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template –resource-types=”[AWS::IAM::Group, AWS::IAM::User]”

Depending on the shell, the command might need to be enclosed in quotation marks as follow; otherwise, you’ll get a “No JSON object could be decoded” error:

aws cloudformation create-stack –stack-name cloudformation-demo –template-url https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template –resource-types=’[“AWS::IAM::Group”, “AWS::IAM::User”]’

The ResourceTypes conditions ensure that CloudFormation creates or updates the right resource types and templates with your CLI or API calls. In the first example, our IAM policy would have blocked the API calls because the example included AWS::IAM resources. If our template included only AWS::EC2::Instance resources, the CLI command would look like this and would succeed:

aws cloudformation create-stack –stack-name cloudformation-demo –template-url https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template –resource-types=’[“AWS::EC2::Instance”]’

The third condition is the StackPolicyURL condition. Before we explain how that works, we need to provide some additional context about stack policies.

Stack Policies

Often, the worst disruptions are caused by unintentional changes to resources. To help in mitigating this risk, CloudFormation provides stack policies, which prevent stack resources from unintentionally being updated or deleted during stack updates. When used in conjunction with IAM, stack policies provide a second layer of defense against both unintentional and malicious changes to your stack resources.

The CloudFormation stack policy is a JSON document that defines what can be updated as part of a stack update operation. To set or update the policy, your IAM users or roles must first have the ability to call the cloudformation:SetStackPolicy action.

You apply the stack policy directly to the stack. Note that this is not an IAM policy. By default, setting a stack policy protects all stack resources with a Deny to deny any updates unless you specify an explicit Allow. This means that if you want to restrict only a few resources, you must explicitly allow all updates by including an Allow on the resource "*" and a Deny for specific resources. 

For example, stack policies are often used to protect a production database because it contains data that will go live. Depending on the field that’s changing, there are times when the entire database could be replaced during an update. In the following example, the stack policy explicitly denies attempts to update your production database:

{
"Statement" : [
{
"Effect" : "Deny",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "LogicalResourceId/ProductionDB_logical_ID"
},
{
"Effect" : "Allow",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "*"
}
]
}

You can generalize your stack policy to include all RDS DB instances or any given ResourceType. To achieve this, you use conditions. However, note that because we used a wildcard in our example, the condition must use the "StringLike" condition and not "StringEquals":

{
"Statement" : [
{
"Effect" : "Deny",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "*",
"Condition" : {
"StringLike" : {
"ResourceType" : ["AWS::RDS::DBInstance", "AWS::AutoScaling::*"]
}
}
},
{
"Effect" : "Allow",
"Action" : "Update:*",
"Principal": "*",
"Resource" : "*"
}
]
}

For more information about stack policies, see Prevent Updates to Stack Resources.

Finally, let’s ensure that all of your stacks have an appropriate pre-defined stack policy. To address this, we return to  IAM policies.

Condition:StackPolicyURL

From within your IAM policy, you can ensure that every CloudFormation stack has a stack policy associated with it upon creation with the StackPolicyURL condition:

{
"Version":"2012-10-17",
"Statement":[
{
"Effect": "Deny",
"Action": [
"cloudformation:SetStackPolicy"
],
"Resource": "*",
"Condition": {
"ForAnyValue:StringNotEquals": {
"cloudformation:StackPolicyUrl": [
"https://s3.amazonaws.com/samplebucket/sampleallowpolicy.json"
]
}
}
},
{
"Effect": "Deny",
"Action": [
"cloudformation:CreateStack",
"cloudformation:UpdateStack"
],
"Resource": "*",
"Condition": {
"ForAnyValue:StringNotEquals": {
"cloudformation:StackPolicyUrl": [
“https://s3.amazonaws.com/samplebucket/sampledenypolicy.json”
]
}
}
},
{
"Effect": "Deny",
"Action": [
"cloudformation:CreateStack",
"cloudformation:UpdateStack",
“cloudformation:SetStackPolicy”
],
"Resource": "*",
"Condition": {
"Null": {
"cloudformation:StackPolicyUrl": "true"
}
}
}]
}

This policy ensures that there must be a specific stack policy URL any time SetStackPolicy is called. In this case, the URL is https://s3.amazonaws.com/samplebucket/sampleallowpolicy.json. Similarly, for any create and update stack operation, this policy ensures that the StackPolicyURL is set to the sampledenypolicy.json document in S3 and that a StackPolicyURL is always specified. From the CLI, a create-stack command would look like this:

aws cloudformation create-stack –stack-name cloudformation-demo –parameters ParameterKey=Password,ParameterValue=CloudFormationDemo –capabilities CAPABILITY_IAM –template-url https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template –stack-policy-url https://s3-us-east-1.amazonaws.com/samplebucket/sampledenypolicy.json

Note that if you specify a new stack policy on a stack update, CloudFormation uses the existing stack policy: it uses the new policy only for subsequent updates. For example, if your current policy is set to deny all updates, you must run a SetStackPolicy command to change the stack policy to the one that allows updates. Then you can run an update command against the stack. To update the stack we just created, you can run this:

aws cloudformation set-stack-policy –stack-name cloudformation-demo –stack-policy-url https://s3-us-east-1.amazonaws.com/samplebucket/sampleallowpolicy.json

Then you can run the update:

aws cloudformation update-stack –stack-name cloudformation-demo –parameters ParameterKey=Password,ParameterValue=NewPassword –capabilities CAPABILITY_IAM –template-url https://s3.amazonaws.com/cloudformation-templates-us-east-1/IAM_Users_Groups_and_Policies.template –stack-policy-url https://s3-us-west-2.amazonaws.com/awshubfiles/sampledenypolicy.json

The IAM policy that we used ensures that a specific stack policy is applied to the stack any time a stack is updated or created.

Conclusion

CloudFormation provides a repeatable way to create and manage related AWS resources. By using a combination of IAM policies, users, and roles, CloudFormation-specific IAM conditions, and stack policies, you can ensure that your CloudFormation stacks are used as intended and minimize accidental resource updates or deletions.

You can learn more about this topic and other CloudFormation best practices in the recording of our re:Invent 2015 session, (DVO304) AWS CloudFormation Best Practices, and in our documentation.

Construct Your Own Launch Stack URL

Post Syndicated from Elliot Yamaguchi original http://blogs.aws.amazon.com/application-management/post/Tx2YSVJV4VMPBHI/Construct-Your-Own-Launch-Stack-URL

launch stack button When you use AWS CloudFormation templates to deliver your solutions, your users can get up and running with just a few clicks or commands. Users can launch an AWS CloudFormation stack with one of your templates, and AWS CloudFormation automatically provisions the specified resources and bootstraps the software running on them.

Although users have several ways to create stacks from your templates, you can provide a quick and direct option for your users by constructing a launch stack URL.  A launch stack URL takes users directly to the Create Stack wizard in the AWS CloudFormation console with the stack name, the template URL, and default parameters already filled in. Launch stack URLs are useful to bloggers, internal teams of a company, trainers, independent software vendors, or anyone who wants to provide a quick up and running solution to their users.

Before you construct a launch stack URL, save your template in an Amazon S3 bucket and grant open and download permissions to users who should have access to your template. From the Amazon S3 console, you also need to retrieve the URL of the template file.

file properties

To construct the launch stack URL, use the following general URL syntax:

https://console.aws.amazon.com/cloudformation/home?region=region#/stacks/new?stackName=stack_name&templateURL=template_location

The region parameter specifies where the stack will be created. If you don’t specify a region, users are directed to the region they last used. The stack_name parameter is a unique name that identifies the stack. The template_location  parameter is the URL of the template file.

As an example, the following launch stack URL launches a stack named myteststack in the us-east-1 region with the mytemplate.template template:

https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=myteststack&templateURL=https://s3.amazonaws.com/samplecfntemplates/mytemplate.template

Anyone who follows that URL is directed to the Create Stack wizard in the AWS CloudFormation console, with the specified stack name and template location, as shown in the following screenshot. Users can still specify parameters or other stack options. If they are not already authenticated, users are directed to the AWS Management Console sign-in page and then redirected to the AWS CloudFormation Create Stack wizard.

create stack wizard

To make your launch stack URL more visible, try our Launch Stack image (launch stack button), which is available at https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png.

At AWS, we use launch stack URLs with our sample templates to enable our customers to quickly launch the templates.

For more information about creating templates, see Working with Templates in the AWS CloudFormation User Guide.

Construct Your Own Launch Stack URL

Post Syndicated from Elliot Yamaguchi original http://blogs.aws.amazon.com/application-management/post/Tx2YSVJV4VMPBHI/Construct-Your-Own-Launch-Stack-URL

launch stack button When you use AWS CloudFormation templates to deliver your solutions, your users can get up and running with just a few clicks or commands. Users can launch an AWS CloudFormation stack with one of your templates, and AWS CloudFormation automatically provisions the specified resources and bootstraps the software running on them.

Although users have several ways to create stacks from your templates, you can provide a quick and direct option for your users by constructing a launch stack URL.  A launch stack URL takes users directly to the Create Stack wizard in the AWS CloudFormation console with the stack name, the template URL, and default parameters already filled in. Launch stack URLs are useful to bloggers, internal teams of a company, trainers, independent software vendors, or anyone who wants to provide a quick up and running solution to their users.

Before you construct a launch stack URL, save your template in an Amazon S3 bucket and grant open and download permissions to users who should have access to your template. From the Amazon S3 console, you also need to retrieve the URL of the template file.

file properties

To construct the launch stack URL, use the following general URL syntax:

https://console.aws.amazon.com/cloudformation/home?region=region#/stacks/new?stackName=stack_name&templateURL=template_location

The region parameter specifies where the stack will be created. If you don’t specify a region, users are directed to the region they last used. The stack_name parameter is a unique name that identifies the stack. The template_location  parameter is the URL of the template file.

As an example, the following launch stack URL launches a stack named myteststack in the us-east-1 region with the mytemplate.template template:

https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/new?stackName=myteststack&templateURL=https://s3.amazonaws.com/samplecfntemplates/mytemplate.template

Anyone who follows that URL is directed to the Create Stack wizard in the AWS CloudFormation console, with the specified stack name and template location, as shown in the following screenshot. Users can still specify parameters or other stack options. If they are not already authenticated, users are directed to the AWS Management Console sign-in page and then redirected to the AWS CloudFormation Create Stack wizard.

create stack wizard

To make your launch stack URL more visible, try our Launch Stack image (launch stack button), which is available at https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png.

At AWS, we use launch stack URLs with our sample templates to enable our customers to quickly launch the templates.

For more information about creating templates, see Working with Templates in the AWS CloudFormation User Guide.

AWS CloudFormation at AWS re:Invent 2015: Breakout Session Recap, Videos, and Slides

Post Syndicated from George Huang original http://blogs.aws.amazon.com/application-management/post/Tx1ZYD0M87D4NW0/AWS-CloudFormation-at-AWS-re-Invent-2015-Breakout-Session-Recap-Videos-and-Slide

The AWS CloudFormation team and others presented and shared many updates and best practices during several 2015 AWS re:Invent sessions in October. We wanted to take the opportunity to show you where our presentation slides and videos are located as well as highlight a few product updates and best practices that we shared at this year’s re:Invent.

DVO304 – AWS CloudFormation Best Practices: slides and video

ARC307 – Infrastructure as Code: slides and video

DVO303 – Scaling Infrastructure Operations with AWS: slides and video

ARC401 – Cloud First: New Architecture for New Infrastructure: slides and video

DVO310 – Benefit from DevOps When Moving to AWS for Windows: slides and video

DVO401 – Deep Dive into Blue/Green Deployments on AWS: slides and video

SEC312 – Reliable Design and Deployment of Security and Compliance: slides and video

AWS CloudFormation Designer

We introduced CloudFormation Designer in early October. During our re:Invent session DVO304 (AWS CloudFormation Best Practices), we introduced CloudFormation Designer and then did a live demo and walkthrough of its key features and use cases.

AWS CloudFormation Designer is a new visual tool that allows you to visually edit your CloudFormation templates as a diagram. It provides a drag-and-drop interface for adding resources to templates, and CloudFormation Designer automatically modifies the underlying JSON when you add or remove resources. You can also use the integrated text editor to view or specify template details, such as resource property values and input parameters.

To learn more about this feature:

Watch the CloudFormation Designer portion of our re:Invent talk to see a demo

View slides 3-13 to learn more about CloudFormation Designer from our re:Invent talk

Updated resource support in CloudFormation

In the same session, we also talked about the five new resources that CloudFormation can provision which we introduced in October. To stay up to-date on CloudFormation resource support updates, please visit here to see a list of all currently supported AWS resources.

Other topics covered in our “AWS CloudFormation Best Practices” breakout session

Using Cost Explorer to budget and estimate a stack’s cost

Collecting audit logs using the CloudTrail integration with CloudFormation

CloudFormation advanced language features

How to extend CloudFormation to resources that are not yet supported by CloudFormation

Security and user-access best practices

Best practices for writing CloudFormation templates when sharing templates with teams or users that have different environments or are using different AWS regions

Please reach us at the AWS CloudFormation forum if you have more feedback or questions. 

AWS CloudFormation at AWS re:Invent 2015: Breakout Session Recap, Videos, and Slides

Post Syndicated from George Huang original http://blogs.aws.amazon.com/application-management/post/Tx1ZYD0M87D4NW0/AWS-CloudFormation-at-AWS-re-Invent-2015-Breakout-Session-Recap-Videos-and-Slide

The AWS CloudFormation team and others presented and shared many updates and best practices during several 2015 AWS re:Invent sessions in October. We wanted to take the opportunity to show you where our presentation slides and videos are located as well as highlight a few product updates and best practices that we shared at this year’s re:Invent.

DVO304 – AWS CloudFormation Best Practices: slides and video

ARC307 – Infrastructure as Code: slides and video

DVO303 – Scaling Infrastructure Operations with AWS: slides and video

ARC401 – Cloud First: New Architecture for New Infrastructure: slides and video

DVO310 – Benefit from DevOps When Moving to AWS for Windows: slides and video

DVO401 – Deep Dive into Blue/Green Deployments on AWS: slides and video

SEC312 – Reliable Design and Deployment of Security and Compliance: slides and video

AWS CloudFormation Designer

We introduced CloudFormation Designer in early October. During our re:Invent session DVO304 (AWS CloudFormation Best Practices), we introduced CloudFormation Designer and then did a live demo and walkthrough of its key features and use cases.

AWS CloudFormation Designer is a new visual tool that allows you to visually edit your CloudFormation templates as a diagram. It provides a drag-and-drop interface for adding resources to templates, and CloudFormation Designer automatically modifies the underlying JSON when you add or remove resources. You can also use the integrated text editor to view or specify template details, such as resource property values and input parameters.

To learn more about this feature:

Watch the CloudFormation Designer portion of our re:Invent talk to see a demo

View slides 3-13 to learn more about CloudFormation Designer from our re:Invent talk

Updated resource support in CloudFormation

In the same session, we also talked about the five new resources that CloudFormation can provision which we introduced in October. To stay up to-date on CloudFormation resource support updates, please visit here to see a list of all currently supported AWS resources.

Other topics covered in our “AWS CloudFormation Best Practices” breakout session

Using Cost Explorer to budget and estimate a stack’s cost

Collecting audit logs using the CloudTrail integration with CloudFormation

CloudFormation advanced language features

How to extend CloudFormation to resources that are not yet supported by CloudFormation

Security and user-access best practices

Best practices for writing CloudFormation templates when sharing templates with teams or users that have different environments or are using different AWS regions

Please reach us at the AWS CloudFormation forum if you have more feedback or questions. 

Persist Streaming Data to Amazon S3 using Amazon Kinesis Firehose and AWS Lambda

Post Syndicated from Derek Graeber original https://blogs.aws.amazon.com/bigdata/post/Tx2MUQB5PRWU36K/Persist-Streaming-Data-to-Amazon-S3-using-Amazon-Kinesis-Firehose-and-AWS-Lambda

Derek Graeber is a Senior Consultant in Big Data Analytics for AWS Professional Services

Streaming data analytics is becoming main-stream (pun intended) in large enterprises as the technology stacks have become more user-friendly to implement. For example, Spark-Streaming connected to an Amazon Kinesis stream is a typical model for real-time analytics. 

But one area that cannot and should not be overlooked is the need to persist streaming data (unchanged) in a reliable and durable fashion – and to do it with ease.  This blog post walks you through a simple and effective way to persist data to Amazon S3 from Amazon Kinesis Streams using AWS Lambda and Amazon Kinesis Firehose, a new managed service from AWS.

Here’s a real use case:  Hearst Publishing is a global media company behind well-known brands such as Cosmopolitan, Elle, Esquire, Seventeen, and Car and Driver, as well as television and cable entities such as A&E Networks and Esquire Network.

Hearst has embarked on the big data journey and needs to collect pertinent data from over 200+ digital sites in real time.  This data gives invaluable insight into the usage of their sites and indicates the most relevant trending topics based on content.  Using these data points, both historical and in real-time, Hearst could monitor and become much more agile in managing the content available to site users by giving key analytical data to content owners.

Hearst chose to use a well-respected cast of characters for an ETL process of streaming data: Streams, Spark on Amazon EMR, and S3. They also realized the need to store the unchanged data right from Streams in parallel to EMR-Spark.  In line with the important big data ethic “never throw data away”, all data pulled from Streams was persisted to S3 for historical reasons and so it can be re-processed either by a different consuming team or re-analyzed with a modified processing scheme in Spark. The Amazon Kinesis Client Library (KCL) and Amazon Kinesis Connector codebase provided a consistent and highly configurable way to get data from Streams to S3:

The KCL has built-in check-pointing for Streams (whether it be TRIM-HORIZON or LATEST).

The KCL integrates very easily with the Amazon Kinesis connectors.

The Connectors framework provided a way to transform, buffer, filter, and emit the Amazon Kinesis records to S3 with ease (among other specified AWS services).We can buffer data and write to S3 based on thresholds with number of records, time since last flush, or actual data buffer size limits.

These features make the KCL–Connector (KCL-C) very powerful and useful; it’s a very popular implementation. The KCL-C setup runs on an EC2 instance or fleet of instances and is easily managed with AWS CloudFormation and Auto Scaling.  The KCL has become the proven way to manage getting data off Streams.  The figure below shows a sample architecture with KCL.

Sample architecture with KCL.

Hearst, evaluating their AWS ecosystem, wanted to move as much as possible to AWS-provided services.  With a lean development team and a focus on data science, there was an interest in not having to monitor EC2 instances.  Thus the question was raised “How can we keep the reliability of KCL-C for our data intact but not have to keep tabs on the EC2 instance?  Can’t AWS provide a service to do this so we can focus on data science?” 

In short, a perfect use case for Firehose and Lambda unfolded.  Looking at the needs of the process, reliability was critical along with the ability to buffer (aggregate) data into larger file sizes and persist to S3. The figure below illustrates a sample architecture with Firehose.

Sample architecture with Firehose.

For this post Java is the codebase, but this can also be done in JavaScript.  The code is available in its entirety on the AWS Big Data Blog repository on GitHub.  Assume all services are set up in the same region.  For more information, see the Amazon Kinesis Firehose Getting Started Guide.

Set up the S3 and Streams services

You need to set up a stream (representing the raw data coming in) and an S3 bucket where the data should reside.  For more information, see Step 1: Create a Stream and Create a Bucket

Review the Lambda function with Firehose

This is where the fun happens.  Take a look at the code:  If you pulled the GitHub repository, this is located in the Java class com.amazonaws.proserv.lambda.KinesisToFirehose.

public class KinesisToFirehose {
private String firehoseEndpointURL = "https://firehose.us-east-1.amazonaws.com";
private String deliveryStreamName = "blogfirehose";
private String deliveryStreamRoleARN = "arn:aws:iam::<AWS Acct Id>:role/firehose_blog_role";
private String targetBucketARN = "arn:aws:s3:::dgraeberaws-blogs";
private String targetPrefix = "blogoutput/";
private int intervalInSec = 60;
private int buffSizeInMB = 2;

private AmazonKinesisFirehoseClient firehoseClient = new AmazonKinesisFirehoseClient();
private LambdaLogger logger;

public void kinesisHandler(KinesisEvent event, Context context){
logger = context.getLogger();
setup();
for(KinesisEvent.KinesisEventRecord rec : event.getRecords()) {
logger.log("Got message ");
String msg = new String(rec.getKinesis().getData().array())+"n";
Record deliveryStreamRecord = new Record().withData (ByteBuffer.wrap(msg.getBytes()));

PutRecordRequest putRecordRequest = new PutRecordRequest()
.withDeliveryStreamName(deliveryStreamName)
.withRecord(deliveryStreamRecord);

logger.log("Putting message");
firehoseClient.putRecord(putRecordRequest);
logger.log("Successful Put");
}
}

}

The following private instance variables should be configured with your particular naming conventions:

firehoseEndpointURL – The AWS endpoint where the Firehose delivery stream is hosted.  Typically, you keep the Lambda function and delivery stream in the same region.

deliveryStreamName – The actual name of the Firehose delivery stream that you are using.

deliveryStreamRoleARN – The AWS ARN of the role which you want the Firehose delivery stream to use when writing to S3.  You will create this role via the console later in this post.

targetBucketARN – The AWS ARN of the bucket to which you want Firehose to write.

targetPrefix – When writing to the S3 bucket and segmenting the object key with a prefix, add the segment in this variable. (At the time of this post, if you want a ‘/’ separator, you need to add it in this variable, for example, ‘somesegment/’.)

intervalInSec –  A buffer for time lapse. Firehose pushes to S3 if this threshold has been met after the last write.

bufferSizeInMB – A buffer for aggregated payload size. Firehose pushes to S3 if this threshold has been met after the last write.

This Lambda function is configured to create the Firehose delivery stream if it does not already exist. In this post, you create the delivery stream manually from the console, being careful to have the proper private instance variable (above) set in the Lambda function to reflect the Firehose delivery stream thus created.

Create the Firehose delivery stream

Now, you can create the Firehose delivery stream using the console.  For more information, see Amazon Kinesis Firehose Getting Started Guide.

Create the Firehose delivery system

In this post, I assume that you do not have a role created that gives you Firehose delivery stream access, so you can create one now.  In the list for IAM role*, choose Create new Firehose delivery IAM role.

Create new Firehose delivery IAM role

For reference, the policy associated with the role is similar to the one below:

Permission policy (Firehose role)

{
"Version": "2012-10-17",
"Statement":
[
{
"Sid": "StmtDemo1",
"Effect": "Allow",
"Action":
["s3:AbortMultipartUpload","s3:GetBucketLocation","s3:GetObject",
"s3:ListBucket","s3:ListBucketMultipartUploads","s3:PutObject"
],
"Resource":
["arn:aws:s3:::*"]
},
{
"Sid": "StmtDemo2",
"Effect": "Allow",
"Action": ["kms:Decrypt","kms:Encrypt"],
"Resource": ["*"]
}
]
}

Trust policy (Firehose role)

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "StmtDemo3",
"Effect": "Allow",
"Principal": {"Service": "firehose.amazonaws.com"},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalId": “YOURACCTID">
}
}
}
]
}

Finish configuring the Firehose delivery stream with the indicated configuration. For this post, set limits of 2 MB and 60 seconds for the buffer size and buffer interval, respectively.

Finishing configuring the Firehose stream

NOTE: For this post, you will not be compressing or encrypting the data when writing to S3 from Firehose.  Your actual implementation may vary.

Confirm and create the Firehose delivery stream

To summarize the configuration, you are:

Defining a name for the Firehose delivery stream

Defining the targeted S3 bucket for output

Adding an S3 prefix to the bucket

Defining the buffer thresholds – in this case, they are 60 seconds and 2 MB (whichever comes first)

Not compressing or encrypting the output data

Create the Lambda JAR distribution

Verify that your instance variables match between your Lambda function and your newly created Firehose delivery stream.  Create the JAR file that Lambda will need.  Because this is a Java project with Maven, execute the mvn clean package task from your project root directory.  Lambda runs Java 8, so you need to compile against the Java 8 JDK.

Create the Lambda function

Now that you have the Lambda code ready to run, create the function itself.  You can do this via CLI or console.  For more information, see Getting Started: Authoring AWS Lambda Code in Java.

When you create the Lambda role that has a Lambda trust relationship, make sure that the policy has access to both Firehose and Streams. Here is an example:

Permissions policy (Lambda role)

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": ["logs:*"],
"Resource": "arn:aws:logs:*:*:*"
},
{
"Effect": "Allow",
"Action": ["kinesis:*","firehose:*"],
"Resource": ["arn:aws:kinesis:*:*:*","arn:aws:firehose:*:*:*"]
}
]
}

Trust policy (Lambda role)

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "StmtDemo4",
"Effect": "Allow",
"Principal": {"Service": "lambda.amazonaws.com"},
"Action": "sts:AssumeRole"
}
]
}

I won’t cover creating a Lambda function in depth in this post, but here are the highlights:

Use the newly-created .jar file:The HANDLER value should be com.amazonaws.proserv.lambda.KinesisToFirehose::kinesisHandler

Use the role you just created with the policy access to Firehose and Streams and the Lambda trust relationship (directly above).

Use the defaults for Memory and Timeout.

After the upload, the Lambda function is in place; all you need to do is set the listener.

On the Event Sources tab under your new Lambda function, add an event source that is the Amazon Kinesis stream you created earlier.  Select a Streams input, add your stream name, and leave the defaults. You are now connected.

Populate streams and verify results

The only thing left to do is add data to the stream and watch the S3 bucket fill up. In the Java project from Git, a helper class pumps dummy messages to Streams (com.amazonaws.proserv.PopulateKinesisData).  If you are running it from your local repository, add your access key information to the resources/AwsCredentials.properties file.  If you are running it from EC2, make sure the role on the instance has Streams permissions. 

After you start adding messages to the stream and the thresholds are hit (2 MB at 60 seconds), you will see your targeted S3 bucket begin to populate with the prefix that you designated and the files written with an object key designating the year, month, day, and hour in which the output file from Firehose was written (prefix/yyyy/mm/dd/hr/*).

Conclusion

In this post, I have shown you how to create a reliable way to persist data from Streams to Amazon S3 using the new managed service Firehose.  Firehose removes the need to manage compute servers and builds on some of the most-used tenets of streaming data persistence:

Aggregated data based on thresholds.

Persist data to a durable repository (in this case, S3).

The Hearst Publishing use case provided a way to reliably persist data from Streams to S3 with an aggregated output that modeled their current scheme – all with a native AWS service. As the data source was Streams, the Firehose service could run in parallel to the existing real-time data processing scheme with no impact.

If you have questions or suggestions, please  leave a comment below.

—————

Related:

How Expedia Implemented Near Real-time Analysis of Interdependent Datasets