Tag Archives: AWS CloudFormation

How to Remove Single Points of Failure by Using a High-Availability Partition Group in Your AWS CloudHSM Environment

Post Syndicated from Tracy Pierce original https://blogs.aws.amazon.com/security/post/Tx7VU4QS5RCK7Q/How-to-Remove-Single-Points-of-Failure-by-Using-a-High-Availability-Partition-Gr

A hardware security module (HSM) is a hardware device designed with the security of your data and cryptographic key material in mind. It is tamper-resistant hardware that prevents unauthorized users from attempting to pry open the device, plug any extra devices in to access data or keys such as subtokens, or damage the outside housing. If any such interference occurs, the device wipes all information stored so that unauthorized parties do not gain access to your data or cryptographic key material. A high-availability (HA) setup could be beneficial because, with multiple HSMs kept in different data centers and all data synced between them, the loss of one HSM does not mean the loss of your data.

In this post, I will walk you through steps to remove single points of failure in your AWS CloudHSM environment by setting up an HA partition group. Single points of failure occur when a single CloudHSM device fails in a non-HA configuration, which can result in the permanent loss of keys and data. The HA partition group, however, allows for one or more CloudHSM devices to fail, while still keeping your environment operational.

Prerequisites

You will need a few things to build your HA partition group with CloudHSM:

  • 2 CloudHSM devices. AWS offers a free two-week trial. AWS will provision the trial for you and send you the CloudHSM information such as the Elastic Network Interface (ENI) and the private IP address assigned to the CloudHSM device so that you may begin testing. If you have used CloudHSM before, another trial cannot be provisioned, but you can set up production CloudHSM devices on your own. See Provisioning Your HSMs.
  • A client instance from which to access your CloudHSM devices. You can create this manually, or via an AWS CloudFormation template. You can connect to this instance in your public subnet, and then it can communicate with the CloudHSM devices in your private subnets.
  • An HA partition group, which ensures the syncing and load balancing of all CloudHSM devices you have created.

The CloudHSM setup process takes about 30 minutes from beginning to end. By the end of this how-to post, you should be able to set up multiple CloudHSM devices and an HA partition group in AWS with ease. Keep in mind that each production CloudHSM device you provision comes with an up-front fee of $5,000. You are not charged for any CloudHSM devices provisioned for a trial, unless you decide to move them to production when the trial ends.

If you decide to move your provisioned devices to your production environment, you will be billed $5,000 per device. If you decide to stop the trial so as not to be charged, you have up to 24 hours after the trial ends to let AWS Support know of your decision.

Solution overview

How HA works

HA is a feature of the Luna SA 7000 HSM hardware device AWS uses for its CloudHSM service. (Luna SA 7000 HSM is also known as the “SafeNet Network HSM” in more recent SafeNet documentation. Because AWS documentation refers to this hardware as “Luna SA 7000 HSM,” I will use this same product name in this post.) This feature allows more than one CloudHSM device to be placed as members in a load-balanced group setup. By having more than one device on which all cryptographic material is stored, you remove any single points of failure in your environment.

You access your CloudHSM devices in this HA partition group through one logical endpoint, which distributes traffic to the CloudHSM devices that are members of this group in a load-balanced fashion. Even though traffic is balanced between the HA partition group members, any new data or changes in data that occur on any CloudHSM device will be mirrored for continuity to the other members of the HA partition group. A single HA partition group is logically represented by a slot, which is physically composed of multiple partitions distributed across all HA nodes. Traffic is sent through the HA partition group, and then distributed to the partitions that are linked. All partitions are then synced so that data is persistent on each one identically.

The following diagram illustrates the HA partition group functionality.

  1. Application servers send traffic to your HA partition group endpoint.
  2. The HA partition group takes all requests and distributes them evenly between the CloudHSM devices that are members of the HA partition group.
  3. Each CloudHSM device mirrors itself to each other member of the HA partition group to ensure data integrity and availability.

Automatic recovery

If you ever lose data, you want a hands-off, quick recovery. Before autoRecovery was introduced, you could take advantage of the redundancy and performance HA partition groups offer, but you were still required to manually intervene when a group member was lost.

HA partition group members may fail for a number of reasons, including:

  • Loss of power to a CloudHSM device.
  • Loss of network connectivity to a CloudHSM device. If network connectivity is lost, it will be seen as a failed device and recovery attempts will be made.

Recovery of partition group members will only work if the following are true:

  • HA autoRecovery is enabled.
  • There are at least two nodes (CloudHSM devices) in the HA partition group.
  • Connectivity is established at startup.
  • The recover retry limit is not reached (if reached or exceeded, the only option is manual recovery).

HA autoRecovery is not enabled by default and must be explicitly enabled by running the following command, which is found in Enabling Automatic Recovery.

>vtl haAdmin –autoRecovery –retry <count>

When enabling autoRecovery, set the –retry and –interval parameters. The –retry parameter can be a value between 0 and 500 (or -1 for infinite retries), and equals the number of times the CloudHSM device will attempt automatic recovery. The –interval parameter is in seconds and can be any value between 60 and 1200. This is the amount of time between automatic recovery tries that the CloudHSM will attempt.

Setting up two Production CloudHSM devices in AWS

Now that I have discussed how HA partition groups work and why they are useful, I will show how to set up your CloudHSM environment and the HA partition group itself. To create an HA partition group environment, you need a minimum of two CloudHSM devices. You can have as many as 16 CloudHSM devices associated with an HA partition group at any given time. These must be associated with the same account and region, but can be spread across multiple Availability Zones, which is the ideal setup for an HA partition group. Automatic recovery is great for larger HA partition groups because it allows the devices to quickly attempt recovery and resync data in the event of a failure, without requiring manual intervention.

Set up the CloudHSM environment

To set up the CloudHSM environment, you must have a few things already in place:

  • An Amazon VPC.
  • At least one public subnet and two private subnets.
  • An Amazon EC2 client instance (m1.small running Amazon Linux x86 64-bit) in the public subnet, with the SafeNet client software already installed. This instance uses the key pair that you specified during creation of the CloudFormation stack. You can find a ready-to-use Amazon Machine Image (AMI) in our Community AMIs. Simply log into the EC2 console, choose Launch Instance, click Community AMIs, and search for CloudHSM. Because we regularly release new AMIs with software updates, searching for CloudHSM will show all available AMIs for a region. Select the AMI with the most recent client version.
  • Two security groups, one for the client instance and one for the CloudHSM devices. The security group for the client instance, which resides in the public subnet, will allow SSH on port 22 from your local network. The security group for the CloudHSM devices, which resides in the private subnet, will allow SSH on port 22 and NTLS on port 1792 from your public subnet. These will both be ingress rules (egress rules allow all traffic).
  • An Elastic IP address for the client instance.
  • An IAM role that delegates AWS resource access to CloudHSM. You can create this role in the IAM console:

    1. Click Roles and then click Create New Role.
    2. Type a name for the role and then click Next Step.
    3. Under AWS Service Roles, click Select next to AWS CloudHSM.
    4. In the Attach Policy step, select AWSCloudHSMRole as the policy. Click Next Step.
    5. Click Create Role.

We have a CloudFormation template available that will set up the CloudHSM environment for you:

  1. Go to the CloudFormation console.
  2. Choose Create Stack. Specify https://cloudhsm.s3.amazonaws.com/cloudhsm-quickstart.json as the Amazon S3 template URL.
  3. On the next two pages, specify parameters such as the Stack name, SSH Key Pair, Tags, and SNS Topic for alerts. You will find SNS Topic under the Advanced arrow. Then, click Create.

When the new stack is in the CREATION_COMPLETE state, you will have the IAM role to be used for provisioning your CloudHSM devices, the private and public subnets, your client instance with Elastic IP (EIP), and the security groups for both the CloudHSM devices and the client instance. The CloudHSM security group will already have its necessary rules in place to permit SSH and NTLS access from your public subnet; however, you still must add the rules to the client instance’s security group to permit SSH access from your allowed IPs. To do this:

  1. In the VPC console, make sure you select the same region as the region in which your HSM VPC resides.
  2. Select the security group in your HSM VPC that will be used for the client instance.
  3. Add an inbound rule that allows TCP traffic on port 22 (SSH) from your local network IP addresses.
  4. On the Inbound tab, from the Create a new rule list, select SSH, and enter the IP address range of the local network from which you will connect to your client instance.
  5. Click Add Rule, and then click Apply Rule Changes.

After adding the IP rules for SSH (port 22) to your client instance’s security group, test the connection by attempting to make a SSH connection locally to your client instance EIP. Make sure to write down all the subnet and role information, because you will need this later.

Create an SSH key pair

The SSH key pair that you will now create will be used by CloudHSM devices to authenticate the manager account when connecting from your client instance. The manager account is simply the user that is permitted to SSH to your CloudHSM devices. Before provisioning the CloudHSM devices, you create the SSH key pair so that you can provide the public key to the CloudHSM during setup. The private key remains on your client instance to complete the authentication process. You can generate the key pair on any computer, as long as you ensure the client instance has the private key copied to it. You can also create the key pair on Linux or Windows. I go over both processes in this section of this post.

In Linux, you will use the ssh-keygen command. By typing just this command into the terminal window, you will receive output similar to the following.

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/user/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/user/.ssh/id_rsa.
Your public key has been saved in /home/user/.ssh/id_rsa.pub.
The key fingerprint is: df:c4:49:e9:fe:8e:7b:eb:28:d5:1f:72:82:fb:f2:69
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|             .   |
|            o    |
|           + .   |
|        S   *.   |
|         . =.o.o |
|          ..+ +..|
|          .o Eo .|
|           .OO=. |
+-----------------+
$

In Windows, use PuTTYgen to create your key pair:

  1. Start PuTTYgen. For Type of key to generate, select SSH-2 RSA.
  2. In the Number of bits in a generated key field, specify 2048.
  3. Click Generate.
  4. Move your mouse pointer around in the blank area of the Key section below the progress bar (to generate some randomness) until the progress bar is full.
  5. A private/public key pair has now been generated.
  6. In the Key comment field, type a name for the key pair that you will remember.
  7. Click Save public key and name your file.
  8. Click Save private key and name your file. It is imperative that you do not lose this key, so make sure to store it somewhere safe.
  9. Right-click the text field labeled Public key for pasting into OpenSSH authorized_keys file and choose Select All.
  10. Right-click again in the same text field and choose Copy.

The following screenshot shows what the PuTTYgen output will look like after you have created the key pair.

You must convert the keys created by PuTTYgen to OpenSSH format for use with other clients by using the following command.

ssh-keygen –i –f puttygen_key > openssh_key

The public key will be used to provision the CloudHSM device and the private key will be stored on the client instance to authenticate the SSH sessions. The public SSH key will look something like the following. If it does not, it is not in the correct format and must be converted using the preceding procedure.

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA6bUsFjDSFcPC/BZbIAv8cAR5syJMB GiEqzFOIEHbm0fPkkQ0U6KppzuXvVlc2u7w0mg
PMhnkEfV6j0YBITu0Rs8rNHZFJs CYXpdoPxMMgmCf/FaOiKrb7+1xk21q2VwZyj13GPUsCxQhRW7dNidaaYTf14sbd9A qMUH4UOUjs
27MhO37q8/WjV3wVWpFqexm3f4HPyMLAAEeExT7UziHyoMLJBHDKMN7 1Ok2kV24wwn+t9P/Va/6OR6LyCmyCrFyiNbbCDtQ9JvCj5
RVBla5q4uEkFRl0t6m9 XZg+qT67sDDoystq3XEfNUmDYDL4kq1xPM66KFk3OS5qeIN2kcSnQ==

Whether you are saving the private key on your local computer or moving it to the client instance, you must ensure that the file permissions are correct. You can do this by running the following commands (throughout this post, be sure to replace placeholder content with your own values). The first command sets the necessary permissions; the second command adds the private key to the authentication agent.

$ chmod 600 ~/.ssh/<private_key_file>
$ ssh-add ~/.ssh/<private_key_file>

Set up the AWS CLI tools

Now that you have your SSH key pair ready, you can set up the AWS CLI tools so that you may provision and manage your CloudHSM devices. If you used the CloudFormation template or the CloudHSM AMI to set up your client instance, you already have the CLI installed. You can check this by running at the command prompt: CloudHSM version. The resulting output should be “Version”: “3.0.5”. If you chose to use your own AMI and install the Luna SA software, you can install the CloudHSM CLI Tools with the following steps. The current version in use is 3.0.5.

$ wget https://s3.amazon.com/cloudhsm-software/CloudHsmCLI.egg
$ sudo easy_install-2.7 –s /usr/local/bin CloudHsmCLI.egg
$ cloudhsm version
{
      “Version”: “<version>”
}

You must also set up file and directory ownership for your user on the client instance and the Chrystoki.conf file. The Chrystoki.conf file is the configuration file for the CloudHSM device. By default, CloudHSM devices come ready from the factory for immediate housing of cryptographic keys and performing cryptographic processes on data, but must be configured to connect to your client instances:

  1. On the client instance, set the owner and write permission on the Chrystoki.conf file.
$ sudo chown <owner> /etc/Chrystoki.conf
$ sudo chmod +w /etc/Chrystoki.conf

The <owner> can be either the user or a group the user belongs to (for example, ec2-user).

  1. On the client instance set the owner of the Luna client directory:
$ sudo chown <owner> -R <luna_client_dir>

The <owner> should be the same as the <owner> of the Chrystoki.conf file. The <luna_client_dir> differs based on the version of the LunaSA client software installed. If these are new setups, use version 5.3 or newer; however, if you have older clients with version 5.1 installed, use version 5.1:

  • Client software version 5.3: /usr/safenet/lunaclient/
  • Client software version 5.1: /usr/lunasa/

You also must configure the AWS CLI tools with AWS credentials to use for the API calls. These can be set by config files, passing the credentials in the commands, or by instance profile. The most secure option, which eliminates the need to hard-code credentials in a config file, is to use an instance profile on your client instance. All CLI commands in this post are performed on a client instance launched with a IAM role that has CloudHSM permissions. If you want to set your credentials in a config file instead, you must remember that each CLI command should include –profile <profilename>, with <profilename> being the name you assigned in the config file for these credentials.See Configuring the AWS CloudHSM CLI Tools for help with setting up the AWS CLI tools.

You will then set up a persistent SSH tunnel for all CloudHSM devices to communicate with the client instance. This is done by editing the ~/.ssh/config file. Replace <CloudHSM_ip_address> with the private IP of your CloudHSM device, and replace <private_key_file> with the file location of your SSH private key created earlier (for example, /home/user/.ssh/id_rsa).

Host <CloudHSM_ip_address>
User manager
IdentityFile <private_key_file>

Also necessary for the client instance to authenticate with the CloudHSM partitions or partition group are client certificates. Depending on the LunaSA client software you are using, the location of these files can differ. Again, if these are new setups, use version 5.3 or newer; however, if you have older clients with version 5.1 installed, use version 5.1:

  • Linux clients

    • Client software version 5.3: /usr/safenet/lunaclient/cert
    • Client software version 5.1: /usr/lunasa/cert
  • Windows clients

    • Client software version 5.3: %ProgramFiles%\SafeNet\LunaClient\cert
    • Client software version 5.1: %ProgramFiles%\LunaSA\cert

To create the client certificates, you can use the OpenSSL Toolkit or the LunaSA client-side vtl commands. The OpenSSL Toolkit is a program that allows you to manage TLS (Transport Layer Security) and SSL (Secure Sockets Layer) protocols. It is commonly used to create SSL certificates for secure communication between internal network devices. The LunaSA client-side vtl commands are installed on your client instance along with the Luna software. If you used either CloudFormation or the CloudHSM AMI, the vtl commands are already installed for you. If you chose to launch a different AMI, you can download the Luna software. After you download the software, run the command linux/64/install.sh as root on a Linux instance and install the Luna SA option. If you install the softtware on a Windows instance, run the command windows\64\LunaClient.msi to install the Luna SA option. I show certificate creation in both OpenSSL Toolkit and LunaSA in the following section.

OpenSSL Toolkit

      $ openssl genrsa –out <luna_client_cert_dir>/<client_name>Key.pem 2048
      $ openssl req –new –x509 –days 3650 –key <luna_client_cert_dir>/<client_cert>Key.pem –out <client_name>.pem

The <luna_client_cert_dir> is the LunaSA Client certificate directory on the client and the <client_name> can be whatever you choose.

LunaSA

      $ sudo vtl createCert –n <client_name>

The output of the preceding LunaSA command will be similar to the following.

Private Key created and written to:

<luna_client_cert_dir>/<client_name>Key.pem

Certificate created and written to:

<luna_client_cert_dir>/<client_name>.pem

You will need these key file locations later on, so make sure you write them down or save them to a file. One last thing to do at this point is create the client Amazon Resource Name (ARN), which you do by running the following command.

$ CloudHSM create-client –certificate-file <luna_client_cert_dir>/<client_name>.pem

{
      “ClientARN”: “<client_arn>”,
      “RequestId”: “<request_id>”
}

Also write down in a safe location the client ARN because you will need it when registering your client instances to the HA partition group.

Provision your CloudHSM devices

Now for the fun and expensive part. Always remember that for each CloudHSM device you provision to your production environment, there is an upfront fee of $5,000. Because you need more than one CloudHSM device to set up an HA partition group, provisioning two CloudHSM devices to production will cost an upfront fee of $10,000.

If this is your first time trying out CloudHSM, you can have a two-week trial provisioned for you at no cost. The only cost will occur if you decide to keep your CloudHSM devices and move them into production. If you are unsure of the usage in your company, I highly suggest doing a trial first. You can open a support case requesting a trial at any time. You must have a paid support plan to request a CloudHSM trial.

To provision the two CloudHSM devices, SSH into your client instance and run the following CLI command.

$ CloudHSM create-CloudHSM \
--subnet-id <subnet_id> \
--ssh-public-key-file <public_key_file> \
--iam-role-arn <iam_role_arn> \
--syslog-ip <syslog_ip_address>

The response should resemble the following.

{
      “CloudHSMArn”: “<CloudHSM_arn>”,
      “RequestId”: “<request_id>”
}

Make note of each CloudHSM ARN because you will need them to initialize the CloudHSM devices and later add them to the HA partition group.

Initialize the CloudHSM devices

Configuring your CloudHSM devices, or initializing them as the process is formally called, is what allows you to set up the configuration files, certificate files, and passwords on the CloudHSM itself. Because you already have your CloudHSM ARNs from the previous section, you can run the describe-hsm command to get the EniId and the EniIp of the CloudHSM devices. Your results should be similar to the following.

$ CloudHSM describe-CloudHSM --CloudHSM-arn <CloudHSM_arn>
{    
     "EniId": "<eni_id>",   
     "EniIp": "<eni_ip>",    
     "CloudHSMArn": "<CloudHSM_arn>",    
     "IamRoleArn": "<iam_role_arn>",    
     "Partitions": [],    
     "RequestId": "<request_id>",    
     "SerialNumber": "<serial_number>",    
     "SoftwareVersion": "5.1.3-1",    
     "SshPublicKey": "<public_key_text>",    
     "Status": "<status>",    
     "SubnetId": "<subnet_id>",    
     "SubscriptionStartDate": "2014-02-05T22:59:38.294Z",    
     "SubscriptionType": "PRODUCTION",    
     "VendorName": "SafeNet Inc."
}

Now that you know the EniId of each CloudHSM, you need to apply the CloudHSM security group to them. This ensures that connection can occur from any instance with the client security group assigned. When a trial is provisioned for you, or you provision CloudHSM devices yourself, the default security group of the VPC is automatically assigned to the ENI.You must change this to the security group that permits ingress ports 22 and 1792 from your client instance.

To apply a CloudHSM security group to an EniId:

  1. Go to the EC2 console, and choose Network Interfaces in the left pane.
  2. Select the EniId of the CloudHSM.
  3. From the Actions drop-down list, choose Change Security Groups. Choose the security group for your CloudHSM device, and then click Save.

To complete the initialization process, you must ensure a persistent SSH connection is in place from your client to the CloudHSM. Remember the ~/.ssh/config file you edited earlier? Now that you have the IP address of the CloudHSM devices and the location of the private SSH key file, go back and fill in that config file’s parameters by using your favorite text editor.

Now, initialize using the initialize-hsm command with the information you gathered from the provisioning steps. The values in red in the following example are meant as placeholders for your own naming and password conventions, and should be replaced with your information during the initialization of the CloudHSM devices.

$CloudHSM initialize-CloudHSM \
--CloudHSM-arn <CloudHSM_arn> \
--label <label> \
--cloning-domain <cloning_domain> \
--so-password <so_password>

The <label> is a unique name for the CloudHSM device that should be easy to remember. You can also use this name as a descriptive label that tells what the CloudHSM device is for. The <cloning_domain> is a secret used to control cloning of key material from one CloudHSM to another. This can be any unique name that fits your company’s naming conventions. Examples could be exampleproduction or exampledevelopment. If you are going to set up an HA partition group environment, the <cloning_domain> must be the same across all CloudHSMs. The <so_password> is the security officer password for the CloudHSM device, and for ease of remembrance, it should be the same across all devices as well. It is important you use passwords and cloning domain names that you will remember, because they are unrecoverable and the loss of them means loss of all data on a CloudHSM device. For your use, we do supply a Password Worksheet if you want to write down your passwords and store the printed page in a secure place.

Configure the client instance

Configuring the client instance is important because it is the secure link between you, your applications, and the CloudHSM devices. The client instance opens a secure channel to the CloudHSM devices and sends all requests over this channel so that the CloudHSM device can perform the cryptographic operations and key storage. Because you already have launched the client instance and mostly configured it, the only step left is to create the Network Trust Link (NTL) between the client instance and the CloudHSM. For this, we will use the LunaSA vtl commands again.

  1. Copy the server certificate from the CloudHSM to the client.
$ scp –i ~/.ssh/<private_key_file> [email protected]<CloudHSM_ip_address>:server.pem
  1. Register the CloudHSM certificate with the client.
$ sudo vtl addServer –n <CloudHSM_ip_address> -c server.pem
New server <CloudHSM_ip_address> successfully added to server list.
  1. Copy the client certificate to the CloudHSM.
$ scp –i ~/.ssh/<private_key_file> <client_cert_directory>/<client_name>.pem [email protected]<CloudHSM_ip_address>:
  1. Connect to the CloudHSM.
$ ssh –i ~/.ssh/<private_key_file> [email protected]<CloudHSM_ip_address>
lunash:>
  1. Register the client.
lunash:> client register –client <client_id> -hostname <client_name>

The <client_id> and <client_name> should be the same for ease of use, and this should be the same as the name you used when you created your client certificate.

  1. On the CloudHSM, log in with the SO password.
lunash:> hsm login
  1. Create a partition on each CloudHSM (use the same name for ease of remembrance).
lunash:> partition create –partition <name> -password <partition_password> -domain <cloning_domain>

The <partition_password> does not have to be the same as the SO password, and for security purposes, it should be different.

  1. Assign the client to the partition.
lunash:> client assignPartition –client <client_id> -partition <partition_name>
  1. Verify that the partition assigning went correctly.
lunash:> client show –client <client_id>
  1. Log in to the client and verify it has been properly configured.
$ vtl verify
The following Luna SA Slots/Partitions were found:
Slot    Serial #         Label
====    =========        ============
1      <serial_num1>     <partition_name>
2      <serial_num2>     <partition_name>

You should see an entry for each partition created on each CloudHSM device. This step lets you know that the CloudHSM devices and client instance were properly configured.

The partitions created and assigned via the previous steps are for testing purposes only and will not be used in the HA parition group setup. The HA partition group workflow will automatically create a partition on each CloudHSM device for its purposes. At this point, you have created the client and at least two CloudHSM devices. You also have set up and tested for validity the connection between the client instance and the CloudHSM devices. The next step in to ensure fault tolerance by setting up the HA partition group.

Set up the HA partition group for fault tolerance

Now that you have provisioned multiple CloudHSM devices in your account, you will add them to an HA partition group. As I explained earlier in this post, an HA partition group is a virtual partition that represents a group of partitions distributed over many physical CloudHSM devices for HA. Automatic recovery is also a key factor in ensuring HA and data integrity across your HA partition group members. If you followed the previous procedures in this post, setting up the HA partition group should be relatively straightforward.

Create the HA partition group

First, you will create the actual HA partition group itself. Using the CloudHSM CLI on your client instance, run the following command to create the HA partition group and name it per your company’s naming conventions. In the following command, replace <label> with the name you chose.

$ CloudHSM create-hapg –group-label <label>

Register the CloudHSM devices with the HA partition group

Now, add the already initialized CloudHSM devices to the HA partition group. You will need to run the following command for each CloudHSM device you want to add to the HA partition group.

$ CloudHSM add-CloudHSM-to-hapg \
--CloudHSM-arn <CloudHSM_arn> \
--hapg-arn <hapg_arn> \
--cloning-domain <cloning_domain> \
--partition-password <partition_password> \
--so-password <so_password>

You should see output similar to the following after each successful addition to the HA partition group.

{
      “Status”: “Addition of CloudHSM <CloudHSM_arn> to HA partition group <hapg_arn> successful”
}

Register the client with the HA partition group

The last step is to register the client with the HA partition group. You will need the client ARN from earlier in the post, and you will use the CloudHSM CLI command register-client-to-hapg to complete this process.

$ CloudHSM register-client-to-hapg \
--client-arn <client_arn> \
--hapg-arn <hapg_arn>
{
      “Status”: “Registration of the client <client_arn> to the HA partition group <hapg_arn> successful”
}

After you register the client with the HA partition group, you have the client configuration file and the server certificates. You have already registered the client with the HA partition group, but now you have to actually assign it as well, which you do by using the get-client-configuration AWS CLI command.

$ CloudHSM get-client-configuration \
--client-arn <client_arn> \
--hapg-arn <hapg_arn> \
--cert-directory <server_cert_location> \
--config-directory /etc/

The configuration file has been copied to /etc/
The server certificate has been copied to <server_cert_location>

The <server_cert_location> will differ depending on the LunaSA client software you are using:

  • Client software version 5.3: /usr/safenet/lunaclient/cert/server
  • Client software version 5.1: /usr/lunasa/cert/server

Lastly, to verify the client configuration, run the following LunaSA vtl command.

$ vtl haAdmin show

In the output, you will see a heading, HA Group and Member Information. Ensure that the number of group members equals the number of CloudHSM devices you added to the HA partition group. If the number does not match what you have provisioned, you might have missed a step in the provisioning process. Going back through the provisioning process usually repairs this. However, if you still encounter issues, opening a support case is the quickest way to get assistance.

Another way to verify the HA partition group setup is to check the /etc/Chrystoki.conf file for output similar to the following.

VirtualToken = {
   VirtualToken00Label = hapg1;
   VirtualToken00SN = 1529127380;
   VirtualToken00Members = 475256026,511541022;
}
HASynchronize = {
   hapg1 = 1;
}
HAConfiguration = {
   reconnAtt = -1;
   AutoReconnectInterval = 60;
   HAOnly = 1;

Summary

You have now completed the process of provisioning CloudHSM devices, the client instance for connection, and your HA partition group for fault tolerance. You can begin using an application of your choice to access the CloudHSM devices for key management and encryption. By accessing CloudHSM devices via the HA partition group, you ensure that all traffic is load balanced between all backing CloudHSM devices. The HA partition group will ensure that each CloudHSM has identical information so that it can respond to any request issued.

Now that you have an HA partition group set up with automatic recovery, if a CloudHSM device fails, the device will attempt to recover itself, and all traffic will be rerouted to the remaining CloudHSM devices in the HA partition group so as not to interrupt traffic. After recovery (manual or automatic), all data will be replicated across the CloudHSM devices in the HA partition group to ensure consistency.

If you have questions about any part of this blog post, please post them on the IAM forum.

– Tracy

Introducing the AWS for DevOps Getting Started Guide

Post Syndicated from Paul Cornell original https://aws.amazon.com/blogs/devops/introducing-the-aws-for-devops-getting-started-guide/

We are pleased to announce the AWS for DevOps Getting Started Guide is now available. As a companion to our DevOps and AWS website, this new resource teaches you, in a hands-on way, how to use services like AWS CodeCommit, AWS CodeDeploy, and AWS CodePipeline for continuous integration and continuous delivery.

Specifically, you will learn how to:

  1. Use AWS CloudFormation to give users access to required AWS services, resources, and actions.
  2. Create a source code repository in AWS CodeCommit and then use AWS CloudFormation to launch an Amazon EC2 instance that connects to the repository.
  3. Download the source code you will deploy and then push it into the repository.
  4. Use AWS CloudFormation to create the deployment target (an Amazon EC2 instance) and AWS resources that are compatible with AWS CodeDeploy, AWS Elastic Beanstalk, or AWS OpsWorks.
  5. Use AWS CloudFormation to create and run a pipeline in AWS CodePipeline to automate continuous delivery of the repository’s source code to the deployment target.
  6. Verify the deployment’s results on the deployment target.
  7. Make a change to the source code and then push it into the repository, triggering an automatic redeployment to the deployment target.
  8. Verify the deployed change on the deployment target.
  9. Use AWS CloudFormation to clean up the resources you created for this walkthrough.

You do not need to know anything about AWS to try this walkthrough. If you don’t have an AWS account, we’ll show you how to create one. And, if you’re new to AWS, our AWS Free Tier lets you to experiment with few, if any, charges to your AWS account.

Perhaps your organization is considering a move to DevOps practices. Or maybe your organization is practicing DevOps now. In either case, if you’re not yet using AWS services, this resource can help you down this path. Please let us know what you think by using the Feedback button on any of the walkthrough’s pages.

Get started now!

Introducing the AWS for DevOps Getting Started Guide

Post Syndicated from Paul Cornell original http://blogs.aws.amazon.com/application-management/post/Tx2WN97Z6P1D7U9/Introducing-the-AWS-for-DevOps-Getting-Started-Guide

We are pleased to announce the AWS for DevOps Getting Started Guide is now available. As a companion to our DevOps and AWS website, this new resource teaches you, in a hands-on way, how to use services like AWS CodeCommit, AWS CodeDeploy, and AWS CodePipeline for continuous integration and continuous delivery.

Specifically, you will learn how to:

  1. Use AWS CloudFormation to give users access to required AWS services, resources, and actions.
  2. Create a source code repository in AWS CodeCommit and then use AWS CloudFormation to launch an Amazon EC2 instance that connects to the repository.
  3. Download the source code you will deploy and then push it into the repository.
  4. Use AWS CloudFormation to create the deployment target (an Amazon EC2 instance) and AWS resources that are compatible with AWS CodeDeploy, AWS Elastic Beanstalk, or AWS OpsWorks.
  5. Use AWS CloudFormation to create and run a pipeline in AWS CodePipeline to automate continuous delivery of the repository’s source code to the deployment target.
  6. Verify the deployment’s results on the deployment target.
  7. Make a change to the source code and then push it into the repository, triggering an automatic redeployment to the deployment target.
  8. Verify the deployed change on the deployment target.
  9. Use AWS CloudFormation to clean up the resources you created for this walkthrough.

You do not need to know anything about AWS to try this walkthrough. If you don’t have an AWS account, we’ll show you how to create one. And, if you’re new to AWS, our AWS Free Tier lets you to experiment with few, if any, charges to your AWS account.

Perhaps your organization is considering a move to DevOps practices. Or maybe your organization is practicing DevOps now. In either case, if you’re not yet using AWS services, this resource can help you down this path. Please let us know what you think by using the Feedback button on any of the walkthrough’s pages.

Get started now!

New AWS Training Bootcamps to Help You Build Technical Skills

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-aws-training-bootcamps-to-help-you-build-technical-skills/

My colleague Janna Pellegrino shared the guest post below to introduce you to a new set of AWS training bootcamps!


Jeff;


We’ve made four of our most popular Technical Bootcamps from AWS re:Invent and Summits part of our broader AWS Training portfolio so you can now attend a class convenient to you.

  • Taking AWS Operations to the Next Level teaches you how to leverage AWS CloudFormation, Chef, and AWS SDKs to automate provisioning and configuration of AWS infrastructure resources and applications. We also cover how to work with AWS Service Catalog. This course is designed for solutions architects and SysOps administrators.
  • Securing Next-Gen Applications at Cloud Scale teaches you how to use a DevSecOps approach to design and build robust security controls at cloud scale for next-generation workloads. We cover design considerations of operating high-assurance workloads on the AWS platform. Labs teach you governance, configuration management, trust-decision automation, audit artifact generation, and native integration of tasks into custom software workloads. This course is for security engineers, developers, solutions architects, and other technical security practitioners.
  • Running Container-Enabled Microservices on AWS teaches you how to manage and scale container-enabled applications by using Amazon ECS. Labs teach you to use Amazon ECS to handle long-running services, build and deploy container images, link services together, and scale capacity to meet demand. This course is for developers, solutions architects, and system administrators.
  • Building a Recommendation Engine on AWS teaches you to build a real-time analytics and geospatial search application using Amazon Elasticsearch Service, Amazon DynamoDB, DynamoDB Streams, Amazon API Gateway, AWS Lambda, and Amazon S3. We discuss a real-world location-aware social application that displays information generated from a model created with Amazon Machine Learning. We also cover best practices for processing and analyzing data, such as the lambda data processing pattern and automating development process, using Swagger, Grunt, and the AWS SDK. This course is for developers, solutions architects, and data scientists.

Janna Pellegrino, AWS Training and Certification

Register for and Attend This July 29 Webinar—Best Practices for Managing Security Operations in AWS

Post Syndicated from Craig Liebendorfer original https://blogs.aws.amazon.com/security/post/Tx2NM3C88RXUQIH/Register-for-and-Attend-This-July-29-Webinar-Best-Practices-for-Managing-Securit

As part of the AWS Webinar Series, AWS will present Best Practices for Managing Security Operations in AWS on Friday, July 29. This webinar will start at 10:30 A.M. and end at 11:30 A.M. Pacific Time.

AWS Security Solutions Architect Henrik Johansson will show you different ways you can use AWS Identity and Access Management (IAM) to control access to your AWS services and integrate your existing authentication system with AWS IAM.

You also will learn:

  • How to deploy and control your AWS infrastructure using code templates, including change management policies with AWS CloudFormation.
  • How to audit and log your AWS service usage.
  • How to use AWS services to add automatic compliance checks to your AWS infrastructure.
  • About the AWS Shared Responsibility Model.

The webinar is free, but space is limited and registration is required. Register today.

– Craig

How to Use AWS CloudFormation to Automate Your AWS WAF Configuration with Example Rules and Match Conditions

Post Syndicated from Ben Potter original https://blogs.aws.amazon.com/security/post/Tx3NYSJHO8RK22S/How-to-Use-AWS-CloudFormation-to-Automate-Your-AWS-WAF-Configuration-with-Exampl

AWS WAF is a web application firewall that integrates closely with Amazon CloudFront (AWS’s content delivery network [CDN]). AWS WAF gives you control to allow or block traffic to your web applications by, for example, creating custom rules that block common attack patterns.

We recently announced AWS CloudFormation support for all current features of AWS WAF. This enables you to leverage CloudFormation templates to configure, customize, and test AWS WAF settings across all your web applications. Using CloudFormation templates can help you reduce the time required to configure AWS WAF. In this blog post, I will show you how to use CloudFormation to automate your AWS WAF configuration with example rules and match conditions.

AWS WAF overview

If you are not familiar with AWS WAF configurations, let me try to catch you up quickly. AWS WAF consists of three main components: a web access control list (web ACL), rules, and filters (also known as a match set). A web ACL is associated with a given CloudFront distribution. Each web ACL is a collection of one or more rules, and each rule can have one or more match conditionswhich are composed of one or more filters. The filters inspect components of the request (such as its headers or URI) to match for certain match conditions.

Solution overview

The solution in this blog post uses AWS CloudFormation in an automated fashion to provision, update, and optionally delete the components that form the AWS WAF solution. The CloudFormation template will deploy the following rules and conditions as part of this solution:

  • A manual IP rule that contains an empty IP match set that must be updated manually with IP addresses to be blocked.
  • An auto IP rule that contains an empty IP match condition for optionally implementing an automated AWS Lambda function, such as is shown in How to Import IP Address Reputation Lists to Automatically Update AWS WAF IP Blacklists and How to Use AWS WAF to Block IP Addresses That Generate Bad Requests.
  • A SQL injection rule and condition to match SQL injection-like patterns in URI, query string, and body.
  • A cross-site scripting rule and condition to match Xss-like patterns in URI and query string.
  • A size-constraint rule and condition to match requests with URI or query string >= 8192 bytes which may assist in mitigating against buffer overflow type attacks.
  • ByteHeader rules and conditions (split into two sets) to match user agents that include spiders for non–English-speaking countries that are commonly blocked in a robots.txt file, such as sogou, baidu, and etaospider, and tools that you might choose to monitor use of, such as wget and cURL. Note that the WordPress user agent is included because it is used commonly by compromised systems in reflective attacks against non–WordPress sites.
  • ByteUri rules and conditions (split into two sets) to match request strings containing install, update.php, wp-config.php, and internal functions including $password, $user_id, and $session.
  • A whitelist IP condition (empty) is included and added as an exception to the ByteURIRule2 rule as an example of how to block unwanted user agents, unless they match a list of known good IP addresses.

All example rules configured by the template as part of the solution will count requests that match the rules for you to test with your web application. This template makes use of CloudFormation to provide a modular, manageable method of creating and updating nested stacks. A nested stack aligns with CloudFormation best practices to separate common components for reuse and ensure you do not reach the template body size limit, which is currently 51,200 bytes. All rules and conditions in this CloudFormation template are referenced with a resource reference number internal to the stack for each resource (for example, ByteBodyCondition1), so you can easily duplicate and extend each component. As with any example CloudFormation template, you can edit and reuse the template to suit your needs.

The following architecture diagram shows the overview of this solution, which consists of a single web ACL and multiple rules with match conditions:

Descriptions of key details in the preceding diagram are as follows:

  1. Requests are resolved by DNS to CloudFront, configured with Web ACL to filter all requests.
  2. AWS WAF Web ACL evaluates each request with configured rules containing conditions.
  3. If a request matches a block condition, the request results in returning an HTTP 403 error (forbidden) to the client computer. If a request matches a count rule, the requests are served. 
  4. The origin configured in CloudFront serves allowed or counted requests.

Deploying the solution

Prerequisites

The following deployment steps assume that you already have a CloudFront distribution that you use to deliver content for your web applications. If you do not already have a CloudFront distribution, see Creating or Updating a Web Distribution Using the CloudFront Console. This solution also uses CloudFormation to simplify the provisioning process. For more information, see What is AWS CloudFormation?

Step 1: Create the example configuration CloudFormation stack

  1. To start the wizard that creates a CloudFormation stack, choose the link for the region in which you want to create AWS resources:
  1. If you are not already signed in to the AWS Management Console, sign in when prompted.
  2. On the Select Template page, choose Next.
  3. On the Specify Details page, specify the following values:

  • Stack name – You can use the default name AWSWafSample, or you can change the name. The stack name cannot contain spaces and must be unique within your AWS account.
  • WAF Web ACL Name – Specify a name for the web ACL that CloudFormation will create. The name that you specify is also used as a prefix for the match conditions and rules that CloudFormation will create, so you can easily find all of the related objects.
  • Action For All Rules – Specify the action for all rules. The default of COUNT will pass all requests and monitor, and BLOCK will block all requests that match.
  • White List CIDR – Specify a single IP range that will be allowed to bypass all rules in CIDR notation. Note that only /8, /16, /24, and /32 are accepted. For a single IP you would enter x.x.x.x/32, for example.
  • Max Size of URI – Select from the list an acceptable size limit for the URI of a request.
  • Max Size of Query String – Select from the list an acceptable size limit for the query string of a request.
  1. Choose Next.
  2. (Optional) On the Options page, enter tags and advanced settings, or leave the fields blank. Choose Next.
  3. On the Review page, review the configuration and then choose Create. CloudFormation then creates the AWS WAF resources.

Step 2: Update your CloudFront distribution settings

After CloudFormation creates the AWS WAF stack, associate the CloudFront distribution with the new AWS WAF web ACL.

To associate your CloudFront distribution with AWS WAF:

  1. Open the CloudFront console.
  2. In the top pane of the console, select the distribution for which you want AWS WAF to monitor requests. (If you do not already have a distribution, see Getting Started with CloudFront.)
  3. In the Distribution Settings pane, choose the General tab, and then choose Edit.
  4. In the AWS WAF Web ACL list, choose the web ACL that CloudFormation created for you in Step 1.
  5. Choose Yes, Edit to save your changes.

Step 3: (Optional) Delete your CloudFormation stack

If you want to delete the CloudFormation stack created in the previous steps (including example rules and match conditions):

  1. Open the CloudFormation console.
  2. Select the check box for the stack; the default name is AWSWafSample
  3. Choose Delete Stack from the Actions drop-down menu.
  4. Choose Yes, Delete to confirm.
  5. To track the progress of the stack deletion, select the check box for the stack, and choose the Events tab in the bottom pane.

Testing the solution

After creating the example CloudFormation stack (Step 1) and associating the AWS WAF web ACL with a CloudFront distribution (Step 2), you can monitor the web requests and determine if the rules require modification to suit your web application.

In the AWS WAF console, you can view a sample of the requests that CloudFront has forwarded to AWS WAF for inspection. For each sampled request, you can view detailed information about the request, such as the originating IP address and the headers included in the request. You can also view which rule the request matched, and whether the rule is configured to allow or block requests.

To view a sample of the web requests that CloudFront has forwarded to AWS WAF:

  1. Sign in to the AWS WAF console.
  2. In the navigation pane, click the name of the web ACL for which you want to view requests.
  3. In the right pane, choose the Requests tab. The Sampled requests table displays the following values for each request:

  • Source IP – Either the IP address that the request originated from or, if the viewer used an HTTP proxy or a load balancer to send the request, the IP address of the proxy or load balancer. 


  • URI – The part of a URL that identifies a resource (for example, /images/daily-ad.jpg). 


  • Matches rule – The first rule in the web ACL for which the web request matched all of the match conditions. If a web request does not match all of the conditions in any rule in the web ACL, the value of Matches rule is Default. Note that when a web request matches all of the conditions in a rule and the action for that rule is Count, AWS WAF continues inspecting the web request based on subsequent rules in the web ACL. In this case, a web request could appear twice in the list of sampled requests: once for the rule that has an action of Count, and again for a subsequent rule or the default action.
  • Action – Whether the action for the corresponding rule is Allow, Block, or Count.
  • Time – The time when AWS WAF received the request from CloudFront.



  1. To refresh the list of sample requests, choose Get new samples.

You may also want to analyze your CloudFront or web application log files for bots, scrapers, or generally unwanted behavior, and modify the rules and match conditions to block them. For further information about CloudFront logs, see Access Logs.

Finally, to enforce blocking of malicious requests for all rules:

  1. Open the CloudFormation console.
  2. Select the check box for the master stack. The default name is AWSWafSample.
  3. Choose Update Stack from the Actions drop-down menu.
  4. Choose Use Current Template and Next.
  5. Choose BLOCK for Actions for All Rules.
  6. Accept changes and choose Next.
  7. To track the progress of the stack update, select the check box for the stack, and choose the Events tab in the bottom pane.

A zipped version of the CloudFormation templates for the example stack and other AWS WAF example solutions are available in our GitHub repository: aws-waf-sample repository.

Summary

This blog post has shown you how to use CloudFormation to automate the configuration of a basic set of rules and match conditions to get started with AWS WAF. If you would like to see more sample rule sets for a specific platform or application, or if you have a comment about this blog post, submit a comment in the “Comments” section below. If you have questions about this blog post, please start a new thread on the AWS WAF forum.

– Ben

How to Centrally Manage AWS Config Rules across Multiple AWS Accounts

Post Syndicated from Chayan Biswas original https://aws.amazon.com/blogs/devops/how-to-centrally-manage-aws-config-rules-across-multiple-aws-accounts/

AWS Config Rules allow you to codify policies and best practices for your organization and evaluate configuration changes to AWS resources against these policies. If you manage multiple AWS accounts, you might want to centrally govern and define these policies for all of the AWS accounts in your organization. With appropriate authorization, you can create a Config rule in one account that uses an AWS Lambda function owned by another account. Such a setup allows you to maintain a single copy of the Lambda function. You do not have to duplicate source code across accounts.

In this post, I will show you how to create Config rules with appropriate cross-account Lambda function authorization. I’ll use a central account that I refer to as the admin-account to create a Lambda function. All of the other accounts then point to the Lambda function owned by the admin-account to create a Config rule. Let’s call one of these accounts the managed-account. This setup allows you to maintain tight control over the source code and eliminates the need to create a copy of Lambda functions in all of the accounts. You no longer have to deploy updates to the Lambda function in these individual accounts.

We will complete these steps for the setup:

  1. Create a Lambda function for a cross-account Config rule in the admin-account.
  2. Authorize Config Rules in the managed-account to invoke a Lambda function in the admin-account.
  3. Create an IAM role in the managed-account to pass to the Lambda function.
  4. Add a policy and trust relationship to the IAM role in the managed-account.
  5. Pass the IAM role from the managed-account to the Lambda function.

Step 1: Create a Lambda Function for a Cross-Account Config Rule

Let’s first create a Lambda function in the admin-account. In this example, the Lambda function checks if log file validation is enabled for all of the AWS CloudTrail trails. Enabling log file validation helps you determine whether a log file was modified or deleted after CloudTrail delivered it. For more information about CloudTrail log file validation, see Validating CloudTrail Log File Integrity.

Note: This rule is an example only. You do not need to create this specific rule to set up cross-account Config rules. You can apply the concept illustrated here to any new or existing Config rule.

To get started, in the AWS Lambda console, choose the config-rule-change-triggered blueprint.

 

Next, modify the evaluateCompliance function and the handler invoked by Lambda. Leave the rest of the blueprint code as is.

function evaluateCompliance(configurationItem, ruleParameters) {
    checkDefined(configurationItem, 'configurationItem');
    checkDefined(configurationItem.configuration, 'configurationItem.configuration');
    checkDefined(ruleParameters, 'ruleParameters');
    //Check if the resource is of type CloudTrail
    if ('AWS::CloudTrail::Trail' !== configurationItem.resourceType) {
        return 'NOT_APPLICABLE';
    }
    //If logfileValidation is enabled, then the trail is compliant
    else if (configurationItem.configuration.logFileValidationEnabled) {
        return 'COMPLIANT';
    }
    else {
        return 'NON_COMPLIANT';
    }
}

In this code snippet, we first ensure that the evaluation is being performed for a trail. Then we check whether the LogFileValidationEnabled property of the trail is set to true. If log file validation is enabled, the trail is marked compliant. Otherwise, the trail is marked noncompliant.

Because this Lambda function is created for reporting evaluation results in the managed-account, the Lambda function will need to be able to call the PutEvaluations Config API (and other APIs, if needed) on the managed-account. We’ll pass the ARN of an IAM role in the managed-account to this Lambda function as a rule parameter. We will need to add a few lines of code to the Lambda function’s handler in order to assume the IAM role passed on by the Config rule in the managed-account:

exports.handler = (event, context, callback) => {
    event = checkDefined(event, 'event');
    const invokingEvent = JSON.parse(event.invokingEvent);
    const ruleParameters = JSON.parse(event.ruleParameters);
    const configurationItem = checkDefined(invokingEvent.configurationItem, 'invokingEvent.configurationItem');
    let compliance = 'NOT_APPLICABLE';
    const putEvaluationsRequest = {}; 
    if (isApplicable(invokingEvent.configurationItem, event)) {
        // Invoke the compliance checking function.
        compliance = evaluateCompliance(invokingEvent.configurationItem, ruleParameters);
    } 
    // Put together the request that reports the evaluation status
    // Note that we're choosing to report this evaluation against the resource that was passed in.
    // You can choose to report this against any other resource type supported by Config 

    putEvaluationsRequest.Evaluations = [{
        ComplianceResourceType: configurationItem.resourceType,
        ComplianceResourceId: configurationItem.resourceId,
        ComplianceType: compliance,
        OrderingTimestamp: configurationItem.configurationItemCaptureTime
    }];
    putEvaluationsRequest.ResultToken = event.resultToken;
    // Assume the role passed from the managed-account
    aws.config.credentials = new aws.TemporaryCredentials({RoleArn: ruleParameters.executionRole});
    let config = new aws.ConfigService({});
    // Invoke the Config API to report the result of the evaluation
    config.putEvaluations(putEvaluationsRequest, callback);
};

In this code snippet, the ARN of the IAM role in the managed-account is passed to this Lambda function as a rule parameter called executionRole. The highlighted lines of code are used to assume the role in the managed-account. Finally, we select the appropriate role (in the admin-account) and save the function.

Make a note of the IAM role in admin-account assigned to the Lambda function and the ARN of the Lambda function. We’ll need to refer these later. You can find the ARN of the Lambda function in the upper-right corner of the AWS Lambda console.

Step 2: Authorize Config Rules in Other Accounts to Invoke a Lambda Function in Your Account

Because the Lambda function we just created will be invoked by the managed-account, we need to add resource policies to allow the managed-account to perform this action. Resource policies to Lambda functions can be applied only through the AWS CLI or SDKs.

Here’s a CLI command you can use to add the resource policy for the managed-account:

$ aws lambda add-permission 
  --function-name cloudtrailLogValidationEnabled 
  --region <region> 
  --statement-id <id> 
  --action "lambda:InvokeFunction" 
  --principal config.amazonaws.com 
  --source-account <managed-account> 

This statement allows only the principal config.amazonaws.com (AWS Config) for the specified source-account to perform the InvokeFunction action on AWS Lambda functions. If more than one account will invoke the Lambda function, each account must be authorized.

Step 3: Create an IAM Role to Pass to the Lambda Function

Next, we need to create an IAM role in the managed-account that can be assumed by the Lambda function. If you want to use an existing role, you can skip to step 4.

Sign in to the AWS IAM console of one of the managed-accounts. In the left navigation, choose Roles, and then choose Create New Role.

On the Set Role Name page, type a name for the role:

Because we are creating this role for cross-account access between the AWS accounts we own, on the Select Role Type page, select Role for Cross-Account Access:

After we choose this option, we must type the account number of the account to which we want to allow access. In our case, we will type the account number of the admin-account.

After we complete this step, we can attach policies to the role. We will skip this step for now. Choose Next Step to review and create the role.

Step 4: Add Policy and Trust Relationships to the IAM Role

From the IAM console of the managed-account, choose the IAM role that the Lambda function will assume, and then click it to modify the role:

We now see options to modify permissions and trust relationships. This IAM role must have, at minimum, permission to call the PutEvaluations Config API in the managed-account. You can attach an existing managed policy or create an inline policy to grant permission to the role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "config:PutEvaluations"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

This policy only allows PutEvaluations action on AWS Config service. You might want to extend permission for the role to perform other actions, depending on the evaluation logic you implement in the Lambda function.

We also need to ensure that the trust relationship is set up correctly. If you followed the steps in this post to create the role, you will see the admin-account has already been added as a trusted entity. This trust policy allows any entity in the admin-account to assume the role.

You can edit the trust relationship to restrict permission only to the role in the admin-account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<admin-account>:role/lambda_config_role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Here, lambda_config_role is the role we assigned to the Lambda function we created in the admin-account.

Step 5: Pass the IAM Role to the Lambda Function

The last step involves creating a custom rule in the managed-account. In the AWS Config console of the managed-account, follow the steps to create a custom Config rule. On the rule creation page, we will provide a name and description and paste the ARN of the Lambda function we created in the admin-account:

Because we want this rule to be triggered upon changes to CloudTrail trails, for Trigger type, select Configuration changes. For Scope of changes, select Resources. For Resources, add CloudTrail:Trail. Finally, add the executionRole rule parameter and paste the ARN of the IAM role: arn:aws:iam::<managed-account>:role/config-rule-admin.

 

Save your changes and then create the rule. After the rule is evaluated, inspect the results:

In this example, there are two CloudTrail trails, one of which is noncompliant. Upon further inspection, we find that the noncompliant trail does not have log file validation enabled:

After we enable log file validation, the rule will be evaluated again and the trail will be marked compliant.

In case you are managing multiple AWS accounts, you may want an easy way to create the Config rule and IAM role in all the accounts in your organization. This can be achieved by using the AWS CloudFormation template I have provided here. Before using this CloudFormation template, replace the admin-account placeholder with the account number of the AWS account you plan to use for centrally managing the Lambda function. Once the Config rule and IAM role are set up in all the managed accounts, you can simply modify the Lambda function in the admin-account to add further checks.

Conclusion

In this blog post, I showed how you can create AWS Config Rules that use Lambda functions with cross-account authorization. This setup allows you to centrally manage the Config rules and associated Lambda functions and retain control over the source code. As an alternative to this approach, you can use a CloudFormation template to create and update Config rules and associated Lambda functions in the managed accounts. The cross-account authorization we set up for the Lambda function in this blog post can also be extended to perform actions beyond reporting evaluation results. To do this, you need to add permission for the relevant APIs in the managed accounts.

We welcome your feedback! Leave comments in the section below or contact us on the AWS Config forum.

How to Centrally Manage AWS Config Rules across Multiple AWS Accounts

Post Syndicated from Chayan Biswas original http://blogs.aws.amazon.com/application-management/post/Tx23LIUFRTWOHNB/How-to-Centrally-Manage-AWS-Config-Rules-across-Multiple-AWS-Accounts

AWS Config Rules allow you to codify policies and best practices for your organization and evaluate configuration changes to AWS resources against these policies. If you manage multiple AWS accounts, you might want to centrally govern and define these policies for all of the AWS accounts in your organization. With appropriate authorization, you can create a Config rule in one account that uses an AWS Lambda function owned by another account. Such a setup allows you to maintain a single copy of the Lambda function. You do not have to duplicate source code across accounts.

In this post, I will show you how to create Config rules with appropriate cross-account Lambda function authorization. I’ll use a central account that I refer to as the admin-account to create a Lambda function. All of the other accounts then point to the Lambda function owned by the admin-account to create a Config rule. Let’s call one of these accounts the managed-account. This setup allows you to maintain tight control over the source code and eliminates the need to create a copy of Lambda functions in all of the accounts. You no longer have to deploy updates to the Lambda function in these individual accounts.

We will complete these steps for the setup:

  1. Create a Lambda function for a cross-account Config rule in the admin-account.
  2. Authorize Config Rules in the managed-account to invoke a Lambda function in the admin-account.
  3. Create an IAM role in the managed-account to pass to the Lambda function.
  4. Add a policy and trust relationship to the IAM role in the managed-account.
  5. Pass the IAM role from the managed-account to the Lambda function.

Step 1: Create a Lambda Function for a Cross-Account Config Rule

Let’s first create a Lambda function in the admin-account. In this example, the Lambda function checks if log file validation is enabled for all of the AWS CloudTrail trails. Enabling log file validation helps you determine whether a log file was modified or deleted after CloudTrail delivered it. For more information about CloudTrail log file validation, see Validating CloudTrail Log File Integrity.

Note: This rule is an example only. You do not need to create this specific rule to set up cross-account Config rules. You can apply the concept illustrated here to any new or existing Config rule.

To get started, in the AWS Lambda console, choose the config-rule-change-triggered blueprint.

 

Next, modify the evaluateCompliance function and the handler invoked by Lambda. Leave the rest of the blueprint code as is.

function evaluateCompliance(configurationItem, ruleParameters) {
    checkDefined(configurationItem, 'configurationItem');
    checkDefined(configurationItem.configuration, 'configurationItem.configuration');
    checkDefined(ruleParameters, 'ruleParameters');
    //Check if the resource is of type CloudTrail
    if ('AWS::CloudTrail::Trail' !== configurationItem.resourceType) {
        return 'NOT_APPLICABLE';
    }
    //If logfileValidation is enabled, then the trail is compliant
    else if (configurationItem.configuration.logFileValidationEnabled) {
        return 'COMPLIANT';
    }
    else {
        return 'NON_COMPLIANT';
    }
}

In this code snippet, we first ensure that the evaluation is being performed for a trail. Then we check whether the LogFileValidationEnabled property of the trail is set to true. If log file validation is enabled, the trail is marked compliant. Otherwise, the trail is marked noncompliant.

Because this Lambda function is created for reporting evaluation results in the managed-account, the Lambda function will need to be able to call the PutEvaluations Config API (and other APIs, if needed) on the managed-account. We’ll pass the ARN of an IAM role in the managed-account to this Lambda function as a rule parameter. We will need to add a few lines of code to the Lambda function’s handler in order to assume the IAM role passed on by the Config rule in the managed-account:

exports.handler = (event, context, callback) => {
    event = checkDefined(event, 'event');
    const invokingEvent = JSON.parse(event.invokingEvent);
    const ruleParameters = JSON.parse(event.ruleParameters);
    const configurationItem = checkDefined(invokingEvent.configurationItem, 'invokingEvent.configurationItem');
    let compliance = 'NOT_APPLICABLE';
    const putEvaluationsRequest = {}; 
    if (isApplicable(invokingEvent.configurationItem, event)) {
        // Invoke the compliance checking function.
        compliance = evaluateCompliance(invokingEvent.configurationItem, ruleParameters);
    } 
    // Put together the request that reports the evaluation status
    // Note that we're choosing to report this evaluation against the resource that was passed in.
    // You can choose to report this against any other resource type supported by Config 

    putEvaluationsRequest.Evaluations = [{
        ComplianceResourceType: configurationItem.resourceType,
        ComplianceResourceId: configurationItem.resourceId,
        ComplianceType: compliance,
        OrderingTimestamp: configurationItem.configurationItemCaptureTime
    }];
    putEvaluationsRequest.ResultToken = event.resultToken;
    // Assume the role passed from the managed-account
    aws.config.credentials = new aws.TemporaryCredentials({RoleArn: ruleParameters.executionRole});
    let config = new aws.ConfigService({});
    // Invoke the Config API to report the result of the evaluation
    config.putEvaluations(putEvaluationsRequest, callback);
};

In this code snippet, the ARN of the IAM role in the managed-account is passed to this Lambda function as a rule parameter called executionRole. The highlighted lines of code are used to assume the role in the managed-account. Finally, we select the appropriate role (in the admin-account) and save the function.


Make a note of the IAM role in admin-account assigned to the Lambda function and the ARN of the Lambda function. We’ll need to refer these later. You can find the ARN of the Lambda function in the upper-right corner of the AWS Lambda console.

Step 2: Authorize Config Rules in Other Accounts to Invoke a Lambda Function in Your Account

Because the Lambda function we just created will be invoked by the managed-account, we need to add resource policies to allow the managed-account to perform this action. Resource policies to Lambda functions can be applied only through the AWS CLI or SDKs.

Here’s a CLI command you can use to add the resource policy for the managed-account:

$ aws lambda add-permission \
  --function-name cloudtrailLogValidationEnabled \
  --region <region> \
  --statement-id <id> \
  --action "lambda:InvokeFunction" \
  --principal config.amazonaws.com \
  --source-account <managed-account> \

This statement allows only the principal config.amazonaws.com (AWS Config) for the specified source-account to perform the InvokeFunction action on AWS Lambda functions. If more than one account will invoke the Lambda function, each account must be authorized.

Step 3: Create an IAM Role to Pass to the Lambda Function

Next, we need to create an IAM role in the managed-account that can be assumed by the Lambda function. If you want to use an existing role, you can skip to step 4.

Sign in to the AWS IAM console of one of the managed-accounts. In the left navigation, choose Roles, and then choose Create New Role.

On the Set Role Name page, type a name for the role:

Because we are creating this role for cross-account access between the AWS accounts we own, on the Select Role Type page, select Role for Cross-Account Access:

After we choose this option, we must type the account number of the account to which we want to allow access. In our case, we will type the account number of the admin-account.

After we complete this step, we can attach policies to the role. We will skip this step for now. Choose Next Step to review and create the role.

Step 4: Add Policy and Trust Relationships to the IAM Role

From the IAM console of the managed-account, choose the IAM role that the Lambda function will assume, and then click it to modify the role:

We now see options to modify permissions and trust relationships. This IAM role must have, at minimum, permission to call the PutEvaluations Config API in the managed-account. You can attach an existing managed policy or create an inline policy to grant permission to the role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "config:PutEvaluations"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

This policy only allows PutEvaluations action on AWS Config service. You might want to extend permission for the role to perform other actions, depending on the evaluation logic you implement in the Lambda function.

We also need to ensure that the trust relationship is set up correctly. If you followed the steps in this post to create the role, you will see the admin-account has already been added as a trusted entity. This trust policy allows any entity in the admin-account to assume the role.

You can edit the trust relationship to restrict permission only to the role in the admin-account:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<admin-account>:role/lambda_config_role"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Here, lambda_config_role is the role we assigned to the Lambda function we created in the admin-account.

Step 5: Pass the IAM Role to the Lambda Function

The last step involves creating a custom rule in the managed-account. In the AWS Config console of the managed-account, follow the steps to create a custom Config rule. On the rule creation page, we will provide a name and description and paste the ARN of the Lambda function we created in the admin-account:

Because we want this rule to be triggered upon changes to CloudTrail trails, for Trigger type, select Configuration changes. For Scope of changes, select Resources. For Resources, add CloudTrail:Trail. Finally, add the executionRole rule parameter and paste the ARN of the IAM role: arn:aws:iam::<managed-account>:role/config-rule-admin.

 

Save your changes and then create the rule. After the rule is evaluated, inspect the results:

In this example, there are two CloudTrail trails, one of which is noncompliant. Upon further inspection, we find that the noncompliant trail does not have log file validation enabled:

After we enable log file validation, the rule will be evaluated again and the trail will be marked compliant.

In case you are managing multiple AWS accounts, you may want an easy way to create the Config rule and IAM role in all the accounts in your organization. This can be achieved by using the AWS CloudFormation template I have provided here. Before using this CloudFormation template, replace the admin-account placeholder with the account number of the AWS account you plan to use for centrally managing the Lambda function. Once the Config rule and IAM role are set up in all the managed accounts, you can simply modify the Lambda function in the admin-account to add further checks.

Conclusion

In this blog post, I showed how you can create AWS Config Rules that use Lambda functions with cross-account authorization. This setup allows you to centrally manage the Config rules and associated Lambda functions and retain control over the source code. As an alternative to this approach, you can use a CloudFormation template to create and update Config rules and associated Lambda functions in the managed accounts. The cross-account authorization we set up for the Lambda function in this blog post can also be extended to perform actions beyond reporting evaluation results. To do this, you need to add permission for the relevant APIs in the managed accounts.

We welcome your feedback! Leave comments in the section below or contact us on the AWS Config forum.

Now Open – AWS Asia Pacific (Mumbai) Region

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/now-open-aws-asia-pacific-mumbai-region/

We are expanding the AWS footprint again, this time with a new region in Mumbai, India. AWS customers in the area can use the new Asia Pacific (Mumbai) Region to better serve end users in India.

New Region
The new Mumbai region has two Availability Zones, raising the global total to 35. It supports Amazon Elastic Compute Cloud (EC2) (C4, M4, T2, D2, I2, and R3 instances are available) and related services including Amazon Elastic Block Store (EBS), Amazon Virtual Private Cloud, Auto Scaling, and  Elastic Load Balancing.

It also supports the following services:

There are now three edge locations (Mumbai, Chennai, and New Delhi) in India. The locations support Amazon Route 53, Amazon CloudFront, and S3 Transfer Acceleration. AWS Direct Connect support is available via our Direct Connect Partners (listed below).

This is our thirteenth region (see the AWS Global Infrastructure map for more information). As usual, you can see the list of regions in the region menu of the Console:

Customers
There are over 75,000 active AWS customers in India, representing a diverse base of industries. In the time leading up to today’s launch, we have provided some of these customers with access to the new region in preview form. Two of them (Ola Cabs and NDTV) were kind enough to share some of their experience and observations with us:

Ola Cabs’ mobile app leverages AWS to redefine point-to-point transportation in more than 100 cities across India. AWS allows OLA to constantly innovate faster with new features and services for their customers, without compromising on availability or the customer experience of their service. Ankit Bhati (CTO and Co-Founder) told us:

We are using technology to create mobility for a billion Indians, by giving them convenience and access to transportation of their choice. Technology is a key enabler, where we use AWS to drive supreme customer experience, and innovate faster on new features & services for our customers. This has helped us reach 100+ cities & 550K driver partners across India. We do petabyte scale analytics using various AWS big data services and deep learning techniques, allowing us to bring our driver-partners close to our customers when they need them. AWS allows us to make 30+ changes a day to our highly scalable micro-services based platform consisting of 100s of low latency APIs, serving millions of requests a day. We have tried the AWS India region. It is great and should help us further enhance the experience for our customers.


NDTV, India’s leading media house is watched by millions of people across the world. NDTV has been using AWS since 2009 to run their video platform and all their web properties. During the Indian general elections in May 2014, NDTV fielded an unprecedented amount of web traffic that scaled 26X from 500 million hits per day to 13 billion hits on Election Day (regularly peaking at 400K hits per second), all running on AWS.  According to Kawaljit Singh Bedi (CTO of NDTV Convergence):

NDTV is pleased to report very promising results in terms of reliability and stability of AWS’ infrastructure in India in our preview tests. Based on tests that our technical teams have run in India, we have determined that the network latency from the AWS India infrastructure Region are far superior compared to other alternatives. Our web and mobile traffic has jumped by over 30% in the last year and as we expand to new territories like eCommerce and platform-integration we are very excited on the new AWS India region launch. With the portfolio of services AWS will offer at launch, low latency, great reliability, and the ability to meet regulatory requirements within India, NDTV has decided to move these critical applications and IT infrastructure all-in to the AWS India region from our current set-up.

 


Here are some of our other customers in the region:

Tata Motors Limited, a leading Indian multinational automotive manufacturing company runs its telematics systems on AWS. Fleet owners use this solution to monitor all vehicles in their fleet on a real time basis. AWS has helped Tata Motors become to more agile and has increased their speed of experimentation and innovation.

redBus is India’s leading bus ticketing platform that sells their tickets via web, mobile, and bus agents. They now cover over 67K routes in India with over 1,800 bus operators. redBus has scaled to sell more than 40 million bus tickets annually, up from just 2 million in 2010. At peak season, there are over 100 bus ticketing transactions every minute. The company also recently developed a new SaaS app on AWS that gives bus operators the option of handling their own ticketing and managing seat inventories. redBus has gone global expanding to new geographic locations such as Singapore and Peru using AWS.

Hotstar is India’s largest premium streaming platform with more than 85K hours of drama and movies and coverage of every major global sporting event. Launched in February 2015, Hotstar quickly became one of the fastest adopted new apps anywhere in the world. It has now been downloaded by more than 68M users and has attracted followers on the back of a highly evolved video streaming technology and high attention to quality of experience across devices and platforms.

Macmillan India has provided publishing services to the education market in India for more than 120 years. Prior to using AWS, Macmillan India has moved its core enterprise applications — Business Intelligence (BI), Sales and Distribution, Materials Management, Financial Accounting and Controlling, Human Resources and a customer relationship management (CRM) system from an existing data center in Chennai to AWS. By moving to AWS, Macmillan India has boosted SAP system availability to almost 100 percent and reduced the time it takes them to provision infrastructure from 6 weeks to 30 minutes.

Partners
We are pleased to be working with a broad selection of partners in India. Here’s a sampling:

  • AWS Premier Consulting Partners – BlazeClan Technologies Pvt. Limited, Minjar Cloud Solutions Pvt Ltd, and Wipro.
  • AWS Consulting Partners – Accenture, BluePi, Cloudcover, Frontier, HCL, Powerupcloud, TCS, and Wipro.
  • AWS Technology Partners – Freshdesk, Druva, Indusface, Leadsquared, Manthan, Mithi, Nucleus Software, Newgen, Ramco Systems, Sanovi, and Vinculum.
  • AWS Managed Service Providers – Progressive Infotech and Spruha Technologies.
  • AWS Direct Connect Partners – AirTel, Colt Technology Services,  Global Cloud Xchange, GPX, Hutchison Global Communications, Sify, and Tata Communications.

Amazon Offices in India
We have opened six offices in India since 2011 – Delhi, Mumbai, Hyderabad, Bengaluru, Pune, and Chennai. These offices support our diverse customer base in India including enterprises, government agencies, academic institutions, small-to-mid-size companies, startups, and developers.

Support
The full range of AWS Support options (Basic, Developer, Business, and Enterprise) is also available for the Mumbai Region. All AWS support plans include an unlimited number of account and billing support cases, with no long-term contracts.

Compliance
Every AWS region is designed and built to meet rigorous compliance standards including ISO 27001, ISO 9001, ISO 27017, ISO 27018, SOC 1, SOC 2, and PCI DSS Level 1 (to name a few). AWS implements an information Security Management System (ISMS) that is independently assessed by qualified third parties. These assessments address a wide variety of requirements which are communicated to customers by making certifications and audit reports available, either on our public-facing website or upon request.

To learn more; take a look at the AWS Cloud Compliance page and our Data Privacy FAQ.

Use it Now
This new region is now open for business and you can start using it today! You can find additional information about the new region, documentation on how to migrate, customer use cases, information on training and other events, and a list of AWS Partners in India on the AWS site.

We have set up a seller of record in India (known as AISPL); please see the AISPL customer agreement for details.


Jeff;

 

Announcing Two New AWS Quick Start Reference Deployments for Compliance

Post Syndicated from Lou Vecchioni original https://blogs.aws.amazon.com/security/post/Tx1AAQ3GYTYNVSC/Announcing-Two-New-AWS-Quick-Start-Reference-Deployments-for-Compliance

As part of the Professional Services Enterprise Accelerator – Compliance program, AWS has published two new Quick Start reference deployments to assist federal government customers and others who need to meet National Institute of Standards and Technology (NIST) SP 800-53 (Revision 4) security control requirements, including those at the high-impact level. The new Quick Starts are AWS Enterprise Accelerator – Compliance: NIST-based Assurance Frameworks and AWS Enterprise Accelerator – Compliance: Standardized Architecture for NIST High-Impact Controls Featuring Trend Micro Deep Security. These Quick Starts address many of the NIST controls at the infrastructure layer. Furthermore, for systems categorized as high impact, AWS has worked with Trend Micro to incorporate its Deep Security product into a Quick Start deployment in order to address many additional high-impact controls at the workload layer (app, data, and operating system). In addition, we have worked with Telos Corporation to populate security control implementation details for each of these Quick Starts into the Xacta product suite for customers who rely upon that suite for governance, risk, and compliance workflows.

These two Quick Starts are designed to help various customers, including those who deploy systems that must:

  • Go through a NIST-based assessment and authorization (A&A).
  • Meet NIST SP 800-171 requirements related to Controlled Unclassified Information (CUI).
  • Provide Trusted Internet Connection (TIC) capabilities.
  • Meet Department of Defense (DoD) Cloud Security Requirements Guide (SRG) requirements for levels 4–5.

Each Quick Start builds a recommended architecture which, when deployed as a package, provides a baseline AWS security-related configuration, and for the NIST high-impact Quick Start, includes a Trend Micro Deep Security configuration. The architectures are instantiated by these Quick Starts through sets of nested AWS CloudFormation templates and user data scripts that build an example environment with a two-VPC, multi-tiered web service. The NIST high-impact version launches Trend Micro Deep Security and deploys a single agent with relevant security configurations on all Amazon EC2 instances within the architecture. For more information about Deep Security, go to Defend your AWS workloads.

The Quick Starts also include:

  • AWS Identity and Access Management (IAM) resources – Policies, groups, roles, and instance profiles.
  • Amazon S3 buckets – Encrypted web content, logging, and backup.
  • A bastion host for troubleshooting and administration.
  • An encrypted Amazon RDS database instance running in multiple Availability Zones.
  • A logging/monitoring/alerting configuration that makes use of AWS CloudTrail, Amazon CloudWatch, and AWS Config Rules.

The recommended architecture supports a wide variety of AWS best practices (all of which are detailed in the document) that include the use of multiple Availability Zones, isolation using public and private subnets, load balancing, and auto scaling.

Both Quick Start packages include a deployment guide with detailed instructions, and a security controls matrix that describes how NIST SP 800-53 controls are addressed by the deployment. The matrix should be reviewed by your IT security assessors and risk decision makers so that they can understand the extent of the implementation of the controls within the architecture. The security controls matrix also identifies the specific resources within the CloudFormation templates that affect each control, and contains cross-references to:

Though it is impossible to automatically implement all system-specific controls within a baseline architecture, these two Quick Start packages can help you address and document a significant portion of the technical controls (including up to 38% of the NIST high-impact controls, in the case of the NIST high-impact Quick Start variant).

In addition, Telos Corporation has integrated the controls compliance and guidance information from these packages into its Xacta product. This makes it simpler for you to inherit this information for your IT governance, risk, and compliance programs, and to document the differences or changes in your security compliance posture. For users of the Xacta product suite, this integration can reduce the effort required to produce bodies of evidence for A&A activities when leveraging AWS Cloud infrastructure. For more information about Telos and Xacta, see Telos and Amazon Web Services: Accelerating Secure and Compliant Cloud Deployments.

To access these new Quick Starts, see the Compliance, security, and identity management section on the AWS Quick Start Reference Deployments page.

If you have questions about these Quick Starts, please contact the AWS Compliance Accelerator team.

– Lou

How to Record SSH Sessions Established Through a Bastion Host

Post Syndicated from Nicolas Malaval original https://blogs.aws.amazon.com/security/post/Tx2HSURNJZPUP68/How-to-Record-SSH-Sessions-Established-Through-a-Bastion-Host

A bastion host is a server whose purpose is to provide access to a private network from an external network, such as the Internet. Because of its exposure to potential attack, a bastion host must minimize the chances of penetration. For example, you can use a bastion host to mitigate the risk of allowing SSH connections from an external network to the Linux instances launched in a private subnet of your Amazon Virtual Private Cloud (VPC).

In this blog post, I will show you how to leverage a bastion host to record all SSH sessions established with Linux instances. Recording SSH sessions enables auditing and can help in your efforts to comply with regulatory requirements.

The solution architecture

In this section, I present the architecture of this solution and explain how you can configure the bastion host to record SSH sessions. Later in this post, I provide instructions about how to implement and test the solution.

Amazon VPC enables you to launch AWS resources on a virtual private network that you have defined. The bastion host runs on an Amazon EC2 instance that is typically in a public subnet of your Amazon VPC. Linux instances are in a subnet that is not publicly accessible, and they are set up with a security group that allows SSH access from the security group attached to the underlying EC2 instance running the bastion host. Bastion host users connect to the bastion host to connect to the Linux instances, as illustrated in the following diagram.

You can adapt this architecture to meet your own requirements. For example, you could have the bastion host in a separate Amazon VPC and a VPC peering connection between the two Amazon VPCs. What matters is that the bastion host remains the only source of SSH traffic to your Linux instances.

This blog post’s solution for recording SSH sessions resides on the bastion host only and requires no specific configuration of Linux instances. You configure the solution by running commands at launch as the root user on an Amazon Linux instance.

Note: It is a best practice to harden your bastion host because it is a critical point of network security. Hardening might include disabling unnecessary applications or services, tuning the network stack, and the like. I do not discuss hardening in detail in this blog post.

When a client connects to an Amazon Linux instance, the default behavior of OpenSSH, the SSH server, is to run an interactive shell. Instead, I configure OpenSSH to execute a custom script that wraps an interactive shell into a script command. By doing so, the script command records everything displayed on the terminal, including keyboard input and full-screen applications such as vim. You can replay the session from the resulting log files by using the scriptreplay command. See the step-by-step example in the “Testing the solution” section later in this blog post.

Note that I intentionally block a few SSH features because they would allow users to create a direct connection between their local computer and the Linux instances, thereby bypassing the solution.

# Create a new folder for the log files
mkdir /var/log/bastion

# Allow ec2-user only to access this folder and its content
chown ec2-user:ec2-user /var/log/bastion
chmod -R 770 /var/log/bastion
setfacl -Rdm other:0 /var/log/bastion

# Make OpenSSH execute a custom script on logins
echo -e "\nForceCommand /usr/bin/bastion/shell" >> /etc/ssh/sshd_config

# Block some SSH features that bastion host users could use to circumvent 
# the solution
awk '!/AllowTcpForwarding/' /etc/ssh/sshd_config > temp && mv temp /etc/ssh/sshd_config
awk '!/X11Forwarding/' /etc/ssh/sshd_config > temp && mv temp /etc/ssh/sshd_config
echo "AllowTcpForwarding no" >> /etc/ssh/sshd_config
echo "X11Forwarding no" >> /etc/ssh/sshd_config

mkdir /usr/bin/bastion

cat > /usr/bin/bastion/shell << 'EOF'

# Check that the SSH client did not supply a command
if [[ -z $SSH_ORIGINAL_COMMAND ]]; then

  # The format of log files is /var/log/bastion/YYYY-MM-DD_HH-MM-SS_user
  LOG_FILE="`date --date="today" "+%Y-%m-%d_%H-%M-%S"`_`whoami`"
  LOG_DIR="/var/log/bastion/"

  # Print a welcome message
  echo ""
  echo "NOTE: This SSH session will be recorded"
  echo "AUDIT KEY: $LOG_FILE"
  echo ""

  # I suffix the log file name with a random string. I explain why 
  # later on.
  SUFFIX=`mktemp -u _XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX`

  # Wrap an interactive shell into "script" to record the SSH session
  script -qf --timing=$LOG_DIR$LOG_FILE$SUFFIX.time $LOG_DIR$LOG_FILE$SUFFIX.data --command=/bin/bash

else

  # The "script" program could be circumvented with some commands 
  # (e.g. bash, nc). Therefore, I intentionally prevent users 
  # from supplying commands.

  echo "This bastion supports interactive sessions only. Do not supply a command"
  exit 1

fi

EOF

# Make the custom script executable
chmod a+x /usr/bin/bastion/shell

# Bastion host users could overwrite and tamper with an existing log file 
# using "script" if they knew the exact file name. I take several measures 
# to obfuscate the file name:
# 1. Add a random suffix to the log file name.
# 2. Prevent bastion host users from listing the folder containing log 
# files. 
# This is done by changing the group owner of "script" and setting GID.
chown root:ec2-user /usr/bin/script
chmod g+s /usr/bin/script

# 3. Prevent bastion host users from viewing processes owned by other 
# users, because the log file name is one of the "script" 
# execution parameters.
mount -o remount,rw,hidepid=2 /proc
awk '!/proc/' /etc/fstab > temp && mv temp /etc/fstab
echo "proc /proc proc defaults,hidepid=2 0 0" >> /etc/fstab

# Restart the SSH service to apply /etc/ssh/sshd_config modifications.
service sshd restart

The preceding commands make OpenSSH execute a custom script on login, which records SSH sessions into log files stored in the folder,/var/log/bastion. For durable storage, the log files are copied at a regular interval to an Amazon S3 bucket, using theaws s3 cp command, which follows.

cat > /usr/bin/bastion/sync_s3 << 'EOF'
# Copy log files to S3 with server-side encryption enabled.
# Then, if successful, delete log files that are older than a day.
LOG_DIR="/var/log/bastion/"
aws s3 cp $LOG_DIR s3://bucket-name/logs/ --sse --region region --recursive && find $LOG_DIR* -mtime +1 -exec rm {} \;

EOF

chmod 700 /usr/bin/bastion/sync_s3

At this point, OpenSSH is configured to record SSH sessions and the log files are copied to Amazon S3. In order to determine the origin of any action performed on the Linux instances using SSH, bastion host users are provided with personal user accounts on the bastion host and they use their personal SSH key pair to log in. Each user account receives the minimum required privileges so that bastion host users are unable to disable or tamper with the solution.

To ease the management of user accounts, the SSH public key of each bastion host user is uploaded to an S3 bucket. At a regular interval, the bastion host retrieves the public keys available in this bucket. For each public key, a user account is created if it does not already exist, and the SSH public key is copied to the bastion host to allow the user to log in with this key pair. For example, if the bastion host finds a file, john.pub, in the bucket, which is John’s SSH public key, it creates a user account, john, and the public key is copied to /home/john/.ssh/authorized_keys. If an SSH public key were to be removed from the S3 bucket, the bastion host would delete the related user account. Personal user account creations and deletions are logged in /var/log/bastion/users_changelog.txt.

The following commands create a shell script for managing personal user accounts and scheduling a cron job to run the shell script every 5 minutes.

# Bastion host users should log in to the bastion host with 
# their personal SSH key pair. The public keys are stored on 
# S3 with the following naming convention: "username.pub". This 
# script retrieves the public keys, creates or deletes local user 
# accounts as needed, and copies the public key to 
# /home/username/.ssh/authorized_keys

cat > /usr/bin/bastion/sync_users << 'EOF'

# The file will log user changes
LOG_FILE="/var/log/bastion/users_changelog.txt"

# The function returns the user name from the public key file name.
# Example: public-keys/sshuser.pub => sshuser
get_user_name () {
  echo "$1" | sed -e 's/.*\///g' | sed -e 's/\.pub//g'
}

# For each public key available in the S3 bucket
aws s3api list-objects --bucket bucket-name --prefix public-keys/ --region region --output text --query 'Contents[?Size>`0`].Key' | sed -e 'y/\t/\n/' > ~/keys_retrieved_from_s3
while read line; do
  USER_NAME="`get_user_name "$line"`"

  # Make sure the user name is alphanumeric
  if [[ "$USER_NAME" =~ ^[a-z][-a-z0-9]*$ ]]; then

    # Create a user account if it does not already exist
    cut -d: -f1 /etc/passwd | grep -qx $USER_NAME
    if [ $? -eq 1 ]; then
      /usr/sbin/adduser $USER_NAME && \
      mkdir -m 700 /home/$USER_NAME/.ssh && \
      chown $USER_NAME:$USER_NAME /home/$USER_NAME/.ssh && \
      echo "$line" >> ~/keys_installed && \
      echo "`date --date="today" "+%Y-%m-%d %H-%M-%S"`: Creating user account for $USER_NAME ($line)" >> $LOG_FILE
    fi

    # Copy the public key from S3, if a user account was created 
    # from this key
    if [ -f ~/keys_installed ]; then
      grep -qx "$line" ~/keys_installed
      if [ $? -eq 0 ]; then
        aws s3 cp s3://bucket-name/$line /home/$USER_NAME/.ssh/authorized_keys --region region
        chmod 600 /home/$USER_NAME/.ssh/authorized_keys
        chown $USER_NAME:$USER_NAME /home/$USER_NAME/.ssh/authorized_keys
      fi
    fi

  fi
done < ~/keys_retrieved_from_s3

# Remove user accounts whose public key was deleted from S3
if [ -f ~/keys_installed ]; then
  sort -uo ~/keys_installed ~/keys_installed
  sort -uo ~/keys_retrieved_from_s3 ~/keys_retrieved_from_s3
  comm -13 ~/keys_retrieved_from_s3 ~/keys_installed | sed "s/\t//g" > ~/keys_to_remove
  while read line; do
    USER_NAME="`get_user_name "$line"`"
    echo "`date --date="today" "+%Y-%m-%d %H-%M-%S"`: Removing user account for $USER_NAME ($line)" >> $LOG_FILE
    /usr/sbin/userdel -r -f $USER_NAME
  done < ~/keys_to_remove
  comm -3 ~/keys_installed ~/keys_to_remove | sed "s/\t//g" > ~/tmp && mv ~/tmp ~/keys_installed
fi

EOF

chmod 700 /usr/bin/bastion/sync_users

cat > ~/mycron << EOF
*/5 * * * * /usr/bin/bastion/sync_s3
*/5 * * * * /usr/bin/bastion/sync_users
0 0 * * * yum -y update --security
EOF
crontab ~/mycron
rm ~/mycron

Be very careful when distributing the key pair associated with the instance running the bastion host. With this key pair, someone could log in as the root or ec2-user user and damage or tamper with the solution. You might even consider launching the instance without a key pair if the operations requiring root access, such as patching, can be scripted and automated.

Also, you should restrict the permissions on the S3 bucket by using bucket or IAM policies. For example, you could make the log files readable only by a compliance team and the SSH public keys managed by a DevOps team.

Implementing the solution

Now that you understand the architecture of this solution, you can follow the instructions in this section to implement in your AWS account this blog post’s solution.

First, you will create two new key pairs. The first key pair will be associated with the instance running the bastion host. The second key pair will be associated with an Amazon Linux instance launched in a private subnet and will be used as the SSH key pair for a bastion host user.

To manually create the two key pairs:

  1. Open the Amazon EC2 console and select a region from the navigation bar.
  2. Click Key Pairs in the left pane.
  3. Click Create Key Pair.
  4. In the Key pair name box, type bastion and click Create. Your browser downloads the private key file as bastion.pem.
  5. Repeat Steps 3 and 4 to create another key pair and name it sshuser.

You then will use AWS CloudFormation to provision the required resources. Click Create a Stack to open the CloudFormation console and create a CloudFormation stack from the template. Click Next and enter bastion in BastionKeyPair and sshuser in InstanceKeyPair. Then, follow the on-screen instructions.

CloudFormation creates the following resources:

  • An Amazon VPC with an Internet gateway attached.
  • A public subnet on this Amazon VPC with a new route table to make it publicly accessible.
  • A private subnet on this Amazon VPC that will not have access to the Internet, for sake of simplicity.
  • An S3 bucket. The log files will be stored in a folder called logs and the SSH public keys in a folder called public-keys.
  • Two security groups. The first security group allows SSH traffic from the Internet, and the second security group allows SSH traffic from the first security group.
  • An IAM role to grant an EC2 instance permissions to upload log files to the S3 bucket and to read SSH public keys.
  • An Amazon Linux instance running the bastion host in the public subnet with the IAM role attached and the user data script entered to configure the solution.
  • An Amazon Linux instance in the private subnet.

After the stack creation has completed (it can take up to 10 minutes), click the Outputs tab in the CloudFormation console and note the values that the process returned: the name of the new S3 bucket, the public IP address of the bastion host, and the private IP address of the Linux instance.

Finally, you will upload the SSH public key for the key pair sshuser to the S3 bucket so that the bastion host creates a new user account:

  1. Retrieve the public key and save it locally to a file named sshuser.pub (see Retrieving the Public Key for Your Key Pair on Linux or Retrieving the Public Key for Your Key Pair on Windows).
  2. Open the S3 console and click the name of the bucket in the buckets list.
  3. Create a new folder called public-keys (see Creating a Folder), and upload the SSH public key sshuser.pub to this folder (see Uploading Objects into Amazon S3).
  4. Wait for a few minutes. You should see a new folder called logs in the bucket with a new file inside it that is recording events related to user account creation and deletion.

Testing the solution

You might recall that the key pair sshuser serves to log in to both the bastion host as a bastion host user and the Linux instance in the private subnet as the privileged ec2-user user. Therefore, to test this solution, you will use SSH agent forwarding to connect from the bastion host to the Linux instance without storing the private key on the bastion host. See this blog post for further information about SSH agent forwarding.

First, you will log in to the bastion host as sshuser with the -A argument to forward the SSH private key.

chmod 600 [path to sshuser.pem]
ssh-add [path to sshuser.pem]
ssh -A [email protected][public IP of the bastion host] –i [path to sshuser.pem]

You should see a welcome message saying that the SSH session will be recorded, as shown in the following screenshot.

Write down the value of the audit key. Then, connect to the Linux instance, run some commands of your choice, and then close the SSH session.

ssh [email protected][private IP of the Linux instance]
[commands of your choice that will be recorded]
exit
exit

You will now replay the SSH session that was just recorded. Each session has two log files: one file contains the data displayed on the terminal, and the other contains the timing data that enables replay with realistic typing and output delays. For simplicity, you will connect as ec2-user to the bastion host and replay from the local copy of log files. 

Note: Under normal circumstances, you would not replay a SSH session on the bastion host, for two reasons. First, recall that you should strictly avoid using the privileged user account, ec2-user. Second, the bastion host does not have read permissions on the folder logs in the S3 bucket and the log files that are older than a day are deleted from the bastion host. Instead, you would use another Linux instance with sufficient permissions on the S3 bucket to download and replay the log files.

ssh [email protected][public IP of the bastion host] –i [path to bastion.pem]
export LOGFILE=`ls /var/log/bastion/[audit key]*.data | cut -d. -f1`
scriptreplay --timing=$LOGFILE.time $LOGFILE.data

You can now delete the CloudFormation stack to clean up the resources that were just created. Note that you need to empty the S3 bucket before it can be deleted by CloudFormation (see Empty a Bucket).

Conclusion

A bastion host is a standard element of network security that provides secure access to private networks over SSH. In this blog post, I have shown you a way to leverage this bastion host to record SSH sessions, which you can play back and use for auditing purposes.

If you have comments, submit them in the “Comments” section below. If you have questions, please start a new thread on the Amazon VPC forum.

– Nicolas

New in AWS Marketplace: Alces Flight – Effortless HPC on Demand

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-in-aws-marketplace-alces-flight-effortless-hpc-on-demand/

In the past couple of years, academic and corporate researchers have begun to see the value of the cloud. Faced with a need to run demanding jobs and to deliver meaningful results as quickly as possible while keeping costs under control, they are now using AWS to run a wide variety of compute-intensive, highly parallel workloads.

Instead of fighting for time on a cluster that must be shared with other researchers, they accelerate their work by launching clusters on demand, running their jobs, and then shutting the cluster down shortly thereafter, paying only for the resources that they consume. They replace tedious RFPs, procurement, hardware builds and acceptance testing with cloud resources that they can launch in minutes. As their needs grow, they can scale the existing cluster or launch a new one.

This self-serve, cloud-based approach favors science over servers and accelerates the pace of research and innovation. Access to shared, cloud-based resources can be granted to colleagues located on the same campus or halfway around the world, without having to worry about potential issues at organizational or network boundaries.

Alces Flight in AWS Marketplace
Today we are making Alces Flight available in AWS Marketplace. This is a fully-featured HPC environment that you can launch in a matter of minutes. It can make use of On-Demand or Spot Instances and comes complete with a job scheduler and hundreds of HPC applications that are all set up and ready to run. Some of the applications include built-in collaborative features such as shared graphical views. For example, here’s the Integrative Genomics Viewer (IGV):

Each cluster is launched into a Virtual Private Cloud (VPC) with SSH and graphical desktop connectivity. Clusters can be of fixed size, or can be Auto Scaled in order to meet changes in demand.  Once launched, the cluster looks and behaves just like a traditional Linux-powered HPC cluster, with shared NFS storage and passwordless SSH access to the compute nodes. It includes access to HPC applications, libraries, tools, and MPI suites.

We are launching Alces Flight in AWS Marketplace today. You can launch a small cluster (up to 8 nodes) for evaluation and testing or a larger cluster for research.

If you subscribe to the product, you can download the AWS CloudFormation template from the Alces site. This template powers all of the products, and is used to quickly launch all of the AWS resources needed to create the cluster.

EC2 Spot Instances give you access to spare AWS capacity at up to a 90% discount from On-Demand pricing and can significantly reduce your cost per core. You simply enter the maximum bid price that you are willing to pay for a single compute node; AWS will manage your bid, running the nodes when capacity is available at the desired price point.

Running Alces Flight
In order to get some first-hand experience with Alces Flight, I launched a cluster of my own. Here are the settings that I used:

I set a tag for all of the resources in the stack as follows:

I confirmed my choices and gave CloudFormation the go-ahead to create my cluster. As expected, the cluster was all set up and ready to go within 5 minutes. Here are some of the events that were logged along the way:

Then I SSH’ed in to the login node and saw the greeting, all as expected:

After I launched my cluster I realized that this post would be more interesting if I had more compute nodes in my cluster. Instead of starting over, I simply modified my CloudFormation stack to have 4 nodes instead of 1, applied the change, and watched as the new nodes came online. Since I specified the use of Spot Instances when I launched the cluster, Auto Scaling placed bids automatically. Once the nodes were online I was able to locate them from within my PuTTY session:

Then I used the pdsh (Parallel Distributed Shell command) to check on the up-time of each compute node:

Learn More
This barely counts as scratching the surface; read Getting Started as Quickly as Possible to learn a lot more about what you can do! You should also watch one or more of the Alces videos to see this cool new product in action.

If you are building and running data-intensive HPC applications on AWS, you may also be interested in another Marketplace offering. The BeeGFS (self-supported or support included) parallel file system runs across multiple EC2 instances, aggregating the processing  power into a single namespace, with all data stored on EBS volumes.  The self-supported product is also available on a 14 day free trial. You can create a cluster file system using BeeGFS and then use it as part of your Alces cluster.


Jeff;

 

Now Available: PCI DSS Quick Start for Deploying PCI DSS In-Scope Workloads

Post Syndicated from Balaji Palanisamy original https://blogs.aws.amazon.com/security/post/Tx2ZHLDGY0EL8Z1/Now-Available-PCI-DSS-Quick-Start-for-Deploying-PCI-DSS-In-Scope-Workloads


Released today, the PCI DSS Quick Start includes learnings from AWS field teams that have migrated and deployed workloads that are in scope for Payment Card Industry Data Security Standard (PCI DSS) compliance. The AWS CloudFormation templates and scripts included in this Quick Start can help you build a standardized environment that supports compliance with the applicable PCI DSS controls. A deployment guide with detailed instructions for deployment and configuration is also included in the Quick Start.

PCI DSS version 3.1 was used as the baseline during the creation of this Quick Start. The included CloudFormation templates employ the concept of nesting to build independent stacks for the global, network, access, and application portions of the architecture.

The first guide in the AWS Enterprise Accelerator – Compliance series targeted organizations that follow the National Institute of Standards and Technology (NIST) 800-53 standards. In the NIST series, we identified positive enterprise patterns and included foundational elements that can kick-start you when deploying regulated workloads. Features such as an Amazon S3 bucket for storing AWS CloudTrail logs and a read-only AWS Identity and Access Management (IAM) role to support visibility for the security teams were included in “NIST Quick Start” templates so that customers could build their technical security controls on top of these pre-built constructs.

“PCI DSS Quick Start” builds on the core CloudFormation templates and includes configurations such as a “write once, read many” S3 policy for the S3 bucket storing PCI-specific logs. “PCI DSS Quick Start” includes other modifications based on customer feedback as well as patterns for the most common use cases.

The core concepts and the architecture of “PCI DSS Quick Start” are described in detail in the following resources that are included in the download:

For more information about the Quick Start series, see AWS Quick Start Reference Deployments.

If the scope of your PCI environment is enterprise scale with the accompanying complexity, and you need assistance with the implementation of the concepts presented in this Quick Start, AWS Professional Services offers a custom “Compliance Playbook for PCI Workloads” to help you with the AWS portion of your PCI DSS compliance program. Contact your AWS Account Manager for further information, or contact us by email.

– Balaji

New AWS Quick Start Reference Deployment – Standardized Architecture for PCI DSS

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/new-aws-quick-start-reference-deployment-standardized-architecture-for-pci-dss/

If you build an application that processes credit card data, you need to conform to PCI DSS (Payment Card Industry Data Security Standard). Adherence to the standard means that you need to meet control objectives for your network, protect cardholder data, implement strong access controls, and more.

In order to help AWS customers to build systems that conform to PCI DSS, we are releasing a new Quick Start Reference Deployment. The new Standardized Architecture for PCI DSS on the AWS Cloud (PDF or HTML) includes a AWS CloudFormation template that deploys a standardized environment that falls in scope for PCI DSS compliance (version 3.1).

The template describes a stack that deploys a multi-tiered Linux-based web application in about 30 minutes. It makes use of child templates, and can be customized as desired. It launches a pair of Virtual Private Clouds (Management and Production) and can accommodate a third VPC for development:

The template sets up the IAM items (policies, groups, roles, and instance profiles), S3 buckets (encrypted web content, logging, and backup), a Bastion host for troubleshooting and administration, an encrypted RDS database instance running in multiple Availability Zones, and a logging / monitoring / alerting package that makes use of AWS CloudTrail, Amazon CloudWatch, and AWS Config Rules. The architecture supports a wide variety of AWS best practices (all of which are detailed in the document) including use of multiple Availability Zones, isolation using public and private subnets, load balancing, auto scaling, and more.

You can use the template to set up an environment that you can use for learning, as a prototype, or as the basis for your own template.

The Quick Start also includes a Security Controls Reference. This document maps the security controls called out by PCI DSS to the relevant architecture decisions, features, and configurations.


Jeff;

PS – Check out our other AWS Enterprise Accelerator Quick Starts!

 

 

How to Automatically Tag Amazon EC2 Resources in Response to API Events

Post Syndicated from Alessandro Martini original https://blogs.aws.amazon.com/security/post/Tx150Z810KS4ZEC/How-to-Automatically-Tag-Amazon-EC2-Resources-in-Response-to-API-Events

Access to manage Amazon EC2 instances can be controlled using tags. You can do this by writing an Identity and Access Management (IAM) policy that grants users permissions to manage EC2 instances that have a specific tag. However, if you also give users permissions to create or delete tags, users can manipulate the values of the tags to gain access and manage additional instances.

In this blog post, I will explore a method to automatically tag an EC2 instance and its associated resources without granting ec2:createTags permission to users. I will use a combination of an Amazon CloudWatch Events rule and AWS Lambda to tag newly created instances. With this solution, your users do not need to have permissions to create tags because the Lambda function will have the permissions to tag the instances. The solution can be automatically deployed in the region of your choice with AWS CloudFormation. I explain the provided solution and the CloudFormation template in the following sections.

Solution overview

This solution has two parts. I first explore the AWS Identity and Access Management (IAM) policy associated with my IAM user, Bob. Then, I explore how you can tag EC2 resources automatically in response to specific API events in my AWS account.

The IAM configuration

My IAM user, Bob, belongs to an IAM group with a customer managed policy called [Your CloudFormation StackName]TagBasedEC2RestrictionsPolicy[Unique ID], as shown in the following screenshot. For convenience, I will refer to this policy as TagBasedEC2RestrictionsPolicy in the remainder of this post. (Throughout this post, you should replace placeholder values with your own AWS information.)

The content of the customer managed policy, TagBasedEC2RestrictionsPolicy, follows.

{
               "Version" : "2012-10-17",
               "Statement" : [
                   {
                       "Sid" : "LaunchEC2Instances",
                      "Effect" : "Allow",
                       "Action" : [
                           "ec2:Describe*",
                           "ec2:RunInstances"
                       ],
                       "Resource" : [
                           "*"
                       ]
                   },
                   {
                       "Sid" : "AllowActionsIfYouAreTheOwner",
                       "Effect" : "Allow",
                       "Action" : [
                           "ec2:StopInstances",
                           "ec2:StartInstances",
                           "ec2:RebootInstances",
                           "ec2:TerminateInstances"
                       ],
                       "Condition" : {
                           "StringEquals" : {
                               "ec2:ResourceTag/PrincipalId" : "${aws:userid}"
                           }
                       },
                       "Resource"  : [
                           "*"
                       ]
                   }
               ]
}
 
This policy explicitly allows all EC2 describe actions and ec2:runInstances (in the LaunchEC2Instances statement). The core of the policy is in the AllowActionsIfYouAreTheOwner statement. This statement applies a condition to EC2 actions we want to limit, in which we allow the action only if a tag named PrincipalId matches your current user ID. I am using the conditional variable, "${aws:userid}", because it is always defined for any type of authenticated user. On the other hand, the AWS variable, aws:username, is only present for IAM users, and not for federated users.

For example, an IAM user cannot see the unique identifier, UserId, from the IAM console, but you can retrieve it with the AWS CLI by using the following command.

aws iam get-user --user-name Bob

The following output comes from that command.

{
    "User": {
        "UserName": "Bob",
        "PasswordLastUsed": "2016-04-29T20:52:37Z",
        "CreateDate": "2016-04-26T19:26:43Z",
        "UserId": "AIDAJ7EQQEKFAVPO3NTQG",
        "Path": "/",
        "Arn": "arn:aws:iam::111122223333:user/Bob"
    }
}

In other cases, such as when assuming an IAM role to access an AWS account, the UserId is a combination of the assumed IAM role ID and the role session name that you specified at the time of the AssumeRole API call.

role-id:role-session-name

For a full list of values that you can substitute for policy variables, see Request Information That You Can Use for Policy Variables.

Because you cannot possibly memorize all these IDs, the automation defined later in this post not only tags resources with the UserId, but also it retrieves the actual userName (or RoleSessionName), and uses this value to define a more human-readable tag, Owner, that can be used to filter or report resources in an easier way.

Tag automation

The IAM user has EC2 rights to launch an EC2 instance. Regardless of how the user creates the EC2 instance (with the AWS Management Console or AWS CLI), he performs a RunInstances API call (#1 in the following diagram). CloudWatch Events records this activity (#2).

A CloudWatch Events rule targets a Lambda function called AutoTag and it invokes the function with the event details (#3). The event details contain the information about the user that completed the action (this information is retrieved automatically from AWS CloudTrail, which must be on for CloudWatch Events to work).

The Lambda function AutoTag scans the event details, and extracts all the possible resource IDs as well as the user’s identity (#4). The function applies two tags to the created resources (#5):

  • Owner, with the current userName.
  • PrincipalId, with the current user’s aws:userid value.

When Amazon Elastic Block Store (EBS) volumes, EBS snapshots and Amazon Machine Images (AMIs) are individually created, they invoke the same Lambda function. This way, you can similarly Allow or Deny actions based on tags for those resources, or identify which resources a user created.

CloudFormation automation

This CloudFormation template creates a Lambda function, and CloudWatch Events trigger that function in the region you choose. Lambda permissions to describe and tag EC2 resources are obtained from an IAM role the template creates along with the function. The template also creates an IAM group into which you can place your user to enforce the behavior described in this blog post. The template also creates a customer managed policy so that you can easily apply it to other IAM entities, such as IAM roles or other existing IAM groups.

Note: Currently, CloudWatch Events is available in six regions, and Lambda is available in five regions. Keep in mind that you can only use this post’s solution in regions where both CloudWatch Events and Lambda are available. As these services grow, you will be able to launch the same template in other regions as well.

The template first defines a Lambda function with an alias (PROD) and two versions (Version $LATEST and Version 1), as shown in the following diagram. This is a best practice for continuous deployment: the definition of an alias decouples the trigger from the actual code version you want to execute, while versioning allows you to work on multiple versions of the same code and test changes while a stable version of your code is still available for current executions. For more information about aliases and versions, see AWS Lambda Function Versioning and Aliases.

The Lambda function

You will find the full Lambda function code in the CloudFormation template. I will explain key parts of the code in this section.

The code starts with the definition of an array called ids. This array will contain all the identifiers of EC2 resources found in a given event. Resource IDs could be instances, EBS volumes, EBS snapshots, ENIs, and AMI IDs.

Array initialization

ids = []

The Lambda function extracts the region, the event name (that is, the name of the invoked API, such as RunInstances and CreateVolume), and the user’s principalID and userName. Depending on the user type, the code extracts the userName from the event detail, or it defines it as the second part of the principalID.

Information extraction       

        region = event['region']
        detail = event['detail']
        eventname = detail['eventName']
        arn = event['detail']['userIdentity']['arn']
        principal = event['detail']['userIdentity']['principalId']
        userType = event['detail']['userIdentity']['type']

        if userType == 'IAMUser':
            user = detail['userIdentity']['userName']

        else:
            user = principal.split(':')[1]

Then, the code continues with EC2 client initialization.

EC2 client initialization

        ec2 = boto3.resource('ec2')

After a few input validations, the code looks for the resource IDs in the event detail. For each API call, the resource IDs can be found in different parts of the response, so the code contains a sequence of if/else statements to map these differences.

In the case of an EC2 instance, the function describes the identified instance to find the attached EBS volumes and ENIs, and adds their IDs to the array’s IDs.

Resource IDs extraction for each API call

        if eventname == 'CreateVolume':
            ids.append(detail['responseElements']['volumeId'])


        elif eventname == 'RunInstances':
            items = detail['responseElements']['instancesSet']['items']
            for item in items:
                ids.append(item['instanceId'])

            base = ec2.instances.filter(InstanceIds=ids)

            #loop through the instances
            for instance in base:
                for vol in instance.volumes.all():
                    ids.append(vol.id)
                for eni in instance.network_interfaces:
                    ids.append(eni.id)

        elif eventname == 'CreateImage':
            ids.append(detail['responseElements']['imageId'])

        elif eventname == 'CreateSnapshot':
            ids.append(detail['responseElements']['snapshotId'])

        else:
            logger.warning('Not supported action')

Now that you have all the identifiers, you can use the EC2 client to tag them all with the following tags:
  • Owner, with the value of the variable user.
  • PrincipalId, with the value of the variable principal.

Resource tagging

       if ids:
            ec2.create_tags(Resources=ids, Tags=[{'Key': 'Owner', 'Value': user}, {'Key': 'PrincipalId', 'Value': principal}])

The CloudWatch Events rule

In order to trigger the Lambda function, you create a CloudWatch Events rule. You define a rule to match events and route them to one or more target Lambda functions. In our case, the events we want to match are the following AWS EC2 API calls:

  • RunInstances
  • CreateVolume
  • CreateSnapshot
  • CreateImage

The target of this rule is our Lambda function. We specifically point to the PROD alias of the function to decouple the trigger from a specific version of the Lambda function code.

The following screenshot shows what the CloudFormation template creates.

IAM group

The CloudFormation template creates an IAM group called [Your CloudFormation StackName]-RestrictedIAMGroup-[Unique ID]. The customer managed policy TagBasedEC2RestrictionsPolicy is associated with the group.

The solution in action

First, deploy the CloudFormation template in the region of your choosing. Specify the following Amazon S3 template URL: https://s3.amazonaws.com/awsiammedia/public/sample/autotagec2resources/AutoTag.template, as shown in the following screenshot.

The template creates the stack only if you confirm you have enabled CloudTrail, as shown in the following screenshot.

The deployment should be completed within 5 minutes. The following screenshot shows the status of the stack as CREATE_COMPLETE.

Now that you have deployed the required automation infrastructure, you can assign IAM users to the created IAM group (ManageEC2InstancesGroup in the following screenshot).

Similarly, if you are using federation and your users access your AWS account through an IAM role, you can attach the customer managed policy, TagBasedEC2RestrictionsPolicy, to the IAM role itself.

In my case, I have added my IAM user, Bob, to the IAM group created by CloudFormation. Note that you have to add the IAM users to this group manually.

When Bob signs in to my AWS account, he can create EC2 instances with no restrictions. However, if he tries to stop an EC2 instance that he did not create, he will get the following error message.

In the meantime, the instance Bob created has been automatically tagged with his user name and user ID.

Bob can now terminate the instance. The following screenshot shows the instance in the process of shutting down.

What’s next?

Now that you know you can tag resources with a Lambda function in response to events, you can apply the same logic to other resources such as Amazon Relational Database Service (RDS) databases or S3 buckets. With resource groups, each user can focus on just the resources he created, and the IAM policy provided in this post assures that no Stop/Start/Reboot/Terminate action is possible on someone else’s instance.

Additionally, tags are useful in custom billing reports to project costs and determine how much money each individual owner is spending. You can activate the Owner tag in the billing console from the Cost Allocation Tags of your billing console to include it in your detailed billing reports. For more information, see Applying Tags.

If you have any comments, submit them in the “Comments” section below. If you have questions about the solution in this blog post, please start a new thread on the EC2 forum.

– Alessandro

Real-time in-memory OLTP and Analytics with Apache Ignite on AWS

Post Syndicated from Babu Elumalai original https://blogs.aws.amazon.com/bigdata/post/Tx3RS3V80XNRJH3/Real-time-in-memory-OLTP-and-Analytics-with-Apache-Ignite-on-AWS

Babu Elumalai is a Solutions Architect with AWS

Organizations are generating tremendous amounts of data, and they increasingly need tools and systems that help them use this data to make decisions. The data has both immediate value (for example, trying to understand how a new promotion is performing in real time) and historic value (trying to understand the month-over-month revenue of launched offers on a specific product).

The Lambda  architecture (not AWS Lambda) helps you gain insight into immediate and historic data by having a speed layer and a batch layer. You can use the speed layer for real-time insights and the batch layer for historical analysis.

In this post, we’ll walk through how to:

  1. Build a Lambda architecture using Apache Ignite
  2. Use Apache Ignite to perform ANSI SQL on real-time data
  3. Use Apache Ignite as a cache for online transaction processing (OLTP) reads

To illustrate these approaches, we’ll discuss a simple order-processing application. We will extend the architecture to implement analytics pipelines and then look at how to use Apache Ignite for real-time analytics.

A classic online application

Let’s assume that you’ve built a system to handle the order-processing pipeline for your organization. You have an immutable stream of order documents that are persisted in the OLTP data store. You use Amazon DynamoDB to store the order documents coming from the application.  

Below is an example order payload for this system:

{'BillAddress': '5719 Hence Falls New Jovannitown  NJ 31939', 'BillCity': 'NJ', 'ShipMethod': '1-day', 'UnitPrice': 14, 'BillPostalCode': 31939, 'OrderQty': 1, 'OrderDate': 20160314050030, 'ProductCategory': 'Healthcare'}

{'BillAddress': '89460 Johanna Cape Suite 704 New Fionamouth  NV 71586-3118', 'BillCity': 'NV', 'ShipMethod': '1-hour', 'UnitPrice': 3, 'BillPostalCode': 71586, 'OrderQty': 1, 'OrderDate': 20160314050030, 'ProductCategory': 'Electronics'}

Here is example code that I used to generate sample order data like the preceding and write the sample orders  into DynamoDB.

The illustration following shows the current architecture for this example.

Your first analytics pipeline

Next, suppose that business users in your organization want to analyze the data using SQL or business intelligence (BI) tools for insights into customer behavior, popular products, and so on. They are considering Amazon Redshift for this. Amazon Redshift is a fast, fully managed, petabyte-scale data warehouse that makes it simple and cost-effective to analyze all your data using ANSI SQL or your existing business intelligence tools.

To use this approach, you have to build a pipeline that can extract your order documents from DynamoDB and store them in Amazon Redshift. Let’s look at the components we can use to build this pipeline:

  • DynamoDB Streams captures a time-ordered sequence of item-level modifications in any DynamoDB table and stores this information in a log for up to 24 hours. Applications can access this log and view the data items as they appeared before and after they were modified, in near real time.
  • AWS Lambda lets you run code without provisioning or managing servers.
  • Amazon Kinesis Firehose can capture and automatically load streaming data into Amazon S3 and Amazon Redshift, enabling near real-time analytics with existing business intelligence tools and dashboards.

You can connect DynamoDB Streams, Lambda, and Amazon Kinesis Firehose to build a pipeline that continuously streams data from DynamoDB to Amazon Redshift:

  1. Create your Amazon Redshift cluster in a VPC in the Amazon VPC service.
  1. Create your order tables in the Amazon Redshift cluster. For reference, use the example Create Table statement following.
create table orderdata(orderid varchar(100), orderdate bigint,ShipMethod varchar(10),BillAddress varchar(200),BillCity varchar(50), BillPostalCode int, OrderQty int, UnitPrice int, productcategory varchar(200))distkey(orderid) sortkey(orderdate,productcategory);

 3.   Create a delivery stream in Amazon Kinesis Firehose that delivers incoming events to Amazon Redshift.

  1. Enable DynamoDB Streams on your DynamoDB table by using this approach. Once you’ve enabled DynamoDB Streams, every write to your DynamoDB table is available asynchronously in your streams.
  1. Create a Lambda function that reads the streams data and writes to the Amazon Kinesis Firehose delivery stream. You can follow the instructions in this blog post to create a Lambda function that will process the streams data. I have written example code in Python that processes the order stream data and writes to Firehose.

Using the preceding steps, you will build an architecture like the one below.

You can use an open-source BI solution like Apache Zeppelin to perform analytics on Amazon Redshift as shown above. Apache Zeppelin is available as a sandbox application on Amazon EMR. In the image below, the visualization shows the shipping methods that customers chose for their orders. Apache Zeppelin creates this visualization from Amazon Redshift.

SQL on the data streams

Business users have been content to perform analytics on data collected in Amazon Redshift to spot trends. But recently, they have been asking AWS whether the latency can be reduced for real-time analysis. At the same time, they want to continue using the analytical tools they’re familiar with.

In this situation, we need a system that lets you capture the data stream in real time and use SQL to analyze it in real time.

In the earlier section, you learned how to build the pipeline to Amazon Redshift with Firehose and Lambda functions. The following illustration shows how to use Apache Spark Streaming on EMR to compute time window statistics from DynamoDB Streams. The computed data can be persisted to Amazon S3 and accessed with SparkSQL using Apache Zeppelin.

Note: For this to work, use DynamoDBStreamsAdapterClient and integrate with Amazon Kinesis client library for Spark Streaming provided under the Amazon Software License (ASL).

This is a great option for doing real-time analytics, but it requires that your analysts know how to use Apache Spark to compute results in real time. In the next section, we’ll introduce Apache Ignite and talk about how you can use it as to implement real-time analytics while letting users interact with the data streams using SQL.

What is Apache Ignite?

As the following image shows, Apache Ignite is an in-memory data fabric built on top of a distributed in-memory computing platform. Apache Ignite is optimized for high performance and can process large-scale datasets in real time—orders of magnitude faster than is possible with traditional disk-based or flash-based technologies.

Connecting the pieces with Apache Ignite

The following illustration shows how you can use Apache Ignite to build the architecture we’ve described. In this architecture, you use Amazon Kinesis Client Library (Amazon KCL) to read from the DynamoDB Streams and stream into Apache Ignite. You can directly query data in Ignite as it becomes available and use SQL through Zeppelin. And because the writes from DynamoDB are replicated asynchronously into Apache Ignite, the Ignite cluster can actually serve as an eventually consistent cache.

Deploying Apache Ignite on AWS

You can either use an AWS CloudFormation template or bootstrap actions with Amazon EMR to deploy an Apache Ignite cluster. We have provided a CloudFormation script that will help you deploy Apache Ignite on AWS. Because Apache Ignite is an in-memory technology, you might need to forecast how much data you want Apache Ignite to hold and provision the cluster based on that forecast so that you don’t run out of memory. The following illustration shows a CloudFormation deployment for Apache Ignite.

Note: Apache Ignite typically performs node discovery through multicast. Because AWS doesn’t support multicast at this point, you can use the S3-based discovery tool TcpDiscoveryS3IpFinder, which comes with the Ignite distribution.

When deploying Ignite using CloudFormation, you should use Auto Scaling groups to launch the cluster across multiple Availability Zones for high availability. In addition, Apache Ignite lets you configure replicas for your data through a backup parameter. You set this parameter to 1 to maintain two copies of your data. Apache Ignite also lets you configure a replicated or partitioned cache. A replicated cache makes Ignite replicate every write across every node in the cluster. Use partitioned mode if you want to horizontally scale the cluster with data.

Streaming data into Apache Ignite

Your order data is already available in the DynamoDB Streams. You need to write a KCL app to consume the streams and publish the order data to Apache Ignite. You need to leverage the DynamoDB Streams adapter for Amazon Kinesis to use the KCL library to reliably consume and process the DynamoDB Streams.

The sample code here will help you get started building a KCL app for Apache Ignite from DynamoDB Streams. Below is an excerpt from the code.

TcpDiscoverySpi spi = new TcpDiscoverySpi();
TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
ipFinder.setAddresses(Arrays.asList("127.0.0.1:47500..47509","<IP_ADDRESS1>:47500..47509","<IP_ADDRESS2>:47500..47509"));
spi.setIpFinder(ipFinder);
IgniteConfiguration cfg = new IgniteConfiguration();
cfg.setDiscoverySpi(spi);
cfg.setClientMode(true);
cfg.setPeerClassLoadingEnabled(true);

Ignite ignite = Ignition.start(cfg);
IgniteDataStreamer<String,orderdata> cache = Ignition.ignite().dataStreamer("<cacheName>");
LOG.info(">>> cache acquired");

recordProcessorFactory = new StreamsRecordProcessorFactory(cache);
workerConfig = new KinesisClientLibConfiguration("ddbstreamsprocessing",
streamArn, streamsCredentials, "ddbstreamsworker")
.withMaxRecords(1000)
.withInitialPositionInStream(InitialPositionInStream.TRIM_HORIZON);

System.out.println("Creating worker for stream: " + streamArn);
worker = new Worker(recordProcessorFactory, workerConfig, adapterClient, dynamoDBClient, cloudWatchClient);
System.out.println("Starting worker...");

int exitCode = 0;
try {
worker.run();
catch (Throwable t) {
System.err.println("Caught throwable while processing data.");
      t.printStackTrace();
      exitCode = 1;
}

The KCL app loops in the background to continuously publish the order stream to Apache Ignite. In this case, leverage the Ignite Data Streamer API to push large data streams. The illustration below shows the data streamer in action and how the data can be consumed with SQL on the other side.

Real-time SQL analytics

This architecture allows business users to seamlessly query the order data with ANSI SQL at very low latencies. Apache Ignite also integrates with Apache Zeppelin and can be used to visualize your SQL results using the Ignite interpreter for Zeppelin. The example below shows a simple visualization on a SQL query run on Apache Ignite through Zeppelin, followed by an interpreter configuration for Ignite.

Apache Ignite also allows you to join multiple tables if you have a highly normalized schema like a star schema. You can make use of affinity collocation to collocate same cache keys together for efficient joins by avoiding moving data across the network.

When users run a SQL query, the query runs across multiple nodes in the cluster, emulating an massively parallel processing (MPP) architecture. In partitioned cache mode, each node is responsible for its own data. This approach allows Ignite to parallelize SQL query execution in memory, resulting in significantly higher performance for analytics.

You can also define indexes on your datasets to further improve performance, and you can configure Ignite to store these indexes as off-heap structures.

Consider running Apache Ignite clusters on R3 instance types on AWS. R3 instances are memory-optimized and are a great fit for memory intensive workloads. We also expect to launch X1 instance types later this year. These instances will feature up to 2 TB of memory and might also be a great choice to run in-memory Ignite clusters in the future.

Sliding window analysis

It’s easy to configure sliding windows on Apache Ignite because you can define an expiration on your dataset. You can configure a time-based window to expire data after, say, 30 seconds to provide a 30-second sliding window of your data. You might need to create a separate cache for this and stream the data into this cache as well.

The following illustration shows an Apache Ignite sliding window.

Using the cluster as an eventually consistent cache

In this design, we stream data continuously to Apache Ignite. Because the data writes are happening on DynamoDB, the Apache Ignite cluster can also be considered an eventually consistent cache for DynamoDB. So your OLTP read logic can be changed to something like the following for cases when you can use eventually consistent reads:

Read Key K1
	Read K1 from Apache Ignite
	If K1 not found
		Cache Miss
		Read from DynamoDB
		Populate Apache Ignite with K1
		Return K1 to client
	Else
		Return K1 to client

Conclusion

In this post, we looked at a business problem and how Apache Ignite can be applied to solving that business problem through its support for an in-memory data fabric. Apache Ignite has other features like ACID-compliant distributed transaction support; publish/subscribe (pub/sub) cluster-wide messaging; the Ignite resilient distributed dataset (RDD), which is an implementation of the native Spark RDD that lets you share Spark RDD across applications; and many more.

To use Apache Ignite in this way, you need to deploy and manage it on AWS. Before you invest in this, consider whether an architecture based on managed services meets your needs. In this scenario you have to manage the Apache Ignite cluster, so you must be careful about choosing your cache size, the level of replication for your data, how you leverage off-heap memory, how to tune the eviction policy for your cache, and how to tune garbage collection for your Java virtual machine. Understand your data well and test thoroughly with different settings to arrive at an optimal configuration.

If you have questions or suggestions, please leave a comment below.

————————————–

Related

Querying Amazon Kinesis Streams Directly with SQL and Spark Streaming

In Case You Missed These: AWS Security Blog Posts from March and April

Post Syndicated from Craig Liebendorfer original https://blogs.aws.amazon.com/security/post/Tx3CTMUP8IVAQX0/In-Case-You-Missed-These-AWS-Security-Blog-Posts-from-March-and-April

In case you missed any of the AWS Security Blog posts from March and April, they are summarized and linked to below. The posts are shown in reverse chronological order (most recent first), and the subject matter ranges from the AWS Config Rules repository to automatically updating AWS WAF IP blacklists.

April

April 28, AWS WAF How-To: How to Import IP Address Reputation Lists to Automatically Update AWS WAF IP Blacklists
A number of organizations maintain reputation lists of IP addresses used by bad actors. Their goal is to help legitimate companies block access from specific IP addresses and protect their web applications from abuse. These downloadable, plaintext reputation lists include Spamhaus’s Don’t Route Or Peer (DROP) List and Extended Drop (EDROP) List, and Proofpoint’s Emerging Threats IP list. Similarly, the Tor project’s Tor exit node list provides a list of IP addresses currently used by Tor users to access the Internet. Tor is a web proxy that anonymizes web requests and is sometimes used by malicious users to probe or exploit websites.

April 27, Federated SSO How-To: How to Set Up Federated Single Sign-On to AWS Using Google Apps
Among the services offered to Google Apps for Work users is a Security Assertion Markup Language (SAML) 2.0–based SSO service. You can use this service to provide one-click SSO to your AWS resources by using your existing Google Apps credentials. For users to whom you grant SSO access, they will see an additional SAML app in your Google Apps account, as highlighted in the following screenshot. When your users click the SAML app, Google Apps authenticates and redirects them to the AWS Management Console. In this blog post, I will show you how you can use Google Apps to set up federated SSO to your AWS resources.

April 21, AWS WAF How-To: How to Prevent Hotlinking by Using AWS WAF, Amazon CloudFront, and Referer Checking
You can use AWS WAF to help prevent hotlinking. AWS WAF is a web application firewall that is closely integrated with Amazon CloudFront (AWS’s content delivery network [CDN]), and it can help protect your web applications from common web exploits that could affect application availability, compromise security, and consume excessive resources. In this blog post, I will show you how to prevent hotlinking by using header inspection in AWS WAF, while still taking advantage of the improved user experience from a CDN such as CloudFront.

April 20, Amazon Cognito Announcement: Amazon Cognito Now Provides Sign-Up and Sign-In Functionality for Your Apps (Beta)
Today, Amazon Cognito launched the beta of a new feature that makes it easy for developers to add sign-up and sign-in functionality to mobile and web apps. With this new feature, you get a simple, fully managed service you can use to create and maintain your user pool that can scale to hundreds of millions of users. This new feature also provides enhanced security functionality, such as email verification, phone number verification, and multi-factor authentication. You benefit from the security and privacy best practices of AWS, and retain full control of your user data. To begin using the new beta feature with your user pool, see the Amazon Cognito page.

April 20, Amazon Inspector Announcement: Now Generally Available: Amazon Inspector
Yesterday, AWS announced that Amazon Inspector, an automated security assessment service, is now available to all customers. Inspector helps you improve the security and compliance of your applications running on Amazon Elastic Compute Cloud (Amazon EC2) by identifying potential security issues, vulnerabilities, or deviations from security standards. You pay only for the assessments you run, with the first 250 assessments free for your first 90 days.

April 19, HIPAA FAQ: Frequently Asked Questions About HIPAA Compliance in the AWS Cloud
Today, we continue a series of AWS cloud compliance FAQs by focusing on the Health Insurance Portability and Accountability Act (HIPAA) and protected health information (PHI). AWS’s Healthcare and Life Science customers are doing important things for their customers in the AWS cloud, and we are excited to work with our partners to help tackle medical advancements at scale. In this blog post, I will share some of the broader questions we hear from customers about HIPAA compliance and PHI in the cloud.

April 14, RDS for SQL Server How-To: How to Enable Windows Integrated Authentication for RDS for SQL Server Using On-Premises Active Directory
If you want to run your SQL Server applications in AWS and secure access with on-premises Active Directory user accounts, this blog post is for you. In this blog post, I walk you through the steps to enable RDS for SQL Server to authenticate with Microsoft AD and configure trusts between Microsoft AD and your on-premises Active Directory. With that configuration in place, you can run your SQL Server databases and applications in AWS, and authenticate access with on-premises Active Directory user accounts.

April 7, AWS Directory Service Announcement: Now Available: Simplified Configuration of Trust Relationships in the AWS Directory Service Console
Today, we made it easier for you to configure trust relationships between AWS Directory Service for Microsoft Active Directory (Enterprise Edition), also referred to as Microsoft AD, and your on-premises Microsoft Active Directory. Establishing trust relationships requires conditional forwarders, which resolve Domain Name System (DNS) queries between the domain names of trusting directories. Now, by completing a single field in the Directory Service console at the same time you create a trust relationship, you can more easily configure conditional forwarders. 

April 7, Compliance FAQ: Frequently Asked Questions About Compliance in the AWS Cloud
Every month, AWS Compliance fields thousands of questions about how to achieve and maintain compliance in the cloud. Among other things, customers are eager to take advantage of the cost savings and security at scale that AWS offers while still maintaining robust security and regulatory compliance. Because regulations across industries and geographies can be complex, we thought it might be helpful to share answers to some of the frequently asked questions we hear about compliance in the AWS cloud, as well as to clear up potential misconceptions about how operating in the cloud might affect compliance.

March

March 29, Amazon CloudWatch Events How-To: How to Detect and Automatically Revoke Unintended IAM Access with Amazon CloudWatch Events
If your account is shared across departments in your organization, monitoring the permissions of your users can become a challenge as the number of users grows. For example, what if a user is granted unintended IAM API access and the user begins making API calls? In this post, I will show a solution that detects API callers who should not have IAM access and automatically revokes those permissions with the help of Amazon CloudWatch Events.

March 28, AWS CloudTrail How-To: How to Easily Identify Your Federated Users by Using AWS CloudTrail
CloudTrail now records two additional AWS Security Token Service (AWS STS) API calls: AssumeRoleWithWebIdentity and AssumeRoleWithSAML. If you already have CloudTrail logging enabled, capturing these AWS STS API calls is enabled by default and requires no additional action from you. If you have not enabled CloudTrail already, see the CloudTrail documentation and AWS CloudTrail FAQs for more information. In this blog post, I will show how you can identify a SAML federated user who terminated an EC2 instance in your AWS account.

March 23, AWS Webinar Announcement: Register for and Attend This March 30 Webinar—Best Practices for Managing Security Operations in AWS
AWS Security Solutions Architect Henrik Johansson will share different ways you can use AWS Identity and Access Management (IAM) to control access to your AWS services and integrate your existing authentication system with AWS IAM. You will learn how you can deploy and control your AWS infrastructure as code by using templates, including change management policies with AWS CloudFormation. In addition, you will explore different options for managing both your AWS access logs and your Amazon Elastic Compute Cloud (EC2) system logs using AWS CloudTrail and Amazon CloudWatch Logs. You will also learn how to implement an audit and compliance validation process using AWS Config and Amazon Inspector.

March 22, AWS Encryption SDK How-To: How to Use the New AWS Encryption SDK to Simplify Data Encryption and Improve Application Availability
The AWS Cryptography team is happy to announce the AWS Encryption SDK. This new SDK makes encryption easier for developers while minimizing errors that could lessen the security of your applications. The new SDK does not require you to be an AWS customer, but it does include ready-to-use examples for AWS customers.

March 16, AD FS Federation How-To: How to Set Up Uninterrupted, Federated User Access to AWS Using AD FS
When the token-signing certificate expires, or is changed, the trust relationship between the claim provider, AD FS, and the relying party, AWS Security Token Service (AWS STS), is broken. Without a valid certificate to prove the calling server’s identity, the receiving party cannot verify the certificate, which terminates the request and thus prevents federated users from being able to access the AWS Management Console. Luckily, this can be avoided! In this blog post, I explain how you can use the AutoCertificateRollover feature in AD FS to enable uninterrupted connections between your claim provider and your relying trust. I also show how to set up a secondary certificate manually in AD FS to avoid service interruption when a server certificate expires.

March 8, AWS WAF How-To: How to Reduce Security Threats and Operating Costs Using AWS WAF and Amazon CloudFront
Successfully blocking bad actors can help reduce security threats to your systems. In addition, you can lower your overall costs, because you no longer have to serve traffic to unintended audiences. In this blog post, I will show you how you can realize these benefits by building a process to help detect content scrapers and bad bots, and then use Amazon CloudFront with AWS WAF (a web application firewall [WAF]) to help block bad actors’ access to your content.

March 7, Restricting VPC Access How-To: How to Automate Restricting Access to a VPC by Using AWS IAM and AWS CloudFormation
Back in September, I wrote about How to Help Lock Down a User’s Amazon EC2 Capabilities to a Single VPC. In that blog post, I highlighted what I have found to be an effective approach to the virtual private cloud (VPC) lockdown scenario. Since that time, I have worked on making the related information easier to implement in your environment. As a result, I have developed an AWS CloudFormation template that automates the creation of the resources necessary to lock down AWS Identity and Access Management (IAM) entities (users, groups, and roles) to a VPC. In this blog post, I explain this CloudFormation template in detail and describe its individual sections in order to help you better understand what happens when you create a CloudFormation stack from the template.

March 1, AWS Config Rules Repository Announcement: Announcing the AWS Config Rules Repository: A New Community-Based Source of Custom Rules for AWS Config
Today, we’re happy to release the AWS Config Rules repository, a community-based source of custom AWS Config Rules. This new repository gives you a streamlined way to automate your assessment and compliance against best practices for security of AWS resources. AWS Config Rules is a service that provides automated, periodic security and compliance checking of AWS resources, and affords customers the ability to forego manual inspection of security configurations.

If you have comments  about any of these posts, please add your comments in the "Comments" section of the appropriate post. If you have questions about or issues implementing the solutions in any of these posts, please start a new thread on the AWS IAM forum.

– Craig

AWS Week in Review – April 25, 2016

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/aws-week-in-review-april-25-2016/

Let’s take a quick look at what happened in AWS-land last week:

Monday

April 25

Tuesday

April 26

Wednesday

April 27

Thursday

April 28

Friday

April 29

Saturday

April 30

Sunday

May 1

New & Notable Open Source

New SlideShare Presentations

New Customer Success Stories

Upcoming Events

Help Wanted

Stay tuned for next week! In the meantime, follow me on Twitter and subscribe to the RSS feed.

Jeff;

Autheos – At the Nexus of Marketing and E-Commerce

Post Syndicated from Jeff Barr original https://aws.amazon.com/blogs/aws/autheos-at-the-nexus-of-marketing-and-e-commerce/

In today’s guest post, Leon Mergen, CTO of Autheos, reviews their company history and their move to AWS.


Jeff;


Adding video to a product page on an e-commerce site is perhaps the single most effective way to drive increased sales — studies have shown sales conversion rates can go up by more than two thirds. In addition, product video viewing data fills a gaping hole in a brand’s / supplier’s ability to assess the effectiveness of their online and offline marketing efforts at driving e-commerce sales. We had built an OK product video distribution platform… but we knew we couldn’t scale globally with the technology we were using. So, in September last year, we decided to transition to AWS, and, while doing so built an e-commerce marketing support tool for Brands which, judging by customer response, is a game changer. This is our story.

The Perils of Good Fortune
Autheos was founded in 2012 when the biggest Webshop in Holland and Belgium asked us to turn an existing piece of technology into a video hosting solution that would automatically find and insert product videos into their product sales pages.  A startup rarely finds itself in a better position to start, so we jumped right in and started coding.  Which was, in retrospect, a mistake for two reasons.

For one thing, we grew too fast.  When you have a great client that really wants your product, the natural reaction is to build it as fast as you can.  So, since there wasn’t a team in place, we (too) quickly on-boarded engineers and outsourced several components to remote development shops, which resulted in classic issues of communication problems and technical incompatibilities.

More importantly, however, since we already had an existing piece of technology, we didn’t take the time to think how we would build it if we were starting from scratch.  It seemed like it would be quicker to adapt it to the new requirements.  And kind of like a home-owner who opts for renovation instead of tear-down and rebuild, we had to make all sorts of compromises as a result.

However, thanks to many all-nighters we managed to meet the deadline and launch a platform that allowed brands such as Philips, LEGO, L’Oreal, and Bethesda to upload product videos (commercials, guides, reviews, and so forth) for free and tag them with a product code and language.

The webshops integrated a small piece of javascript code that enabled them to query our video database in real-time with a product code and language, display a custom button if a video was found, and pop up the right videos(s) for the product, in the desired language.

Click here to see an example video on Bol.com (the biggest webshop in Benelux); our video is behind the button.

The results: less work for the bebshop (no more manual gathering of videos, decoding/encoding, hosting and matching them with the right products) and more sales. Our client convinced its Brands to start uploading their videos, and kickstarted our exponential growth. Soon we had so many Brands using our platform, and so many videos in our database, that nearly all major webshops in Benelux wanted to work with us as well (often pushed to do so by Brands, who didn’t want the hassle of interfacing / integrating with many different webshops).

This might sound great, but remember how we built the product in a rush with legacy code?  After three years of fire-fighting, interspersed with frequent moments of disbelief when we found out that certain features we wanted to offer were impossible due to limitations in our backend, we decided enough was enough… it was time to start over.

A New Beginning with AWS
Our key requirements were that we needed to seamlessly scale globally, log and process all of our data, and provide high performance access to our ever growing database of product videos. Besides this, we needed to make sure we could ship new features and products quickly without impacting wider operations. Oh, and we wanted to be up and running with the new platform in 6 months. As the de-facto standard for web applications, the choice of AWS was an easy one. However, we soon realized that it wasn’t just an easy decision, it was a really smart one too.

Elastic Transcoder was the main reason for us to decide to go with AWS. Before working with ET, we used a custom transcoding service that had been built by an outsourced company in Eastern Europe. As a result of hosting the service there on antiquated servers, the transcoding service suffered from lots of downtime, and caused many headaches. Elastic Transcoder allows us to forget about all these problems, and gives us stable transcoding service which we can scale on-demand.

When we moved our application servers to AWS, we also activated Amazon CloudFront. This was a no-brainer for us even though there are many other CDNs available, as CloudFront integrates unbelievably well within AWS. Essentially it just worked. With a few clicks we were able to build a transcoding pipeline that directly uploads its result to CloudFront. We make a single API call, and AWS takes care of the rest, including CDN hosting. It’s really that easy.

As we generate a huge number of log records every day, we had to make sure these were stored in a flexible and scalable environment. A regular PostgreSQL server would have worked, however, this would never have been cost-efficient at our scale. So we started running some prototypes with Amazon Redshift, the PostgreSQL compatible data warehousing solution by AWS. We set up Kinesis Firehose to stream data from our application servers to [redshift_u], writing it off in batches (in essence creating a full ETL process as a service), something that would have taken a major effort with a traditional webhost. Doing this outside of AWS would have taken months; with AWS we managed to set all of this up in three days.

Managing this data through data mining frameworks was the next big challenge, for which many solutions exist in market. However, Amazon has great solutions in an integrated platform that enabled us to test and implement rapidly. For batch processing we use Spark, provided by Amazon EMR. For temporary hooking into data streams – e.g. our monitoring systems – we use AWS Data Pipeline, which gives us access to the stream of data as it is generated by our application servers, comparable to what Apache Kafka would give you.

Everything we use is accessible through an SDK, which allows us to run integration tests effectively in an isolated environment. Instead of having to mock services, or setting up temporary services locally and in our CI environment, we use the AWS SDK to easily create and clean up AWS services. The flexibility and operational effectiveness this brings is incredible, as our whole production environment can be replicated in a programmable setup, in which we can simulate specific experiments. Furthermore, we catch many more problems by actually integrating all services in all automated tests, something you would otherwise only catch during manual testing / staging.

Through AWS CloudFormation and AWS CodeDeploy we seamlessly built our cloud using templates, and integrated this with our testing systems in order to support our Continuous Deployment setup. We could, of course, have used Chef or Puppet with traditional webhosts, but the key benefit in using the AWS services for this is that we have instant access to a comprehensive ecosystem of tools and features with which we can integrate (and de-integrate) as we go.

Unexpected Bounty
One month in, things were going so smoothly that we did something that we had never done before in the history of the company:  we expanded our goals during a project without pushing out the delivery date.  We always knew that we had data that could be really valuable for Brands, but since our previous infrastructure made it really difficult to access or work with this data, we had basically ignored it.  However, when we had just finished our migration to Redshift, one of our developers read an article about the powerful combination of Redshift and Periscope.  So we decided to prototype an e-commerce data analysis tool.

A smooth connection with our Redshift tables was made almost instantly, and we saw our 500+ million records visualized in a few graphs that the Periscope team prepared for us.  Jaws dropped and our product manager went ahead and built an MVP. A few weeks of SQL courses, IRC spamming and nagging the Periscope support team later, and we had an alpha product.

We have shown this to a dozen major Brands and the response has been all we could hope for… a classic case of the fabled product / market fit. And it would not have happened without AWS.

An example of the dashboard for one of our Founding Partners (a global game development company).

Jackpot
With a state of the art platform, promising new products, and the backend infrastructure to support global viral growth we finally had a company that could attract the attention of professional investors… and within a few weeks of making our new pitch we had closed our first outside investment round.

We’ve come a long way from working with a bare bones transcoding server, to building a scalable infrastructure and best-in-class products that are ready to take over the world!

Our very first transcoding server.

What’s Next?
Driving viral spread globally to increase network effects, we are signing up new Webshops and Brands at a tremendous pace.  We are putting the finishing touches on the first version of our ecommerce data analysis product for Brand marketers, and speccing out additional products and features for Brands and Webshops working with the Autheos Network.  And of course we are looking for amazing team members to help make this happen. If you would like to join us on the next stage of our journey, please look at our website for current openings — and yes, we are looking for DevOps engineers!

And lastly, since this is the Amazon Web Services blog, we can’t resist being cheeky and thus herewith take the opportunity to invite Mr. Bezos to sit down with us to see if we can become the global product video partner for Amazon.  One thing’s for sure: our infrastructure is the best!

— Leon Mergen, CTO – [email protected]