We’re excited to announce the completion of Trusted Information Security Assessment Exchange (TISAX) certification on December 08, 2020 for the London and Paris regions. These regions were assessed at the HIGH protection level (AL 2) for the control domains Information Handling and Data Protection, according to article 28 (“Processor”) of the European General Data Protection Regulation (GDPR).
The TISAX certification helps provide automotive industry organizations with the assurance they need to build secure applications and services in the cloud. The certification was established by the German Association of the Automotive Industry (VDA) and is governed by the European Network Exchange (ENX).
With this scope expansion, a total of 10 regions globally (Seattle, Frankfurt, Ireland, Oregon, Ohio, Northern Virginia, Canada, Seoul, London, and Paris) are currently certified for TISAX and demonstrate a consistent and standardized approach to information security systems for the automotive industry.
An independent third-party auditor conducted and accredited the assessment. Automotive customers can rely on the AWS TISAX assessment results and labels published on the ENX Portal for timely exchange of compliance status with their supply chains. The scope ID and assessment ID are STRN58 and AYZ39G, respectively.
AWS Launch Wizard is a console-based service to quickly and easily size, configure, and deploy third party applications, such as Microsoft SQL Server Always On and HANA based SAP systems, on AWS without the need to identify and provision individual AWS resources. AWS Launch Wizard offers an easy way to deploy enterprise applications and optimize costs. Instead of selecting and configuring separate infrastructure services, you go through a few steps in the AWS Launch Wizard and it deploys a ready-to-use application on your behalf. It reduces the time you need to spend on investigating how to provision, cost and configure your application on AWS.
You can now use AWS Launch Wizard to deploy and configure self-managed Microsoft Windows Server Active Directory Domain Services running on Amazon Elastic Compute Cloud (EC2) instances. With Launch Wizard, you can have fully-functioning, production-ready domain controllers within a few hours—all without having to manually deploy and configure your resources.
You can use AWS Directory Service to run Microsoft Active Directory (AD) as a managed service, without the hassle of managing your own infrastructure. If you need to run your own AD infrastructure, you can use AWS Launch Wizard to simplify the deployment and configuration process.
In this post, I walk through creation of a cross-region Active Directory domain using Launch Wizard. First, I deploy a single Active Directory domain spanning two regions. Then, I configure Active Directory Sites and Services to match the network topology. Finally, I create a user account to verify replication of the Active Directory domain.
Figure 1: Diagram of resources deployed in this post
Prerequisites
You must have a VPC in your home. Additionally, you must have remote regions that have CIDRs that do not overlap with each other. If you need to create VPCs and subnets that do not overlap, please refer here.
Each subnet used must have outbound internet connectivity. Feel free to either use a NAT Gateway or Internet Gateway.
The VPCs must be peered in order to complete the steps in this post. For information on creating a VPC Peering connection between regions, please refer here.
If you choose to deploy your Domain Controllers to a private subnet, you must have an RDP jump / bastion instance setup to allow you to RDP to your instance.
Deploy Your Domain Controllers in the Home Region using Launch Wizard
In this section, I deploy the first set of domain controllers into the us-east-1 the home region using Launch Wizard. I refer to US-East-1 as the homeregion, and US-West-2 as the remoteregion.
Assign a Controller IP address for each domain controller
Remote Desktop Gateway preferences: Disregard for now, this is set up later.
Check the I confirm that a public subnet has been set up. Each of the selected private subnets have outbound connectivity enabled check box.
Select Next.
In the Define infrastructure requirements page, set the following inputs.
Storage and compute: Based on infrastructure requirements
Number of AD users: Up to 5000 users
Select Next.
In the Review and deploy page, review your selections. Then, select Deploy.
Note that it may take up to 2 hours for your domain to be deployed. Once the status has changed to Completed, you can proceed to the next section. In the next section, I prepare Active Directory Sites and Services for the second set of domain controller in my other region.
Configure Active Directory Sites and Services
In this section, I configure the Active Directory Sites and Services topology to match my network topology. This step ensures proper Active Directory replication routing so that domain clients can find the closest domain controller. For more information on Active Directory Sites and Services, please refer here.
Retrieve your Administrator Credentials from Secrets Manager
From the AWS Secrets Manager Console in us-east-1, select the Secret that begins with LaunchWizard-UsEast1AD.
In the middle of the Secret page, select Retrieve secret value.
This will display the username and password key with their values.
You need these credentials when you RDP into one of the domain controllers in the next steps.
Rename the Default First Site
Log in to the one of the domain controllers in us-east-1.
Select Start, type dssite and hit Enter on your keyboard.
The Active Directory Sites and Services MMC should appear.
Expand Sites. There is a site named Default-First-Site-Name.
Right click on Default-First-Site-Name select Rename.
Enter us-east-1 as the name.
Leave the Active Directory Sites and Services MMC open for the next set of steps.
Create a New Site and Subnet Definition for US-West-2
Using the Active Directory Sites and Services MMC from the previous steps, right click on Sites.
Select New Site… and enter the following inputs:
Name: us-west-2
Select DEFAULTIPSITELINK.
Select OK.
A pop up will appear telling you there will need to be some additional configuration. Select OK.
Expand Sites and right click on Subnets and select New Subnet.
Enter the following information:
Prefix: the CIDR of your us-west-2 VPC. An example would be 1.0.0/24
Site: select us-west-2
Select OK.
Leave the Active Directory Sites and Services MMC open for the following set of steps.
Configure Site Replication Settings
Using the Active Directory Sites and Services MMC from the previous steps, expand Sites, Inter-Site Transports, and select IP. You should see an object named DEFAULTIPSITELINK,
Right click on DEFAULTIPSITELINK.
Select Properties. Set or verify the following inputs on the General tab:
In the DEFAULTIPSITELINKProperties, select the Attribute Editor tab and modify the following:
Scroll down and double click on Enter 1 for the Value, then select OK twice.
For more information on these settings, please refer here.
Close the Active Directory Sites and Services MMC, as it is no longer needed.
Prepare Your Home Region Domain Controllers Security Group
In this section, I modify the Domain Controllers Security Group in us-east-1. This allows the domain controllers deployed in us-west-2 to communicate with each other.
Assign a Controller IP address for each domain controller
Remote Desktop Gateway preferences: disregard for now, as I set this later.
Check the I confirm that a public subnet has been set up. Each of the selected private subnets have outbound connectivity enabled check box
In the Define infrastructure requirements page set the following:
Storage and compute: Based on infrastructure requirements
Number of AD users: Up to 5000 users
In the Review and deploy page, review your selections. Then, select Deploy.
Note that it may take up to 2 hours to deploy domain controllers. Once the status has changed to Completed, proceed to the next section. In this next section, I prepare Active Directory Sites and Services for the second set of domain controller in another region.
Prepare Your Remote Region Domain Controllers Security Group
In this section, I modify the Domain Controllers Security Group in us-west-2. This allows the domain controllers deployed in us-west-2 to communicate with each other.
Select the Domain Controllers Security Group that was created by your Launch Wizard Active Directory.
Select Edit inbound rules. The Security Group should start with LaunchWizard-UsWest2AD-EC2ADStackExistingVPC-
Choose Add rule and enter the following:
Type: Select All traffic
Protocol: All
Port range: All
Source: Select Custom
Enter the CIDR of your remote VPC. An example would be 0.0.0/24
Choose Save rules.
Create an AD User and Verify Replication
In this section, I create a user in one region and verify that it replicated to the other region. I also use AD replication diagnostics tools to verify that replication is working properly.
Create a Test User Account
Log in to one of the domain controllers in us-east-1.
Select Start, type dsa and press Enter on your keyboard. The Active Directory Users and Computers MMC should appear.
Right click on the Users container and select New > User.
Enter the following inputs:
First name: John
Last name: Doe
User logon name: jdoe and select Next
Password and Confirm password: Your choice of complex password
Uncheck User must change password at next logon
Select Next.
Select Finish.
Verify Test User Account Has Replicated
Log in to the one of the domain controllers in us-west-2.
Select Start and type dsa.
Then, press Enter on your keyboard. The Active Directory Users and Computers MMC should appear.
Select Users. You should see a user object named John Doe.
Note that if the user is not present, it may not have been replicated yet. Replication should not take longer than 60 seconds from when the item was created.
Summary
Congratulations, you have created a cross-region Active Directory! In this post you:
Launched a new Active Directory forest in us-east-1 using AWS Launch Wizard.
Configured Active Directory Sites and Service for a multi-region configuration.
Launched a set of new domain controllers in the us-west-2 region using AWS Launch Wizard.
Created a test user and verified replication.
This post only touches on a couple of features that are available in the AWS Launch Wizard Active Directory deployment. AWS Launch Wizard also automates the creation of a Single Tier PKI infrastructure or trust creation. One of the prime benefits of this solution is the simplicity in deploying a fully functional Active Directory environment in just a few clicks. You no longer need to do the undifferentiated heavy lifting required to deploy Active Directory. For more information, please refer to AWS Launch Wizard documentation.
Many Amazon Web Services (AWS) customers use Active Directory to centralize user authentication and authorization for a variety of applications and services. For these customers, Active Directory is a critical piece of their IT infrastructure.
AWS also offers Amazon Relational Database Service (Amazon RDS) for SQL Server. Amazon RDS enables you to prioritize application development by managing time-consuming database administration tasks including provisioning, backups, software patching, monitoring, and hardware scaling. If you require Windows authentication with Amazon RDS for SQL Server, Amazon RDS for SQL Server instances need to be integrated with AWS Managed Microsoft AD.
With the release of AWS Managed Microsoft AD cross-Region support, you only need one distinct AWS Managed Microsoft AD that spans multiple AWS Regions; this simplifies directory management and configuration. Additionally, it simplifies trusts between the AWS Managed Microsoft AD domain and your on-premises domain. Now, only a single trust between your on-premises domain and AWS Managed Microsoft AD domain is required, as compared to the previous pattern of only one AWS Managed Microsoft AD per Region—each of which would require a trust if you wanted to allow on-premises objects access to your AWS Managed Microsoft AD domain. Further, AWS Managed Microsoft AD cross-Region support provides an additional benefit when using your on-premises users and groups with Amazon RDS for SQL Server: You only need a single, one-way, outgoing trust between your multi-Region AWS Managed Microsoft AD and your on-premises domain.
As detailed in this post, to enable AWS Managed Microsoft AD cross-Region support, you create a new AWS Managed Microsoft AD and extend it to multiple Regions (as shown in Figure 1 below). Once you’ve extended your directory, you deploy an Amazon RDS SQL Server instance in each Region, integrating it to the same directory. Finally, you install SQL Server Management Studio (SSMS) on an instance joined to the AWS Managed Microsoft AD directory. You use that instance to connect to the RDS SQL Server instances using the same domain user account.
Figure 1: High level diagram of resources deployed in this post
The architecture in Figure 1 includes a network connection between the Regions. That connection isn’t required for the AWS Managed Microsoft AD to function. If you don’t require network connectivity between your regions, you can disregard the network link in the diagram. Since you will be using a single Amazon Elastic Compute Cloud (Amazon EC2) instance in one Region, the network connection is needed between Amazon VPCs in the two Regions to allow that instance to connect to a domain controller in each Region.
Prerequisites for AWS Managed Microsoft AD cross-Region Support
An AWS Managed Microsoft AD deployed in a Region of your choice. If you don’t have one already deployed, you can follow the instructions in Create Your AWS Managed Microsoft AD directory to create one. For this post, I recommend that you use us-east-1.
Extend your AWS Managed Microsoft AD to another Region
We’ve made the process to extend your directory to another Region straightforward. There is no cost to add another Region; you only pay for the resources for your directory running in the new Region. See here for additional information on pricing changes with new Regions. For example, in this post you will be extending your directory into the us-east-2 region. There will be an additional cost for two new domain controllers. Figure 3 shows the additional cost to extend the directory.
Let’s walk through the steps of setting up Windows Authentication with Amazon RDS for SQL Server instances in multiple Regions using a single cross-Region AWS Managed Microsoft AD.
To extend your directory to another Region:
In the AWS Directory Service console navigation pane, choose Directories.
Note: You should see a list of your AWS Managed Microsoft AD directories.
Choose the Directory ID of the directory you want to expand to another Region.
Go to the Directory details page. In the Multi-region replication section, select Add Region.
Figure 2: Directory details and new multi-Region replication pane
On the Add region page:
For Region to add, select the Region you want to extend your directory to.
For Subnets, select two unique subnets in the Amazon VPC that you selected in the preceding step.
Once you have everything to your liking, choose Add.
Figure 3: Add a Region
In the background, AWS is provisioning two new AWS managed domain controllers in the Region you selected. It could take up to 2 hours for your directory to become available in the Region.
Note: Your managed domain controllers in the home Region are fully functional during this process.
On the Directory details page, in Multi-Region replication, the status should be Active when the process has completed. Now you’re ready to deploy your Amazon RDS SQL Server instances.
Enable Amazon RDS for SQL Server
Integrating Amazon RDS into AWS Managed Microsoft AD is exactly the same process as it was before the cross-Region feature was released. This post goes through that original process with only one change, which is that you select the same directory ID for both Regions.
Create an Amazon RDS SQL Server instance in each Region using the same directory
The steps for creating an Amazon RDS SQL Server instance in each Region are the same. The following process will create the first instance. Once you’ve completed the process, you change the AWS Management Console Region to the Region you extended your directory to and repeat the process.
To create an Amazon RDS SQL Server instance:
In the AWS Managed Microsoft AD directory primary Region, go to the Amazon RDS console navigation pane and choose Create database.
Choose Microsoft SQL Server.
You can leave the default values, except for the following settings:
Under Settings select Master and Confirm password.
Under Connectivity, expand Additional connectivity configuration:
Choose Create new to create a new VPC security group.
Enter a name in New VPC security group name.
Select No preference for Availability Zone.
Enter 1433 for Database port.
Figure 4: Connectivity settings
Select the Enable Microsoft SQL Server Windows authentication check box and then choose Browse Directory.
Figure 5: Enable Microsoft SQL Server Windows authentication selected
Select your directory and select Choose.
Figure 6: Select a directory
Choose Create database.
Repeat these steps in your expanded Region. Note that the Directory ID will be the same for both Regions. You can complete the next section while your Amazon RDS SQL instances are provisioning.
Create an Active Directory user and group to delegate SQL administrative rights
The following steps walk you through the process of creating an Active Directory user and group for delegation. Following this process, you add the user to the group you just created and to the AWS Delegated Server Administrators group.
To create a user and group:
Log in to the domain-joined instance with a domain user account that has permissions to create Active Directory users and groups.
Choose Start, enter dsa.msc, and press Enter.
In Active Directory Users and Computers, right-click on the Users OU, select New, and then Group. The New Object – Group window pops up.
Fill in the Group name boxes with your choice of name.
In Active Directory Users and Computers, right-click on your Users OU and select New and then User. The New Object – User window pops up.
Fill in the boxes with your choice of information, and then choose Next.
Enter your choice of password and clear User must change password at next logon, then choose Next.
On the confirmation page, choose Finish.
Double-click on the user you just created. The user account properties window appears.
Select the Member of tab.
Choose Add.
Enter the name of the group that you previously created and choose Check Names. Next, enter AWS Delegated Server Administrators and choose Check Names again. If you do not receive any error, choose OK, and then OK again.
The Member of tab for the user should include the two groups you just added. Choose OK to close the properties page.
Delegate SQL Server permissions in each Region using the Active Directory group you just created
The following steps guide you through the process of modifying the Amazon RDS SQL security group, installing SQL Server Management Studio (SSMS), and delegating permission in SQL to your Active Directory group.
Modify the Amazon RDS SQL security group
In these next steps, you modify the security group you created with your Amazon RDS instances, allowing your Windows Server instance to connect to the Amazon RDS SQL Server instances over port 1433.
Note: If desired, you can disable IE Enhanced Security Configuration and download directly to the Windows Server instance using IE or any other browser, and skip to step 3.
RDP into your Windows Server instance and copy SSMS-Setup-ENU.exe to your RDP session.
Run the file on your Windows Server instance.
Choose Install.
Figure 8: Install SMMS
It might take a few minutes to install. When complete, choose Close.
Delegate permissions in SSMS
All of the following steps are performed on the Windows Server instance from Prerequisite 3. Log in to the Amazon RDS SQL instance using the SQL master user account. Next, create a SQL login for the Active Directory group you created previously and give it elevated permission to the Amazon RDS SQL instance.
To delegate permissions:
Start SMMS.
On the Connect to Server window, enter or select:
Server name – Your Amazon RDS SQL Server endpoint.
Authentication – Select SQL Server Authentication.
Login – Enter the master user name you used when you launched your Amazon RDS SQL instance. The default is admin.
Password – Enter the password for the master user name.
Choose Connect.
Figure 9: Connect to server
In SMMS, Choose New Query at the top of the window.
Figure 10: New query
In the query window, enter the following query. Replace <CORP\SQL-Admins> with the name of the group you created earlier.
CREATE LOGIN [<CORP\SQL-Admins>] FROM WINDOWS WITH DEFAULT_DATABASE = [master],
DEFAULT_LANGUAGE = [us_english];
Figure 11: Query SQL database
Choose Execute on the menu bar. You should see a Commands completed successfully message.
Figure 12: Commands completed successfully
Next, navigate to the Logins directory on the navigation page. Right-click on the group you added with the SQL command in step 5 and select Properties.
Figure 13: Open group properties
Select Server Roles and select the processadmin and setupadmin checkboxes. Then choose OK.
Figure 14: Configure server roles
You can log off from the instance. For the next steps, you log in to the instance using the user account you created previously.
Repeat these steps on the Amazon RDS SQL instance in the other Region.
Connect to the Amazon RDS SQL Server with the same Active Directory user in both Regions
All of the steps are performed on the Windows Server instance from Prerequisite 3. You must log in to the instance using the account you created earlier. You then log in to the Amazon RDS SQL instance using Windows authentication with that account.
Log in to the instance with the user account you created earlier.
Start SSMS.
On the Connect to Server window, enter or select:
Server name: Your Amazon RDS SQL Server endpoint.
Authentication: Select Windows Authentication.
Choose Connect.
Figure 15: Connect to server
You should be logged in to SSMS. If you aren’t logged in, make sure you added your user account to the group you created earlier and try again.
Repeat these steps using the other Amazon RDS SQL instance endpoint for the server name. You should be able to connect to both Amazon RDS SQL instances using the same user account.
Summary
In this post, you extended your AWS Managed Microsoft AD into a new Region. You then deployed Amazon RDS for SQL Server in multiple Regions attached to the same AWS Managed Microsoft AD directory. You then tested authentication to both Amazon RDS SQL instances using the same Active Directory user.
To learn more about using AWS Managed Microsoft AD or AD Connector, visit the AWS Directory Service documentation. For general information and pricing, see the AWS Directory Service home page. If you have comments about this blog post, submit a comment in the Comments section below. If you have implementation or troubleshooting questions, start a new thread on the AWS Directory Service forum or contact AWS Support.
When you connect an external identity provider (IdP) to AWS Single Sign-On (SSO) using Security Assertion Markup Language (SAML) 2.0 standard, you must create all users and groups into AWS SSO before you can make any assignments to AWS accounts or applications. If your IdP supports user and group provisioning by way of the System for Cross-Domain Identity Management (SCIM), we strongly recommend using SCIM to simplify ongoing lifecycle management for your users and groups in AWS SSO.
If your IdP doesn’t yet support automatic provisioning, you will need to create your users and groups manually in AWS SSO. Although manual creation of users and groups is the least complicated option to get started, it can be tedious and prone to errors.
In this post, we show you how to use a comma-separated values (CSV) file to bulk create users and groups in AWS SSO.
How it works
AWS SSO supports automatic provisioning of user and group information from an external IdP into AWS SSO using the SCIM protocol. For this solution, you use a PowerShell script to simulate a SCIM server, to provision users and groups from a CSV file into AWS SSO. You create and populate the CSV file with your user and group information that is then used by the PowerShell script. Next, on your Windows, Linux, or macOS system with PowerShell Core installed, you run the PowerShell script. The PowerShell script reads users and groups from the CSV file and then programmatically creates the users and groups in AWS SSO using your SCIM configuration for AWS SSO.
Assumptions
In this blog post, we assume the following:
You already have an AWS SSO-enabled account (free). For more information, see Enable AWS SSO.
You have the permissions needed to add users and groups in AWS SSO.
You’re using a Windows, MacOS, or Linux system with PowerShell Core installed.
If you’re not using a system with PowerShell Core installed, you’re using a Windows 7 or later system, with PowerShell 4.0 or later installed.
Note: This article was authored and the code tested on a Microsoft Windows Server 2019 system with PowerShell installed.
Enable automatic provisioning
In this step, you enable automatic provisioning in AWS SSO. You use the automatic provisioning endpoints for AWS SSO to connect and create users and groups in AWS SSO.
To enable automatic provisioning in AWS SSO
On the AWS SSO Console, go to the Single Sign-On page and then go to Settings.
Change the provisioning from Manual to SCIM by selecting Enable automatic provisioning.
Figure 1: Enable automatic provisioning
Copy the SCIM endpoint and the Access token (you can have up to two access token IDs). You use these values later.
Figure 2: Copy the SCIM endpoint and access token
Bulk create users and groups into AWS SSO
In this section, you create your users and groups from a CSV file into AWS SSO. To do this, you create a CSV file with your users’ profile information (for example: first name, last name, display name, and other values.). You also create a PowerShell script to connect to AWS SSO and create the users and groups from the CSV file in AWS SSO.
To bulk create your users from a CSV file
Create a file called csv-example-users.csv with the following column headings: firstName, lastName, userName, displayName, emailAddress, and memberOf.
Note: The memberOf column will include all the groups you want to add the user to in AWS SSO. If the group you plan to add a user to isn’t in AWS SSO, the script automatically creates the group for you. If you want to add a user to multiple groups, you can add the group names separated by semicolons in the memberOf column.
Populate the CSV file csv-example-users.csv with the users you want to create in AWS SSO.
Note: Before you populate the CSV file, take note of the existing users, groups, and group membership in AWS SSO. Make sure that none of the users or groups in the CSV file already exists in AWS SSO.
Note: For this to work, every user in the csv-example-users.csv must have a firstName, lastName, userName, displayName, and emailAddress value specified. If any of these values are missing, that user isn’t created. The userName and emailAddress values must not contain any spaces.
Figure 3: Create the CSV file and populate it with the users to create in AWS SSO
Next, create a create_users.ps1 file and copy the following PowerShell code to it. Use a text editor like Notepad or TextEdit to edit the create_users.ps1 file.
Replace <SCIMENDPOINT> with the SCIM endpoint value you copied earlier.
Replace <BEARERTOKEN> with the Access token value you copied earlier.
Replace <CSVLOCATION> with the location of your CSV file (for example, C:\Users\testuser\Downloads\csv-example-users.csv. Relative paths are also accepted).
#Input SCIM configuration and CSV file location
$Url = "<SCIMENDPOINT>"
$Bearertoken = "<BEARERTOKEN>"
$CSVfile = "<CSVLOCATION>"
$Headers = @{ Authorization = "Bearer $Bearertoken" }
#Get users from CSV file and store in variable
$Users = Import-Csv -Delimiter "," -Path "$CSVfile"
#Read groups in CSV and groups in AWS SSO
$Groups = $Users.memberOf -split ";"
$Groups = $Groups | Sort-Object -Unique | where {$_ -ne ""}
foreach($Group in $Groups){
$SSOgroup = @{
"displayName" = $Group.trim()
}
#Store group attribute in json format
$Groupjson = $SSOgroup | ConvertTo-Json
#Create groups in AWS SSO
try {
$Response = Invoke-RestMethod -ContentType application/json -Uri "$Url/Groups" -Method POST -Headers $Headers -Body $Groupjson -UseBasicParsing
Write-Host "Create group: The group $($Group) has been created successfully." -foregroundcolor green
}
catch
{
$ErrorMessage = $_.Exception.Message
if ($ErrorMessage -eq "The remote server returned an error: (409) Conflict.")
{
Write-Host "Error creating group: A group with the name $($Group) already exists." -foregroundcolor yellow
}
else
{
Write-Host "Error has occurred: $($ErrorMessage)" -foregroundcolor Red
}
}
}
#Loop through each user
foreach ($User in $Users)
{
#Get user attributes from each field
$SSOuser = @{
name = @{ familyName = $User.lastName.trim(); givenName = $User.firstName.trim() }
displayName = $User.displayName.trim()
userName = $User.userName
emails = @(@{ value = $User.emailAddress; type = "work"; primary = "true" })
active = "true"
}
#Store user attributes in json format
$Userjson = $SSOuser | ConvertTo-Json
#Create users in AWS SSO
try {
$Response = Invoke-RestMethod -ContentType application/json -Uri "$Url/Users" -Method POST -Headers $Headers -Body $Userjson -UseBasicParsing
Write-Host "Create user: The user $($User.userName) has been created successfully." -foregroundcolor green
}
catch
{
$ErrorMessage = $_.Exception.Message
if ($ErrorMessage -eq "The remote server returned an error: (409) Conflict.")
{
Write-Host "Error creating user: A user with the same username $($User.userName) already exist" -foregroundcolor yellow
}
else
{
Write-Host "Error has occurred: $($ErrorMessage)" -foregroundcolor Red
}
}
#Get user information
$UserName = $User.userName
$UserId = (Invoke-RestMethod -ContentType application/json -Uri "$Url/Users`?filter=userName%20eq%20%22$UserName%22" -Method GET -Headers $Headers).Resources.id
$Groups = $User.memberOf -split ";"
#Loop through each group and add user to group
foreach($Group in $Groups){
If (-not [string]::IsNullOrWhiteSpace($Group))
{
#Get the GroupName and GroupId
$GroupName = $Group.trim()
$GroupId = (Invoke-RestMethod -ContentType application/json -Uri "$Url/Groups`?filter=displayName%20eq%20%22$GroupName%22" -Method GET -Headers $Headers).Resources.id
#Store group membership in variable.
$AddUserToGroup = @{
Operations = @(@{ op = "add"; path = "members"; value = @(@{ value = $UserId })})
}
#Convert to json format
$AddUsertoGroupjson = $AddUserToGroup | ConvertTo-Json -Depth 4
#Add users to group in AWS SSO
try {
$Responses = Invoke-RestMethod -ContentType application/json -Uri "$Url/Groups/$GroupId" -Method PATCH -Headers $Headers -Body $AddUsertoGroupjson -UseBasicParsing
Write-Host "Add user to group: The user $($User.userName) has been added successfully to group $($GroupName)." -foregroundcolor green
}
catch
{
$ErrorMessage = $_.Exception.Message
if ($ErrorMessage -eq "The remote server returned an error: (409) Conflict.")
{
Write-Host "Error adding user to group: The user $($User.userName) is already added to group $($GroupName)." -foregroundcolor yellow
}
else
{
Write-Host "Error has occurred: $($ErrorMessage)" -foregroundcolor Red
}
}
}
}
}
Use Windows PowerShell to run the script create_users.ps1, as shown in the following figure.
Figure 4: Run PowerShell script to create users from CSV in AWS SSO
Use the AWS SSO console to verify that the users and groups were successfully created. In the AWS SSO console, select Users from the left menu, as shown in figure 5.
Figure 5: View the newly created users in AWS SSO console
Use the AWS SSO console to verify that the groups were successfully created. In the AWS SSO console, select Groups from the left menu, as shown in figure 6.
Figure 6: View the newly created groups in AWS SSO console
Your users, groups, and group memberships have been created in AWS SSO. You can now manage access for your identities in AWS SSO across your own applications, third-party applications (SaaS), and Amazon Web Services (AWS) environments.
How to run the PowerShell scripts on Linux and macOS
While this post focuses on running the PowerShell script on a Windows system. You can also run the PowerShell script on a Linux or macOS system that has PowerShell Core installed. You can then follow the steps in this post to create the required CSV files for creating a user and group and adding a user to a group. Then, on your Linux or macOS system, you can run the PowerShell script using the following command.
pwsh -File <Path to PowerShell Script>
Conclusion
In this post, we showed you how to programmatically create users and groups from a CSV file into AWS SSO. This solution isn’t a replacement for automatic provisioning. However, it can help you to quickly get up and running with AWS SSO by reducing the administration burden of manually creating users in AWS SSO.
If you have feedback about this post, submit comments in the Comments section below.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
Amazon Macie is a fully managed data security and data privacy service that uses machine learning and pattern matching to discover and protect your sensitive data in Amazon Web Services (AWS). It gives you the ability to automatically scan for sensitive data and get an inventory of your Amazon Simple Storage Service (Amazon S3) buckets. Macie also gives you the added ability to detect which buckets are public, unencrypted, and accessible from other AWS accounts.
In this post, we’ll walk through how to use Macie to detect sensitive data in Amazon DynamoDB tables by exporting the data to Amazon S3 so that Macie can scan the data. An example of why you would deploy a solution like this is if you have potentially sensitive data stored in DynamoDB tables. When we’re finished, you’ll have a solution that can set up on-demand or scheduled Macie discovery jobs to detect sensitive data exported from DynamoDB to S3.
Architecture
In figure 1, you can see an architectural diagram explaining the flow of the solution that you’ll be deploying.
Figure 1: Solution architecture
Here’s a brief overview of the steps that you’ll take to deploy the solution. Some steps you will do manually, while others will be handled by the provided AWS CloudFormation template. The following outline describes the steps taken to extract the data from DynamoDB and store it in S3, which allows Macie to run a discovery job against the data.
Deploy an AWS Lambda function to export DynamoDB data to S3.
Set up an Amazon EventBridge rule to schedule export of the DynamoDB data.
Create a Macie discovery job to discover sensitive data from the DynamoDB data export.
View the results of the Macie discovery job.
The goal is that when you finish, you have a solution that you can use to set up either on-demand or scheduled Macie discovery jobs to detect sensitive data that was exported from DynamoDB to S3.
Prerequisite: Enable Macie
If Macie hasn’t been enabled in your account, complete Step 1 in Getting started with Amazon Macie to enable Macie. Once you’ve enabled Macie, you can proceed with the deployment of the CloudFormation template.
Deploy the CloudFormation template
In this section, you start by deploying the CloudFormation template that will deploy all the resources needed for the solution. You can then review the output of the resources that have been deployed.
Sign in to the AWS Management Console and navigate to the CloudFormation console.
Choose Upload a template file, and then select the CloudFormation template that you downloaded in the previous step. Choose Next.
Figure 2 – Uploading the CloudFormation template to be deployed
For Stack Name, name your stack macie-blog, and then choose Next.
Figure 3: Naming your CloudFormation stack
For Configure stack options, keep the default values and choose Next.
At the bottom of the Review screen, select the I acknowledge that AWS CloudFormation might create IAM resources check box, and then choose Create stack.
Figure 4: Acknowledging that this CloudFormation template will create IAM roles
You should then see the following screen. It may take several minutes for the CloudFormation template to finish deploying.
Figure 5: CloudFormation stack creation in progress
View CloudFormation output
Once the CloudFormation template has been completely deployed, choose the Outputs tab, and you will see the following screen. Here you’ll find the names and URLs for all the AWS resources that are needed to complete the remainder of the solution.
Figure 6: Completed CloudFormation stack output
For easier reference, open a new browser tab to your AWS Management Console and leave this tab open. This will make it easier to quickly copy and paste the resource URLs as you navigate to different resources during this walkthrough.
Import DynamoDB data
In this section, we walk through importing the test dataset to DynamoDB. You first start by downloading the test CSV datasets, then upload those datasets to S3 and run the Lambda function that imports the data to DynamoDB. Finally, you review the data that was imported into DynamoDB.
Now that you’ve downloaded the test datasets, you’ll need to navigate to the data import S3 bucket and upload the data.
To upload the datasets to the S3 import bucket
Navigate to the CloudFormation Outputs tab, where you’ll find the bucket information.
Figure 7: S3 bucket output values for the CloudFormation stack
Copy the ImportS3BucketURL link and navigate to the URL.
Upload the two test CSV datasets, people.csv and accounts.csv, to your S3 bucket.
After the upload is complete, you should see the two CSV files in the S3 bucket. You’ll use these files as your test DynamoDB data.
Figure 8: Test S3 datasets in the S3 bucket
View the data import Lambda function
Now that you have your test data staged for loading, you’ll import it into DynamoDB by using a Lambda function that was deployed with the CloudFormation template. To start, navigate to the CloudFormation console and get the URL to the Lambda function that will handle the data import to DynamoDB, as shown in figure 9.
Figure 9: CloudFormation output information for the People DynamoDB table
To run the data import Lambda function
Copy the LambdaImportS3DataToDynamoURL link and navigate to the URL. You will see the Import-Data-To-DynamoDB Lambda function, as shown in figure 10.
Figure 10: The Lambda function that imports data to DynamoDB
Choose the Test button in the upper right-hand corner. In the dialog screen, for Event name, enter Test and replace the value with {}.
Your screen should now look as shown in figure 11. Choose Create.
Figure 11: Configuring a test event to manually run the Lambda function
Choose the Test button again in the upper right-hand corner. You should now see the Lambda function running, as shown in figure 12.
Figure 12: View of the Lambda function running
Once the Lambda function is finished running, you can expand the Details section. You should see a screen similar to the one in figure 13. When you see this screen, the test datasets have successfully been imported into the DynamoDB tables.
Figure 13: View of the data import Lambda function after it runs successfully
View the DynamoDB test dataset
Now that you have the datasets imported, you can look at the data in the console.
To view the test dataset
Navigate to the two DynamoDB tables. You can do this by getting the URL values from the CloudFormation Outputs tab. Figure 14 shows the URL for the accounts tables.
Figure 14: Output values for CloudFormation stack DynamoDB account tables
Figure 15 shows the URL for the people tables.
Figure 15: Output values for CloudFormation stack DynamoDB people tables
Copy the AccountsDynamoDBTableURL link value and navigate to it in the browser. Then choose the Items tab.
Figure 16: View of DynamoDB account-info-macie table data
You should now see a screen showing data similar to the screen in figure 16. This DynamoDB table stores the test account data that you will use to run a Macie discovery job against after the data has been exported to S3.
Navigate to the PeopleDynamoDBTableURL link that is in the CloudFormation output. Then choose the Items tab.
Figure 17: View of DynamoDB people table data
You should now see a screen showing data similar to the screen in figure 17. This DynamoDB table stores the test people data that you will use to run a Macie discovery job against after the data has been exported to S3.
Export DynamoDB data to S3
In the previous section, you set everything up and staged the data to DynamoDB. In this section, you will export data from DynamoDB to S3.
View the EventBridge rule
The EventBridge rule that was deployed earlier allows you to automatically schedule the export of DynamoDB data to S3. You will can export data in hours, in minutes, or in days. The purpose of the EventBridge rule is to allow you to set up an automated data pipeline from DynamoDB to S3. For demonstration purposes, you’ll run the Lambda function that the EventBridge rule uses manually, so that you can see the data be exported to S3 without having to wait.
To view the EventBridge rule
Navigate to the CloudFormation Outputs tab for the CloudFormation stack you deployed earlier.
Figure 18: CloudFormation output information for the EventBridge rule
Navigate to the EventBridgeRule link. You should see the following screen.
On this screen, you can see that we’ve set the event schedule to run every hour. The interval can be changed to fit your business needs. We have set it for 1 hour for demonstration purposes only. To make changes to the interval, you can choose the Edit button to make changes and then save the rule.
In the Target(s) section, we’ve configured a Lambda function named Export-DynamoDB-Data-To-S3 to handle the process of exporting data to the S3 bucket the Macie discovery job will run against. We will cover the Lambda function that handles the export of the data from DynamoDB next.
View the data export Lambda function
In this section, you’ll take a look at the Lambda function that handles the exporting of DynamoDB data to the S3 bucket that Macie will run its discovery job against.
To view the Lambda function
Navigate to the CloudFormation Outputs tab for the CloudFormation stack you deployed earlier.
Figure 20: CloudFormation output information for the Lambda function that exports DynamoDB data to S3
Copy the link value for LambdaExportDynamoDBDataToS3URL and navigate to the URL in your browser. You should see the Python code that will handle the exporting of data to S3. The code has been commented so that you can easily follow it and refactor it for your needs.
Scroll to the Environment variables section.
Figure 21: Environment variables used by the Lambda function
You will see two environment variables:
bucket_to_export_to – This environment variable is used by the function as the S3 bucket location to save the DynamoDB data to. This is the bucket that the Macie discovery will run against.
dynamo_db_tables – This environment variable is a comma-delimited list of DynamoDB tables that will be read and have data exported to S3. If there was another table that you wanted to export data from, you would simply add it to the comma-delimited list and it would be part of the export.
Export DynamoDB data
In this section, you will manually run the Lambda function to export the DynamoDB tables data to S3. As stated previously, you would normally allow the EventBridge rule to handle the automated export of the data to S3. In order to see the export in action, you’re going to manually run the function.
To run the export Lambda function
In the console, scroll back to the top of the screen and choose the Test button.
Name the test dynamoDBExportTest, and for the test data create an empty JSON object “{}” as shown in figure 22.
Figure 22: Configuring a test event to manually test the data export Lambda function
Choose Create.
Choose the Test button again to run the Lambda function to export the DynamoDB data to S3.
Figure 23: View of the screen where you run the Lambda function to export data
It could take about one minute to export the data from DynamoDB to S3. Once the Lambda function exports the data, you should see a screen similar to the following one.
Figure 24: The result after you successfully run the data export Lambda function
View the exported DynamoDB data
Now that the DynamoDB data has been exported for Macie to run discovery jobs against, you can navigate to S3 to verify that the files exported to the bucket.
To view the data, navigate to the CloudFormation stack Output tab. Find the ExportS3BucketURL, shown in figure 25, and navigate to the link.
Figure 25: CloudFormation output information for the S3 buckets that the DynamoDB data was exported to
You should then see two different JSON files for the two DynamoDB tables that data was exported from, as shown in figure 26.
Figure 26: View of S3 objects that were exported to S3
This is the file naming convention that’s used for the files:
Next, you’ll create a Macie discovery job to run against the files in this S3 bucket to discover sensitive data.
Create the Macie discovery job
In this section, you’ll create a Macie discovery job and view the results after the job has finished running.
To create the discovery job
In the AWS Management Console, navigate to Macie. In the left-hand menu, choose Jobs.
Figure 27: Navigation menu to Macie discovery jobs
Choose the Create job button.
Figure 28: Macie discovery job list screen
Using the Bucket Name filter, search for the S3 bucket that the DynamoDB data was exported to. This can be found in the CloudFormation stack output, as shown in figure 29.
Figure 29: CloudFormation stack output
Select the value you see for ExportS3BucketName, as shown in figure 30.
Note: The value you see for your bucket name will be slightly different, based on the random characters added to the end of the bucket name generated by CloudFormation.
Figure 30: Selecting the S3 bucket to include in the Macie discovery job
Once you’ve found the S3 bucket, select the check box next to it, and then choose Next.
On the Review S3 Buckets screen, if you’re satisfied with the selected buckets, choose Next.
Following are some important options when setting up Macie data discovery jobs.
Scheduling You have the following scheduling options for the data discovery job:
Daily
Weekly
Monthly
Data Sampling This allows you to randomly sample a percentage of the data that the Macie discovery job will run against.
Object criteria This enables you to target objects based on certain metadata values. The values are:
Tags – Target objects with certain tags.
Last modified – Target objects based on when they were last modified.
File extensions – Target objects based on file extensions.
Object size – Target objects based on the file size.
You can include or exclude objects based on these object criteria filters.
Set the discovery job scope
For demonstration purposes, this will be a one-time discovery job.
To set the discovery job scope
On the Scope page that appears after you create the job, set the following options for the job scope:
Select the One-time job option.
Leave Sampling depth set to 100%, and choose Next.
Figure 31: Selecting the objects that should be in scope for this discovery job
On the Custom data identifiers screen, select account_number, and then choose Next.With the custom identifier, you can create custom business logic to look for certain patterns in files stored in S3. In this example, the job generates a finding for any file that contains data with the following format:
Account Number Format: Starts with “XYZ-” followed by 11 numbers
The logic to create a custom data identifier can be found in the CloudFormation template.
Figure 32: Custom data identifiers
Give your discovery job the name dynamodb-macie-discovery-job. For Description, enter Discovery job to detect sensitive data exported from DynamoDB, and choose Next.
Figure 33: Giving the Macie discovery job a name and description
You will then see the Review and create screen, as shown in figure 34.
Figure 34: The Macie discovery job review screen
Note: Macie must have proper permissions to decrypt objects that are part of the Macie discovery job. The CloudFormation template that you deployed during the initial setup has already deployed an AWS Key Management Service (AWS KMS) key with the proper permissions.
For this proof of concept you won’t store the results, so you can select the check box next to Override this requirement. If you wanted to store detailed results of the discovery job long term, you would configure a repository for data discovery results. To view detailed steps for setting this up, see Storing and retaining discovery results with Amazon Macie.
Submit the discovery job
Next, you can submit the discovery job. On the Review and create screen, choose the Submit button to start the discovery job. You should see a screen similar to the following.
Figure 35: A Macie discovery job run that is in progress
The amount of data that is being scanned dictates how long the job will take to run. You can choose the Refresh button at the top of the screen to see the updated status of the job. This job, based on the size of the test dataset, will take about seven minutes to complete.
Review the job results
Now that the Macie discovery job has run, you can review the results to see what sensitive data was discovered in the data exported from DynamoDB.
You should see the following screen once the job has successfully run.
Figure 36: View of the completed Macie discovery job
On the right, you should see another pane with more information related to the discovery job. The pane should look like the following screen.
Figure 37: Summary showing which S3 bucket the discovery job ran against and start and complete time
Note: If you don’t see this pane, choose on the discovery job to have this information displayed.
To review the job results
On the page for the discovery job, in the Show Results list, select Show findings.
Figure 38: Option to view discovery job findings
The Findings screen appears, as follows.
Figure 39: Viewing the list of findings generated by the Macie discovery job
The discovery job that you ran has two different “High Severity” finding types:
SensitiveData:S3Object/Personal – The object contains personal information, such as full names or identification numbers.
SensitiveData:S3Object/Multiple – The object contains more than one type of sensitive data.
Choose the SensitiveData:S3Object/Personal finding type, and you will see an information pane appear to the right, as shown in figure 40.Some of the key information that you can find here:
Severity – What the severity of the finding is: Low, Medium, or High. Resource – The S3 bucket where the S3 object exists that caused the finding to be generated. Region – The Region where the S3 bucket exists.
Figure 40: Viewing the severity of the discovery job finding
Since the finding is based on the detection of personal information in the S3 object, you get the number of times and type of personal data that was discovered, as shown in figure 41.
Figure 41: Viewing the number of social security numbers that were discovered in the finding
Here you can see that 10 names were detected in the data that you exported from the DynamoDB table. Occurrences of name equals 10 line ranges, which tells you that the names were found on 10 different lines in the file. If you choose the 10 line ranges link, you are given the starting line and column in the document where the name was discovered.
The S3 object that triggered the finding is displayed in the Resource affected section, as shown in figure 42.
Figure 42: The S3 object that generated the Macie finding
Now that you know which S3 object contains the sensitive data, you can investigate further to take appropriate action to protect the data.
View the Macie finding details
In this section, you will walk through how to read and download the objects related to the Macie discovery job.
To download and view the S3 object that contains the finding
In the Overview section of the finding details, select the value for the Resource link. You will then be taken to the object in the S3 bucket.
Figure 43: Viewing the S3 bucket where the object is located that generated the Macie finding
You can then download the S3 object from the S3 bucket to view the file content and further investigate the file content for sensitive data. Select the check box next to the S3 object, and choose the Download button at the top of the screen.Next, we will look at the SensitiveData:S3Object/Multiple finding type that was generated. This finding type lets us know that there are multiple types of potentially sensitive data related to an object stored in S3.
In the left navigation menu, navigate back to the Jobs menu.
Choose the job that you created in the previous steps. In the Show Results list, select Show Findings.
Select the SensitiveData:S3Object/Multiple finding type. An information pane appears to the right. As with the previous finding, you will see the severity, Region, S3 bucket location, and other relevant information about the finding. For this finding, we will focus on the Custom data identifiers and Personal info sections.
Figure 44: Details about the sensitive data that was discovered by the Macie discovery job
Here you can see that the discovery job found 10 names on 10 different lines in the file. Also, you can see that 10 account numbers were discovered on 10 different lines in the file, based on the custom identifier that was included as part of the discovery job.
This finding demonstrates how you can use the built-in Macie identifiers, such as names, and also include custom business logic based on your organization’s needs by using Macie custom data identifiers.
To view the data and investigate further, follow the same steps as in the previous finding you investigated.
Navigate to the top of the screen and in the Overview section, locate the Resource.
Figure 45: Viewing the S3 bucket where the object is located that generated the Macie finding
Choose Resource, which will take you to the S3 object to download. You can now view the contents of the file and investigate further.
You’ve now created a Macie discovery job to scan for sensitive data stored in an S3 bucket that originated in DynamoDB. You can also automate this solution further by using EventBridge rules to detect Macie findings to take actions against those objects with sensitive data.
Solution cleanup
In order to clean up the solution that you just deployed, complete the following steps. Note that you need to do these steps to stop data from being exported from DynamoDB to S3 every 1 hour.
To perform cleanup
Navigate to the S3 buckets used to import and export data. You can find the bucket names in the CloudFormation Outputs tab in the console, as shown in figure 7 and figure 25.
After you’ve navigated to each of the buckets, delete all objects from the bucket.
Navigate to the CloudFormation console, and then delete the CloudFormation stack named macie-blog. After the stack is deleted, the solution will no longer be deployed in your AWS account.
Summary
After deploying the solution, we hope you have a better understanding of how you can use Macie to detect sensitive from other data sources, such as DynamoDB, as outlined in this post. The following are links to resources that you can use to further expand your knowledge of Amazon Macie capabilities and features.
As organizations scale up their Amazon Web Services (AWS) presence, they are faced with the challenge of administering user identities and controlling access across multiple accounts and Regions. As this presence grows, managing user access to cloud resources such as Amazon Elastic Compute Cloud (Amazon EC2) becomes increasingly complex. AWS Directory Service for Microsoft Active Directory (also known as an AWS Managed Microsoft AD) makes it easier and more cost-effective for you to manage this complexity. AWS Managed Microsoft AD is built on highly available, AWS managed infrastructure. Each directory is deployed across multiple Availability Zones, and monitoring automatically detects and replaces domain controllers that fail. In addition, data replication and automated daily snapshots are configured for you. You don’t have to install software, and AWS handles all patching and software updates. AWS Managed Microsoft AD enables you to leverage your existing on-premises user credentials to access cloud resources such as the AWS Management Console and EC2 instances.
This blog post describes how EC2 resources launched across multiple AWS accounts and Regions can automatically domain-join a centralized AWS Managed Microsoft AD. The solution we describe in this post is implemented for both Windows and Linux instances. Removal of Computer objects from Active Directory upon instance termination is also implemented. The solution uses Amazon DynamoDB to centrally store account and directory information in a central security account. We also provide AWS CloudFormation templates and platform-specific domain join scripts for you to use with AWS Lambda as a quick start solution.
Architecture
The following diagram shows the domain-join process for EC2 instances across multiple accounts and Regions using AWS Managed Microsoft AD.
Figure 1: EC2 domain join architecture
The event flow works as follows:
An EC2 instance is launched in a peered virtual private cloud (VPC) of a workload or security account. VPCs that are hosting EC2 instances need to be peered with the VPC that contains AWS Managed Microsoft AD to enable network connectivity with Active Directory.
The CloudWatch event is forwarded to a regional CloudWatch event bus in the security account.
If the CloudWatch event bus is in the same Region as AWS Managed Microsoft AD, it delivers the event to an Amazon Simple Queue Service (Amazon SQS) queue, referred to as the domain-join queue in this post.
If the CloudWatch event bus is in a different Region from AWS Managed Microsoft AD, it delivers the event to an Amazon Simple Notification Service (Amazon SNS) topic. The event is then delivered to the domain-join queue described in step 4, through the Amazon SNS topic subscription.
Messages in the domain-join queue are held for five minutes to allow for EC2 instances to stabilize after they reach the “running” state. This delay allows time for installation of additional software components and agents through the use of EC2 user data and AWS Systems Manager Distributor.
After the holding period is over, messages in the domain-join queue invoke the AWS AD Join/Leave Lambda function. The Lambda function does the following:
Retrieves the AWS account ID that originated the event from the message and retrieves account-specific configurations from a DynamoDB table. This configuration identifies AWS Managed Microsoft AD domain controller IPs, credentials required to perform EC2 domain join, and an AWS Identity and Access Management (IAM) role that can be assumed by the Lambda function to invoke AWS Systems Manager Run Command.
Retrieves EC2 instance information, such as the instance state, platform, and tags, and validates the instance state.
Retrieves platform-specific domain-join scripts that are deployed with the Lambda function’s code bundle, and configures invocation of those scripts by using data read from the DynamoDB table (bash script for Linux instances and PowerShell script for Windows instances).
Uses AWS Systems Manager Run Command to invoke the domain-join script on the instance. Run Command enables you to remotely and securely manage the configuration of your managed instances.
The domain-join script runs on the instance. It uses script parameters and instance attributes to configure the instance and perform the domain join. The adGroupName tag value is used to configure the Active Directory user group that will have permissions to log in to the instance. The instance is rebooted to complete the domain join process. Various software components are installed on the instance when the script runs. For the Linux instance, sssd, realmd, krb5, samba-common, adcli, unzip, and packageit are installed. For the Windows instance, the RDS-RD-Server feature is installed.
Removal of EC2 instances from AWS Managed Microsoft AD upon instance termination follows a similar sequence of steps. Each instance that is domain joined creates an Active Directory domain object under the “Computer” hierarchy. This domain object needs to be removed upon instance termination so that a new instance that uses the same private IP address in the subnet (at a future time) can successfully domain join and enable instance access with Active Directory credentials. Removal of the Active Directory Computer object is done by running the leaveDomaini.ps1 script (included with this blog) through Run Command on the Active Directory Tools instance identified in Figure 1.
Prerequisites and setup
To build the solution outlined in this post, you need:
AD Tools. To install AD Tools and use it to create the required users:
Launch a Windows EC2 instance in the same account and Region, and domain-join it with the directory you created in the previous step. Log in to the instance through Remote Desktop Protocol (RDP) and install AD Tools as described in Installing the Active Directory Administration Tools.
After the AD Tools are installed, launch the AD Users & Computers application to create domain users, and assign those users to an Active Directory security group (for example, my_UserGroup) that has permission to access domain-joined instances.
Create a least-privileged user for performing domain joins as described in Delegate Directory Join Privileges for AWS Managed Microsoft AD. The identity of this user is stored in the DynamoDB table and read by the AD Join Lambda function to invoke Active Directory join scripts.
Store the password for the least-privileged user in an encrypted Systems Manager parameter. The password for this user is stored in the secure string System Manager parameter and read by the AD Join Lambda function at runtime while processing Amazon SQS messages.
Assign a unique tag key and value to identify the AD Tools instance. This instance will be invoked by the Lambda function to delete Computer objects from Active Directory upon termination of domain-joined instances.
All VPCs that are hosting EC2 instances to be domain joined must be peered with the VPC that hosts the relevant AWS Managed Microsoft AD. Alternatively, AWS Transit Gateway could be used to establish this connectivity.
In addition to having network connectivity to the AWS Managed Microsoft AD domain controllers, domain join scripts that run on EC2 instances must be able to resolve relevant Active Directory resource records. In this solution, we leverage Amazon Route 53 Outbound Resolver to forward DNS queries to the AWS Managed Microsoft AD DNS servers, while still preserving the default DNS capabilities that are available to the VPC. Learn more about deploying Route 53 Outbound Resolver and resolver rules to resolve your directory DNS name to DNS IPs.
Each domain-join EC2 instance must have a Systems Manager Agent (SSM Agent) installed and an IAM role that provides equivalent permissions as provided by the AmazonEC2RoleforSSM built-in policy. The SSM Agent is used to allow domain-join scripts to run automatically. See Working with SSM Agent for more information on installing and configuring SSM Agents on EC2 instances.
Solution deployment
The steps in this section deploy AD Join solution components by using the AWS CloudFormation service.
The CloudFormation template provided with this solution (mad_auto_join_leave.json) deploys resources that are identified in the security account’s AWS Region that hosts AWS Managed Microsoft AD (the top left quadrant highlighted in Figure 1). The template deploys a DynamoDB resource with 5 read and 5 write capacity units. This should be adjusted to match your usage. DynamoDB also provides the ability to auto-scale these capacities. You will need to create and deploy additional CloudFormation stacks for cross-account, cross-Region scenarios.
To deploy the solution
Create a versioned Amazon Simple Storage Service (Amazon S3) bucket to store a zip file (for example, adJoinCode.zip) that contains Python Lambda code and domain join/leave bash and PowerShell scripts. Upload the source code zip file to an S3 bucket and find the version associated with the object.
Navigate to the AWS CloudFormation console. Choose the appropriate AWS Region, and then choose Create Stack. Select With new resources.
Choose Upload a template file (for this solution, mad_auto_join_leave.json), select the CloudFormation stack file, and then choose Next.
Enter the stack name and values for the other parameters, and then choose Next.
Figure 2: Defining the stack name and parameters
The parameters are defined as follows:
S3CodeBucket: The name of the S3 bucket that holds the Lambda code zip file object.
adJoinLambdaCodeFileName: The name of the Lambda code zip file that includes Lambda Python code, bash, and Powershell scripts.
adJoinLambdaCodeVersion: The S3 Version ID of the uploaded Lambda code zip file.
DynamoDBTableName: The name of the DynamoDB table that will hold account configuration information.
CreateDynamoDBTable: The flag that indicates whether to create a new DynamoDB table or use an existing table.
ADToolsHostTagKey: The tag key of the Windows EC2 instance that has AD Tools installed and that will be used for removal of Active Directory Computer objects upon instance termination.
ADToolsHostTagValue: The tag value for the key identified by the ADToolsHostTagKey parameter.
Acknowledge creation of AWS resources and choose to continue to deploy AWS resources through AWS CloudFormation.The CloudFormation stack creation process is initiated, and after a few minutes, upon completion, the stack status is marked as CREATE_COMPLETE. The following resources are created when the CloudFormation stack deploys successfully:
An AD Join Lambda function with associated scripts and IAM role.
A CloudWatch Events rule to detect the “running” and “terminated” states for EC2 instances.
An SQS event queue to hold the EC2 instance “running” and “terminated” events.
CloudWatch event mapping to the SQS event queue and further to the Lambda function.
A DynamoDB table to hold the account configuration (if you chose this option).
The DynamoDB table hosts account-level configurations. Account-specific configuration is required for an instance from a given account to join the Active Directory domain. Each DynamoDB item contains the account-specific configuration shown in the following table. Storing account-level information in the DynamoDB table provides the ability to use multiple AWS Managed Microsoft AD directories and group various accounts accordingly. Additional account configurations can also be stored in this table for implementation of various centralized security services (instance inspection, patch management, and so on).
Attribute
Description
accountId
AWS account number
adJoinUserName
User ID with AD Join permissions
adJoinUserPwParam
Encrypted Systems Manager parameter containing the AD Join user’s password
dnsIP1
Domain controller 1 IP address2
dnsIP2
Domain controller 2 IP address
assumeRoleARN
Amazon Resource Name (ARN) of the role assumed by the AD Join Lambda function
Following is an example of how you could insert an item (row) in a DynamoDB table for an account.
(Update with your own values as appropriate for your environment.)
In the preceding example, adJoinLambdaRole is assumed by the AD Join Lambda function (if needed) to establish cross-account access using AWS Security Token Service (AWS STS). The role needs to provide sufficient privileges for the AD Join Lambda function to retrieve instance information and run cross-account Systems Manager commands.
adJoinUserName identifies a user with the minimum privileges to do the domain join; you created this user in the prerequisite steps.
adJoinUserPwParam identifies the name of the encrypted Systems Manager parameter that stores the password for the AD Join user. You created this parameter in the prerequisite steps.
Solution test
After you successfully deploy the solution using the steps in the previous section, the next step is to test the deployed solution.
To test the solution
Navigate to the AWS EC2 console and launch a Linux instance. Launch the instance in a public subnet of the available VPC.
Choose an IAM role that gives at least AmazonEC2RoleforSSM permissions to the instance.
Add an adGroupName tag with the value that identifies the name of the Active Directory security group whose members should have access to the instance.
Make sure that the security group associated with your instance has permissions for your IP address to log in to the instance by using the Secure Shell (SSH) protocol.
Wait for the instance to launch and perform the Active Directory domain join. You can navigate to the AWS SQS console and observe a delayed message that represents the CloudWatch instance “running” event. This message is processed after five minutes; after that you can observe the Lambda function’s message processing log in CloudWatch logs.
Log in to the instance with Active Directory user credentials. This user must be the member of the Active Directory security group identified by the adGroupName tag value. Following is an example login command.
Similarly, launch a Windows EC2 instance to validate the Active Directory domain join by using Remote Desktop Protocol (RDP).
Terminate domain-joined instances. Log in to the AD Tools instance to validate that the Active Directory Computer object that represents the instance is deleted.
The AD Join Lambda function invokes Systems Manager commands to deliver and run domain join scripts on the EC2 instances. The AWS-RunPowerShellScript command is used for Microsoft Windows instances, and the AWS-RunShellScript command is used for Linux instances. Systems Manager command parameters and execution status can be observed in the Systems Manager Run Command console.
The AD user used to perform the domain join is a least-privileged user, as described in Delegate Directory Join Privileges for AWS Managed Microsoft AD. The password for this user is passed to instances by way of SSM Run Commands, as described above. The password is visible in the SSM Command history log and in the domain join scripts run on the instance. Alternatively, all script parameters can be read locally on the instance through the “adjoin” encrypted SSM parameter. Refer to the domain join scripts for details of the “adjoin” SSM parameter.
Additional information
Directory sharing
AWS Managed Microsoft AD can be shared with other AWS accounts in the same Region. Learn how to use this feature and seamlessly domain join Microsoft Windows EC2 instances and Linux instances.
autoadjoin tag
Launching EC2 instances with an autoadjoin tag key with a “false” value excludes the instance from the automated Active Directory join process. You might want to do this in scenarios where you want to install additional agent software before or after the Active Directory join process. You can invoke domain join scripts (bash or PowerShell) by using user data or other means. However, you’ll need to reboot the instance and re-run scripts to complete the domain join process.
Summary
In this blog post, we demonstrated how you could automate the Active Directory domain join process for EC2 instances to AWS Managed Microsoft AD across multiple accounts and Regions, and also centrally manage this configuration by using AWS DynamoDB. By adopting this model, administrators can centrally manage Active Directory–aware applications and resources across their accounts.
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
Data is a crucial part of every business and is used for strategic decision making at all levels of an organization. To extract value from their data more quickly, Amazon Web Services (AWS) customers are building automated data pipelines—from data ingestion to transformation and analytics. As part of this process, my customers often ask how to prevent sensitive data, such as personally identifiable information, from being ingested into data lakes when it’s not needed. They highlight that this challenge is compounded when ingesting unstructured data—such as files from process reporting, text files from chat transcripts, and emails. They also mention that identifying sensitive data inadvertently stored in structured data fields—such as in a comment field stored in a database—is also a challenge.
In this post, I show you how to integrate Amazon Macie as part of the data ingestion step in your data pipeline. This solution provides an additional checkpoint that sensitive data has been appropriately redacted or tokenized prior to ingestion. Macie is a fully managed data security and privacy service that uses machine learning and pattern matching to discover sensitive data in AWS.
When Macie discovers sensitive data, the solution notifies an administrator to review the data and decide whether to allow the data pipeline to continue ingesting the objects. If allowed, the objects will be tagged with an Amazon Simple Storage Service (Amazon S3) object tag to identify that sensitive data was found in the object before progressing to the next stage of the pipeline.
This combination of automation and manual review helps reduce the risk that sensitive data—such as personally identifiable information—will be ingested into a data lake. This solution can be extended to fit your use case and workflows. For example, you can define custom data identifiers as part of your scans, add additional validation steps, create Macie suppression rules to archive findings automatically, or only request manual approvals for findings that meet certain criteria (such as high severity findings).
Solution overview
Many of my customers are building serverless data lakes with Amazon S3 as the primary data store. Their data pipelines commonly use different S3 buckets at each stage of the pipeline. I refer to the S3 bucket for the first stage of ingestion as the raw data bucket. A typical pipeline might have separate buckets for raw, curated, and processed data representing different stages as part of their data analytics pipeline.
Typically, customers will perform validation and clean their data before moving it to a raw data zone. This solution adds validation steps to that pipeline after preliminary quality checks and data cleaning is performed, noted in blue (in layer 3) of Figure 1. The layers outlined in the pipeline are:
Ingestion – Brings data into the data lake.
Storage – Provides durable, scalable, and secure components to store the data—typically using S3 buckets.
Processing – Transforms data into a consumable state through data validation, cleanup, normalization, transformation, and enrichment. This processing layer is where the additional validation steps are added to identify instances of sensitive data that haven’t been appropriately redacted or tokenized prior to consumption.
Consumption – Provides tools to gain insights from the data in the data lake.
Figure 1: Data pipeline with sensitive data scan
The application runs on a scheduled basis (four times a day, every 6 hours by default) to process data that is added to the raw data S3 bucket. You can customize the application to perform a sensitive data discovery scan during any stage of the pipeline. Because most customers do their extract, transform, and load (ETL) daily, the application scans for sensitive data on a scheduled basis before any crawler jobs run to catalog the data and after typical validation and data redaction or tokenization processes complete.
You can expect that this additional validation will add 5–10 minutes to your pipeline execution at a minimum. The validation processing time will scale linearly based on object size, but there is a start-up time per job that is constant.
If sensitive data is found in the objects, an email is sent to the designated administrator requesting an approval decision, which they indicate by selecting the link corresponding to their decision to approve or deny the next step. In most cases, the reviewer will choose to adjust the sensitive data cleanup processes to remove the sensitive data, deny the progression of the files, and re-ingest the files in the pipeline.
Additional considerations for deploying this application for regular use are discussed at the end of the blog post.
Application components
The following resources are created as part of the application:
S3 buckets store data in various stages of processing: A raw data bucket for uploading objects for the data pipeline, a scanning bucket where objects are scanned for sensitive data, a manual review bucket holding objects where sensitive data was discovered, and a scanned data bucket for starting the next ingestion step of the data pipeline.
Lambda functions execute the logic to run the sensitive data scans and workflow.
Note: the application uses various AWS services, and there are costs associated with these resources after the Free Tier usage. See AWS Pricing for details. The primary drivers of the solution cost will be the amount of data ingested through the pipeline, both for Amazon S3 storage and data processed for sensitive data discovery with Macie.
The architecture of the application is shown in Figure 2 and described in the text that follows.
Figure 2: Application architecture and logic
Application logic
Objects are uploaded to the raw data S3 bucket as part of the data ingestion process.
A scheduled EventBridge rule runs the sensitive data scan Step Functions workflow.
triggerMacieScan Lambda function moves objects from the raw data S3 bucket to the scan stage S3 bucket.
triggerMacieScan Lambda function creates a Macie sensitive data discovery job on the scan stage S3 bucket.
checkMacieStatus Lambda function checks the status of the Macie sensitive data discovery job.
isMacieStatusCompleteChoiceStep Functions Choice state checks whether the Macie sensitive data discovery job is complete.
If yes, the getMacieFindingsCount Lambda function runs.
getMacieFindingsCount Lambda function counts all of the findings from the Macie sensitive data discovery job.
isSensitiveDataFound Step Functions Choice state checks whether sensitive data was found in the Macie sensitive data discovery job.
If there was sensitive data discovered, run the triggerManualApproval Lambda function.
If there was no sensitive data discovered, run the moveAllScanStageS3Files Lambda function.
moveAllScanStageS3Files Lambda function moves all of the objects from the scan stage S3 bucket to the scanned data S3 bucket.
triggerManualApproval Lambda function tags and moves objects with sensitive data discovered to the manual review S3 bucket, and moves objects with no sensitive data discovered to the scanned data S3 bucket. The function then sends a notification to the ApprovalRequestNotification Amazon SNS topic as a notification that manual review is required.
Email is sent to the email address that’s subscribed to the ApprovalRequestNotification Amazon SNS topic (from the application deployment template) for the manual review user with the option to Approve or Deny pipeline ingestion for these objects.
Manual review user assesses the objects with sensitive data in the manual review S3 bucket and selects the Approve or Deny links in the email.
The decision request is sent from the Amazon API Gateway to the receiveApprovalDecision Lambda function.
manualApprovalChoice Step Functions Choice state checks the decision from the manual review user.
If denied, run the deleteManualReviewS3Files Lambda function.
If approved, run the moveToScannedDataS3Files Lambda function.
deleteManualReviewS3Files Lambda function deletes the objects from the manual review S3 bucket.
moveToScannedDataS3Files Lambda function moves the objects from the manual review S3 bucket to the scanned data S3 bucket.
The next step of the automated data pipeline will begin with the objects in the scanned data S3 bucket.
Prerequisites
For this application, you need the following prerequisites:
You can use AWS Cloud9 to deploy the application. AWS Cloud9 includes the AWS CLI and AWS SAM CLI to simplify setting up your development environment.
Deploy the application with AWS SAM CLI
You can deploy this application using the AWS SAM CLI. AWS SAM uses AWS CloudFormation as the underlying deployment mechanism. AWS SAM is an open-source framework that you can use to build serverless applications on AWS.
To deploy the application
Initialize the serverless application using the AWS SAM CLI from the GitHub project in the aws-samples repository. This will clone the project locally which includes the source code for the Lambda functions, Step Functions state machine definition file, and the AWS SAM template. On the command line, run the following:
sam init --location gh: aws-samples/amazonmacie-datapipeline-scan
Alternatively, you can clone the Github project directly.
Deploy your application to your AWS account. On the command line, run the following:
sam deploy --guided
Complete the prompts during the guided interactive deployment. The first deployment prompt is shown in the following example.
Configuring SAM deploy
======================
Looking for config file [samconfig.toml] : Found
Reading default arguments : Success
Setting default arguments for 'sam deploy'
=========================================
Stack Name [maciepipelinescan]:
Settings:
Stack Name – Name of the CloudFormation stack to be created.
AWS Region – Region—for example, us-west-2, eu-west-1, ap-southeast-1—to deploy the application to. This application was tested in the us-west-2 and ap-southeast-1 Regions. Before selecting a Region, verify that the services you need are available in those Regions (for example, Macie and Step Functions).
Parameter StepFunctionName – Name of the Step Functions state machine to be created—for example, maciepipelinescanstatemachine).
Parameter BucketNamePrefix – Prefix to apply to the S3 buckets to be created (S3 bucket names are globally unique, so choosing a random prefix helps ensure uniqueness).
Parameter ApprovalEmailDestination – Email address to receive the manual review notification.
Parameter EnableMacie – Whether you need Macie enabled in your account or Region. You can select yes or no; select yes if you need Macie to be enabled for you as part of this template, select no, if you already have Macie enabled.
Confirm changes and provide approval for AWS SAM CLI to deploy the resources to your AWS account by responding y to prompts, as shown in the following example. You can accept the defaults for the SAM configuration file and SAM configuration environment prompts.
#Shows you resources changes to be deployed and require a 'Y' to initiate deploy
Confirm changes before deploy [y/N]: y
#SAM needs permission to be able to create roles to connect to the resources in your template
Allow SAM CLI IAM role creation [Y/n]: y
ReceiveApprovalDecisionAPI may not have authorization defined, Is this okay? [y/N]: y
ReceiveApprovalDecisionAPI may not have authorization defined, Is this okay? [y/N]: y
Save arguments to configuration file [Y/n]: y
SAM configuration file [samconfig.toml]:
SAM configuration environment [default]:
Note: This application deploys an Amazon API Gateway with two REST API resources without authorization defined to receive the decision from the manual review step. You will be prompted to accept each resource without authorization. A token (Step Functions taskToken) is used to authenticate the requests.
This creates an AWS CloudFormation changeset. Once the changeset creation is complete, you must provide a final confirmation of y to Deploy the changeset? [y/N] when prompted as shown in the following example.
Changeset created successfully. arn:aws:cloudformation:ap-southeast-1:XXXXXXXXXXXX:changeSet/samcli-deploy1605213119/db681961-3635-4305-b1c7-dcc754c7XXXX
Previewing CloudFormation changeset before deployment
======================================================
Deploy this changeset? [y/N]:
Your application is deployed to your account using AWS CloudFormation. You can track the deployment events in the command prompt or via the AWS CloudFormation console.
After the application deployment is complete, you must confirm the subscription to the Amazon SNS topic. An email will be sent to the email address entered in Step 3 with a link that you need to select to confirm the subscription. This confirmation provides opt-in consent for AWS to send emails to you via the specified Amazon SNS topic. The emails will be notifications of potentially sensitive data that need to be approved. If you don’t see the verification email, be sure to check your spam folder.
Test the application
The application uses an EventBridge scheduled rule to start the sensitive data scan workflow, which runs every 6 hours. You can manually start an execution of the workflow to verify that it’s working. To test the function, you will need a file that contains data that matches your rules for sensitive data. For example, it is easy to create a spreadsheet, document, or text file that contains names, addresses, and numbers formatted like credit card numbers. You can also use this generated sample data to test Macie.
We will test by uploading a file to our S3 bucket via the AWS web console. If you know how to copy objects from the command line, that also works.
Upload test objects to the S3 bucket
Navigate to the Amazon S3 console and upload one or more test objects to the <BucketNamePrefix>-data-pipeline-raw bucket. <BucketNamePrefix> is the prefix you entered when deploying the application in the AWS SAM CLI prompts. You can use any objects as long as they’re a supported file type for Amazon Macie. I suggest uploading multiple objects, some with and some without sensitive data, in order to see how the workflow processes each.
Start the Scan State Machine
Navigate to the Step Functions state machines console. If you don’t see your state machine, make sure you’re connected to the same region that you deployed your application to.
Choose the state machine you created using the AWS SAM CLI as seen in Figure 3. The example state machine is maciepipelinescanstatemachine, but you might have used a different name in your deployment.
Figure 3: AWS Step Functions state machines console
Select the Start execution button and copy the value from the Enter an execution name – optional box. Change the Input – optional value replacing <execution id> with the value just copied as follows:
{
“id”: “<execution id>”
}
In my example, the <execution id> is fa985a4f-866b-b58b-d91b-8a47d068aa0c from the Enter an execution name – optional box as shown in Figure 4. You can choose a different ID value if you prefer. This ID is used by the workflow to tag the objects being processed to ensure that only objects that are scanned continue through the pipeline. When the EventBridge scheduled event starts the workflow as scheduled, an ID is included in the input to the Step Functions workflow. Then select Start execution again.
Figure 4: New execution dialog box
You can see the status of your workflow execution in the Graph inspector as shown in Figure 5. In the figure, the workflow is at the pollForCompletionWait step.
Figure 5: AWS Step Functions graph inspector
The sensitive discovery job should run for about five to ten minutes. The jobs scale linearly based on object size, but there is a start-up time per job that is constant. If sensitive data is found in the objects uploaded to the <BucketNamePrefix>-data-pipeline-upload S3 bucket, an email is sent to the address provided during the AWS SAM deployment step, notifying the recipient requesting of the need for an approval decision, which they indicate by selecting the link corresponding to their decision to approve or deny the next step as shown in Figure 6.
Figure 6: Sensitive data identified email
When you receive this notification, you can investigate the findings by reviewing the objects in the <BucketNamePrefix>-data-pipeline-manual-review S3 bucket. Based on your review, you can either apply remediation steps to remove any sensitive data or allow the data to proceed to the next step of the data ingestion pipeline. You should define a standard response process to address discovery of sensitive data in the data pipeline. Common remediation steps include review of the files for sensitive data, deleting the files that you do not want to progress, and updating the ETL process to redact or tokenize sensitive data when re-ingesting into the pipeline. When you re-ingest the files into the pipeline without sensitive data, the files will not be flagged by Macie.
The workflow performs the following:
If you select Approve, the files are moved to the <BucketNamePrefix>-data-pipeline-scanned-data S3 bucket with an Amazon S3 SensitiveDataFound object tag with a value of true.
If you select Deny, the files are deleted from the <BucketNamePrefix>-data-pipeline-manual-review S3 bucket.
If no action is taken, the Step Functions workflow execution times out after five days and the file will automatically be deleted from the <BucketNamePrefix>-data-pipeline-manual-review S3 bucket after 10 days.
Clean up the application
You’ve successfully deployed and tested the sensitive data pipeline scan workflow. To avoid ongoing charges for resources you created, you should delete all associated resources by deleting the CloudFormation stack. In order to delete the CloudFormation stack, you must first delete all objects that are stored in the S3 buckets that you created for the application.
To delete the application
Empty the S3 buckets created in this application (<BucketNamePrefix>-data-pipeline-raw S3 bucket, <BucketNamePrefix>-data-pipeline-scan-stage, <BucketNamePrefix>-data-pipeline-manual-review, and <BucketNamePrefix>-data-pipeline-scanned-data).
Before using this application in a production data pipeline, you will need to stop and consider some practical matters. First, the notification mechanism used when sensitive data is identified in the objects is email. Email doesn’t scale: you should expand this solution to integrate with your ticketing or workflow management system. If you choose to use email, subscribe a mailing list so that the work of reviewing and responding to alerts is shared across a team.
Second, the application is run on a scheduled basis (every 6 hours by default). You should consider starting the application when your preliminary validations have completed and are ready to perform a sensitive data scan on the data as part of your pipeline. You can modify the EventBridge Event Rule to run in response to an Amazon EventBridge event instead of a scheduled basis.
Third, the application currently uses a 60 second Step Functions Wait state when polling for the Macie discovery job completion. In real world scenarios, the discovery scan will take 10 minutes at a minimum, likely several orders of magnitude longer. You should evaluate the typical execution times for your application execution and tune the polling period accordingly. This will help reduce costs related to running Lambda functions and log storage within CloudWatch Logs. The polling period is defined in the Step Functions state machine definition file (macie_pipeline_scan.asl.json) under the pollForCompletionWait state.
Fourth, the application currently doesn’t account for false positives in the sensitive data discovery job results. Also, the application will progress or delete all objects identified based on the decision by the reviewer. You should consider expanding the application to handle false positives through automation rather than manual review / intervention (such as deleting the files from the manual review bucket or removing the sensitive data tags applied).
Last, the solution will stop the ingestion of a subset of objects into your pipeline. This behavior is similar to other validation and data quality checks that most customers perform as part of the data pipeline. However, you should test to ensure that this will not cause unexpected outcomes and address them in your downstream application logic accordingly.
Conclusion
In this post, I showed you how to integrate sensitive data discovery using Macie as an additional validation step in an automated data pipeline. You’ve reviewed the components of the application, deployed it using the AWS SAM CLI, tested to validate that the application functions as expected, and cleaned up by removing deployed resources.
You now know how to integrate sensitive data scanning into your ETL pipeline. You can use automation and—where required—manual review to help reduce the risk of sensitive data, such as personally identifiable information, being inadvertently ingested into a data lake. You can take this application and customize it to fit your use case and workflows, such as using custom data identifiers as part of your scans, adding additional validation steps, creating Macie suppression rules to define cases to archive findings automatically, or only request manual approvals for findings that meet certain criteria (such as high severity findings).
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the Amazon Macie forum.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
Amazon Elasticsearch Service (Amazon ES) provides fine-grained access control, powered by the Open Distro for Elasticsearch security plugin. The security plugin adds Kibana authentication and access control at the cluster, index, document, and field levels that can help you secure your data. You now have many different ways to configure your Amazon ES domain to provide access control. In this post, I offer basic configuration information to get you started.
Figure 1: A high-level view of data flow and security
Figure 1 details the authentication and access control provided in Amazon ES. The left half of the diagram details the different methods of authenticating. Looking horizontally, requests originate either from Kibana or directly access the REST API. When using Kibana, you can use a login screen powered by the Open Distro security plugin, your SAML identity provider, or Amazon Cognito. Each of these methods results in an authenticated identity: SAML providers via the response, Amazon Cognito via an AWS Identity and Access Management (IAM) identity, and Open Distro via an internal user identity. When you use the REST API, you can use AWS Signature V4 request signing (SigV4 signing), or user name and password authentication. You can also send unauthenticated traffic, but your domain should be configured to reject all such traffic.
The right side of the diagram details the access control points. You can consider the handling of access control in two phases to better understand it—authentication at the edge by IAM and authentication in the Amazon ES domain by the Open Distro security plugin.
First, requests from Kibana or direct API calls have to reach your domain endpoint. If you follow best practices and the domain is in an Amazon Virtual Private Cloud (VPC), you can use Amazon Elastic Compute Cloud (Amazon EC2) security groups to allow or deny traffic based on the originating IP address or security group of the Amazon EC2 instances. Best practice includes least privilege based on subnet ACLs and security group ingress and egress restrictions. In this post, we assume that your requests are legitimate, meet your access control criteria, and can reach your domain.
When a request reaches the domain endpoint—the edge of your domain—, it can be anonymous or it can carry identity and authentication information as described previously. Each Amazon ES domain carries a resource-based IAM policy. With this policy, you can allow or deny traffic based on an IAM identity attached to the request. When your policy specifies an IAM principal, Amazon ES evaluates the request against the allowed Actions in the policy and allows or denies the request. If you don’t have an IAM identity attached to the request (SAML assertion, or user name and password) you should leave the domain policy open and pass traffic through to fine-grained access control in Amazon ES without any checks. You should employ IAM security best practices and add additional IAM restrictions for direct-to-API access control once your domain is set up.
The Open Distro for Elasticsearch security plugin has its own internal user database for user name and password authentication and handles access control for all users. When traffic reaches the Elasticsearch cluster, the plugin validates any user name and password authentication information against this internal database to identify the user and grant a set of permissions. If a request comes with identity information from either SAML or an IAM role, you map that backend role onto the roles or users that you have created in Open Distro security.
The Amazon ES console provides a guided wizard that lets you configure—and reconfigure—your Amazon ES domain. Step 1 offers you the opportunity to select some predefined configurations that carry through the wizard. In step 2, you choose the instances to deploy in your domain. In Step 3, you configure the security. This post focuses on step 3. See also these tutorials that explain using an IAM master user and using an HTTP-authenticated master user.
Note: At the time of writing, you cannot enable fine-grained access control on existing domains; you must create a new domain and enable the feature at domain creation time. You can use fine-grained access control with Elasticsearch versions 6.8 and later.
Set your endpoint
Amazon ES gives you a DNS name that resolves to an IP address that you use to send traffic to the Elasticsearch cluster in the domain. The IP address can be in the IP space of the public internet, or it can resolve to an IP address in your VPC. While—with fine-grained access control—you have the means of securing your cluster even when the endpoint is a public IP address, we recommend using VPC access as the more secure option. Shown in Figure 2.
Figure 2: Select VPC access
With the endpoint in your VPC, you use security groups to control which ports accept traffic and limit access to the endpoints of your Amazon ES domain to IP addresses in your VPC. Make sure to use least privilege when setting up security group access.
Enable fine-grained access control
You should enable fine-grained access control. Shown in Figure 3.
Figure 3: Enabled fine-grained access control
Set up the master user
The master user is the administrator identity for your Amazon ES domain. This user can set up additional users in the Amazon ES security plugin, assign roles to them, and assign permissions for those roles. You can choose user name and password authentication for the master user, or use an IAM identity. User name and password authentication, shown in Figure 4, is simpler to set up and—with a strong password—may provide sufficient security depending on your use case. We recommend you follow your organization’s policy for password length and complexity. If you lose this password, you can return to the domain’s dashboard in the AWS Management Console and reset it. You’ll use these credentials to log in to Kibana. Following best practices on choosing your master user, you should move to an IAM master user once setup is complete.
Note: Password strength is a function of length, complexity of characters (e.g., upper and lower case letters, numbers, and special characters), and unpredictability to decrease the likelihood the password could be guessed or cracked over a period of time.
Figure 4: Setting up the master username and password
Do not enable Amazon Cognito authentication
When you use Kibana, Amazon ES includes a login experience. You currently have three choices for the source of the login screen:
The Open Distro security plugin
Amazon Cognito
Your SAML-compliant system
You can apply fine-grained access control regardless of how you log in. However, setting up fine-grained access control for the master user and additional users is most straightforward if you use the login experience provided by the Open Distro security plugin. After your first login, and when you have set up additional users, you should migrate to either Cognito or SAML for login, taking advantage of the additional security they offer. To use the Open Distro login experience, disable Amazon Cognito authentication, as shown in Figure 5.
Figure 5: Amazon Cognito authentication is not enabled
If you plan to integrate with your SAML identity provider, check the Prepare SAML authentication box. You will complete the set up when the domain is active.
Figure 6: Choose Prepare SAML authentication if you plan to use it
Use an open access policy
When you create your domain, you attach an IAM policy to it that controls whether your traffic must be signed with AWS SigV4 request signing for authentication. Policies that specify an IAM principal require that you use AWS SigV4 signing to authenticate those requests. The domain sends your traffic to IAM, which authenticates signed requests to resolve the user or role that sent the traffic. The domain and IAM apply the policy access controls and either accept the traffic or reject it based on the commands. This is done down to the index level for single-index API calls.
When you use fine-grained access control, your traffic is also authenticated by the Amazon ES security plugin, which makes the IAM authentication redundant. Create an open access policy, as shown in Figure 7, which doesn’t specify a principal and so doesn’t require request signing. This may be acceptable, since you can choose to require an authenticated identity on all traffic. The security plugin authenticates the traffic as above, providing access control based on the internal database.
Figure 7: Selected open access policy
Encrypted data
Amazon ES provides an option to encrypt data in transit and at rest for any domain. When you enable fine-grained access control, you must use encryption with the corresponding checkboxes automatically checked and not changeable. These include Transport Layer Security (TLS) for requests to the domain and for traffic between nodes in the domain, and encryption of data at rest through AWS Key Management Service (KMS). Shown in Figure 8.
Figure 8: Enabled encryption
Accessing Kibana
When you complete the domain creation wizard, it takes about 10 minutes for your domain to activate. Return to the console and the Overview tab of your Amazon ES dashboard. When the Domain Status is Active, select the Kibana URL. Since you created your domain in your VPC, you must be able to access the Kibana endpoint via proxy, VPN, SSH tunnel, or similar. Use the master user name and password that you configured earlier to log in to Kibana, as shown in Figure 9. As detailed above, you should only ever log in as the master user to set up additional users—administrators, users with read-only access, and others.
Figure 9: Kibana login page
Conclusion
Congratulations, you now know the basic steps to set up the minimum configuration to access your Amazon ES domain with a master user. You can examine the settings for fine-grained access control in the Kibana console Security tab. Here, you can add additional users, assign permissions, map IAM users to security roles, and set up your Kibana tenancy. We’ll cover those topics in future posts.
In this blog post, I show you how to improve the distributed denial of service (DDoS) resilience of your self-managed Domain Name System (DNS) service by using AWS Global Accelerator and AWS Shield Advanced. You can use those services to incorporate some of the techniques used by Amazon Route 53 to protect against DDoS attacks.
DNS routes users to your application by quickly translating a human-readable domain name to a machine-readable IP address. When protecting the availability of your application against DDoS attacks, it’s important to consider every part of the stack, including domain name resolution. The recommended best practice is to create hosted zones on Route 53, a scalable, highly available DNS service that’s protected against large DDoS attacks and query floods. Route 53 uses anycast routing to serve DNS queries from more than 150 edge locations around the globe. With anycast routing, DNS queries are served from locations that are closer to your users and the globally distributed DDoS mitigation capacity of Amazon Web Services (AWS) reduces the impact of attacks.
Optionally, you can also build your own DNS service on Amazon Elastic Compute Cloud (Amazon EC2). For example, you can run your own proprietary DNS server to take advantage of custom features that you wrote to integrate with an existing DNS service that isn’t running on AWS. When you register a domain name, you’re usually required to provide at least two name servers that can respond to queries from your users. It’s possible to build a DNS service on only two instances, but that provides limited DDoS resilience.
Solution overview
To protect your self-managed DNS service using this solution, you need a strong understanding of DNS and how to operate a distributed, self-managed DNS service on Amazon EC2. This solution improves upon an existing self-managed DNS service by significantly enhancing its ability to withstand DDoS attacks. There are two components that you add to your application:
You use Global Accelerator to provide your application with two static IP addresses that act as a fixed entry point to Amazon EC2 instances in multiple AWS Regions. Global Accelerator uses anycast to route your traffic to a point of entry close to the source of the traffic. In addition to providing availability and performance benefits, this gives you access to global DDoS mitigation capacity through AWS.
You use Shield Advanced to monitor the availability of your application and automatically engage the AWS Shield Response Team (SRT) if its availability is affected by a DDoS attack. When you associate a Route 53 health check to your protected resources, Shield Advanced uses the health of the application as an input for detection and as a signal to SRT to contact your operations center when needed. You can also engage with SRT to write custom mitigations for your application. For your self-managed DNS service use case, this can include mitigations like DNS packet validation and suspicion scoring that gives a higher priority to queries that are more likely to be legitimate traffic for your application.
As part of this solution, you will build a DNS canary that uses Amazon CloudWatch to update the status of a Route 53 health check if your self-managed DNS service stops responding to queries. An example architecture using Amazon EC2 based DNS behind Global Accelerator and Shield is shown in figure 1.
Figure 1: Amazon EC2 based DNS behind Global Accelerator and Shield
Create and configure an accelerator
To begin, create an accelerator and add your existing DNS servers as endpoints. The newly created accelerator will receive queries and forward them to your DNS service.
Starting with the TCP listener, enter the following in the form boxes for each Region specified in the previous step:
Endpoint type: SelectEC2 instance from the drop-down list.
Endpoint: Select a DNS instance from the drop-down list.
Weight: 128
If you have additional DNS instances in the Region, choose Add endpoint and repeat the preceding steps, but select a DNS instance that hasn’t been added as an endpoint.
Repeat all of the preceding steps for the UDP listener, then choose Create accelerator.
When you choose the Create accelerator button, you’re redirected to a Global Accelerator console page that lists all the accelerators in your account. On this page, you can view the global IPs and DNS name allocated to your newly created accelerator, in addition to the current status.
Wait until the status of the accelerators changes to Deployed before proceeding with any tests.
Configure Shield Advanced and Shield Advanced proactive engagement
Protect your accelerator with Shield Advanced, monitor the health of your application, and configure proactive engagement. When you turn on proactive engagement, the SRT will directly contact you if an Amazon Route 53 health check associated with your protected resource becomes unhealthy during an event that’s detected by Shield Advanced.
To configure proactive engagement
Step 1: Create a Route 53 health check
If you already have a Route 53 health check that monitors the health of your DNS service, you can proceed to step 2 of this section. If you don’t yet have a health check, you can use this AWS CloudFormation template to create one. The template will:
Create a Lambda function that queries your DNS server through the accelerator global IPs. This function posts metrics to CloudWatch to indicate whether the query was successful or not.
Create a CloudWatch alarm that will detect when DNS queries fail.
Create a Route 53 health check that tracks the CloudWatch alarm and changes status to unhealthy when the alarm changes to the Alarm state.
Step 2: Subscribe to Shield Advanced
Please note that with AWS Shield Advanced, you pay a monthly fee of $3,000 per month per organization. In addition, you also pay for AWS Shield Advanced Data Transfer usage fees for AWS resources enabled for advanced protection.
In the AWS Shield navigation bar, choose Getting started, and then choose Subscribe to Shield Advanced.
On the Subscribe to Shield Advanced page, read the terms of agreement, and then select all of the check boxes to indicate that you accept the terms.
Choose Subscribe to Shield Advanced.
Step 3: Add resources to protect
Do one of the following, depending on if you were already subscribed to Shield Advanced.
If you just subscribed to Shield Advanced by completing Step 2 above, choose Add resources to protect.
If you were already subscribed to Shield Advanced, open the Shield console and choose Protected Resources, and then choose Add resources to protect.
In the Choose resources to protect with Shield Advanced page, select the Regions and resource types that you want to protect, then choose Load resources.
Select the resources that you want to protect, and then choose Protect with Shield Advanced.
In the Configure health check based DDoS detection page, under the Protected resources section, select a Route 53 health check to add—either one that you created previously, or a health check created by the AWS CloudFormation template—as the Associated Health Check.
Choose Next until you reach the Review and configure DDoS mitigation and visibility page, and then review the settings and choose Finish configuration.
Step 4: Add contacts
Navigate to the Overview tab of the AWS Shield console.
In the Proactive engagements and contacts section, choose Edit under the Contacts heading.
In the Add contact form, add the contact’s Email, Phone number, and Notes.
Choose Save.
Step 5: Request proactive engagement
Choose Edit proactive engagement feature.
Select Enable.
Choose Save.
Step 6: Configuration review with the SRT
After you enable proactive engagement, the state will be Proactive engagement requested and pending.
SRT will contact you to schedule a configuration review. The review will include a review of your Route 53 health check configuration and a consultation about custom mitigations that can be configured to support your DNS use case. Following this review, SRT will complete your request to enable proactive engagement.
Summary
DNS is a foundational part of the user experience for any application that is accessed via a human readable domain name. Your DNS service should be highly available, DDoS resilient, and accessible to your users with minimal latency. If you run your own DNS service on Amazon EC2, you can improve the DDoS resiliency using Global Accelerator and Shield Advanced. This solution provides your users with a low latency path to your DNS service and provides you with some of the DDoS mitigation that protects Route 53. To learn more about DDoS best practices, see AWS Best Practices for DDoS Resiliency.
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the AWS Shield forum or contact AWS Support.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
Architect, deploy, run, and optimize workloads in the AWS Cloud by mapping the AESCSF 2019 framework to the best practices described in the five pillars of the AWS Well-Architected Framework.
Identify areas of control alignment and potential control gaps with respect to the AESCSF 2019 framework.
The AESCSF 2019 framework comprises 11 domains. Each domain contains one or more objectives, with each objective broken down into specific individual practices. Nine of the 11 domains also contain examples of anti-patterns or specific indicators of bad practice.
The AEMO describes the AESCSF 2019 framework as:
“focussed on cyber security maturity and […] therefore not prescriptive in relation to security controls. It describes what your organisation should strive to achieve, but not how they should achieve it.”
It’s important to note that security and compliance is a shared responsibility between AWS and our customers. AWS is responsible for the security of the cloud (that is, the infrastructure that runs all of the services in the AWS Cloud) but customers are responsible for the security of the systems and applications they deploy in the cloud.
The AWS AESCSF 2019 Workbook helps customers align with the AESCSF 2019 framework by providing control mappings for:
Security of the cloud by mapping AESCSF 2019 framework practices to control statements from the AWS Compliance Program.
Security in the cloud guidance by mapping AESCSF 2019 framework practices to the five pillars of the AWS Well-Architected Framework, and also to AWS Config managed rules and Amazon GuardDuty findings, where available or applicable.
The AWS AESCSF 2019 Workbook does not provide mappings to the anti-patterns, because these are specifically focused on helping customers identify bad practices within their organizations.
The downloadable workbook contains two embedded formats:
Microsoft Excel – Coverage includes AWS responsibility control statements and Well-Architected Framework best practices.
Dynamic HTML – Coverage is the same as in the Microsoft Excel format, with the added feature that the Well-Architected Framework best practices are mapped to AWS Config managed rules and Amazon GuardDuty findings, where available or applicable.
The workbook is available for download through AWS Artifact, accessible through your AWS account.
If you have feedback about this post, submit comments in the Comments section below.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
At Amazon Web Services (AWS), we encourage our customers to take advantage of encryption to help secure their data. Encryption is a core component of a good data protection strategy, but people sometimes have questions about how to manage encryption in the cloud to meet the growth pace and complexity of today’s enterprises. Encryption can seem like a difficult task—people often think they need to master complicated systems to encrypt data—but the cloud can simplify it.
In response to frequently asked questions from executives and IT managers, this post provides an overview of how AWS makes encryption less difficult for everyone. In it, I describe the advantages to encryption in the cloud, common encryption questions, and some AWS services that can help.
Cloud encryption advantages
The most important thing to remember about encryption on AWS is that you always own and control your data. This is an extension of the AWS shared responsibility model, which makes the secure delivery and operation of your applications the responsibility of both you and AWS. You control security in the cloud, including encryption of content, applications, systems, and networks. AWS manages security of the cloud, meaning that we are responsible for protecting the infrastructure that runs all of the services offered in the AWS Cloud.
Encryption in the cloud offers a number of advantages in addition to the options available in on-premises environments. This includes on-demand access to managed services that enable you to more easily create and control the keys used for cryptographic operations, integrated identity and access management, and automating encryption in transit and at rest. With the cloud, you don’t manage physical security or the lifecycle of hardware. Instead of the need to procure, configure, deploy, and decommission hardware, AWS offers you a managed service backed by hardware that meets the security requirements of FIPS 140-2. If you need to use that key tens of thousands of times per second, the elastic capacity of AWS services can scale to meet your demands. Finally, you can use integrated encryption capabilities with the AWS services that you use to store and process your data. You pay only for what you use and can instead focus on configuring and monitoring logical security, and innovating on behalf of your business.
Addressing three common encryption questions
For many of the technology leaders I work with, agility and risk mitigation are top IT business goals. An enterprise-wide cloud encryption and data protection strategy helps define how to achieve fine-grained access controls while maintaining nearly continuous visibility into your risk posture. In combination with the wide range of AWS services that integrate directly with AWS Key Management Service (AWS KMS), AWS encryption services help you to achieve greater agility and additional control of your data as you move through the stages of cloud adoption.
The configuration of AWS encryption services is part of your portion of the shared responsibility model. You’re responsible for your data, AWS Identity and Access Management (IAM) configuration, operating systems and networks, and encryption on the client-side, server-side, and network. AWS is responsible for protecting the infrastructure that runs all of the services offered in AWS.
That still leaves you with responsibilities around encryption—which can seem complex, but AWS services can help. Three of the most common questions we get from customers about encryption in the cloud are:
How can I use encryption to prevent unauthorized access to my data in the cloud?
How can I use encryption to meet compliance requirements in the cloud?
How do I demonstrate compliance with company policies or other standards to my stakeholders in the cloud?
Let’s look closely at these three questions and some ways you can address them in AWS.
How can I use encryption to prevent unauthorized access to my data in the cloud?
Start with IAM
The primary way to protect access to your data is access control. On AWS, this often means using IAM to describe which users or roles can access resources like Amazon Simple Storage Service (Amazon S3) buckets. IAM allows you to tightly define the access for each user—whether human or system—and set the conditions in which that access is allowed. This could mean requiring the use of multi-factor authentication, or making the data accessible only from your Amazon Virtual Private Cloud (Amazon VPC).
Encryption allows you to introduce an additional authorization condition before granting access to data. When you use AWS KMS with other services, you can get further control over access to sensitive data. For example, with S3 objects that are encrypted by KMS, each IAM user must not only have access to the storage itself but also have authorization to use the KMS key that protects the data. This works similarly for Amazon Elastic Block Store (Amazon EBS). For example, you can allow an entire operations team to manage Amazon EBS volumes and snapshots, but, for certain Amazon EBS volumes that contain sensitive data, you can use a different KMS master key with different permissions that are granted only to the individuals you specify. This ability to define more granular access control through independent permission on encryption keys is supported by all AWS services that integrate with KMS.
When you configure IAM for your users to access your data and resources, it’s critical that you consider the principle of least privilege. This means you grant only the access necessary for each user to do their work and no more. For example, instead of granting users access to an entire S3 bucket, you can use IAM policy language to specify the particular Amazon S3 prefixes that are required and no others. This is important when thinking about the difference between using a service—data plane events—and managing a service—management plane events. An application might store and retrieve objects in an S3 bucket, but it’s rarely the case that the same application needs to list all of the buckets in an account or configure the bucket’s settings and permissions.
Making clear distinctions between who can use resources and who can manage resources is often referred to as the principle of separation of duties. Consider the circumstance of having a single application with two identities that are associated with it—an application identity that uses a key to encrypt and decrypt data and a manager identity that can make configuration changes to the key. By using AWS KMS together with services like Amazon EBS, Amazon S3, and many others, you can clearly define which actions can be used by each persona. This prevents the application identity from making configuration or permission changes while allowing the manager to make those changes but not use the services to actually access the data or use the encryption keys.
Use AWS KMS and key policies with IAM policies
AWS KMS provides you with visibility and granular permissions control of a specific key in the hierarchy of keys used to protect your data. Controlling access to the keys in KMS is done using IAM policy language. The customer master key (CMK) has its own policy document, known as a key policy. AWS KMS key policies can work together with IAM identity policies or you can manage the permissions for a KMS CMK exclusively with key policies. This gives you greater flexibility to separately assign permissions to use the key or manage the key, depending on your business use case.
Encryption everywhere
AWS recommends that you encrypt as much as possible. This means encrypting data while it’s in transit and while it’s at rest.
For customers seeking to encrypt data in transit for their public facing applications, our recommended best practice is to use AWS Certificate Manager (ACM). This service automates the creation, deployment, and renewal of public TLS certificates. If you’ve been using SSL/TLS for your websites and applications, then you’re familiar with some of the challenges related to dealing with certificates. ACM is designed to make certificate management easier and less expensive.
One way ACM does this is by generating a certificate for you. Because AWS operates a certificate authority that’s already trusted by industry-standard web browsers and operating systems, public certificates created by ACM can be used with public websites and mobile applications. ACM can create a publicly trusted certificate that you can then deploy into API Gateway, Elastic Load Balancing, or Amazon CloudFront (a globally distributed content delivery network). You don’t have to handle the private key material or figure out complicated tooling to deploy the certificates to your resources. ACM helps you to deploy your certificates either through the AWS Management Console or with automation that uses AWS Command Line Interface (AWS CLI) or AWS SDKs.
One of the challenges related to certificates is regularly rotating and renewing them so they don’t unexpectedly expire and prevent your users from using your website or application. Fortunately, ACM has a feature that updates the certificate before it expires and automatically deploys the new certificate to the resources associated with it. No more needing to make a calendar entry to remind your team to renew certificates and, most importantly, no more outages because of expired certificates.
Many customers want to secure data in transit for services by using privately trusted TLS certificates instead of publicly trusted TLS certificates. For this use case, you can use AWS Certificate Manager Private Certificate Authority (ACM PCA) to issue certificates for both clients and servers. ACM PCA provides an inexpensive solution for issuing internally trusted certificates and it can be integrated with ACM with all of the same integrative benefits that ACM provides for public certificates, including automated renewal.
For encrypting data at rest, I strongly encourage using AWS KMS. There is a broad range of AWS storage and database services that support KMS integration so you can implement robust encryption to protect your data at rest within AWS services. This lets you have the benefit of the KMS capabilities for encryption and access control to build complex solutions with a variety of AWS services without compromising on using encryption as part of your data protection strategy.
How can I use encryption to meet compliance requirements in the cloud?
The first step is to identify your compliance requirements. This can often be done by working with your company’s risk and compliance team to understand the frameworks and controls that your company must abide by. While the requirements vary by industry and region, the most common encryption compliance requirements are to encrypt your data and make sure that the access control for the encryption keys (for example by using AWS KMS CMK key policies) is separate from the access control to the encrypted data itself (for example through Amazon S3 bucket policies).
Another common requirement is to have separate encryption keys for different classes of data, or for different tenants or customers. This is directly supported by AWS KMS as you can have as many different keys as you need within a single account. If you need to use even more than the 10,000 keys AWS KMS allows by default, contact AWS Support about raising your quota.
For compliance-related concerns, there are a few capabilities that are worth exploring as options to increase your coverage of security controls.
Amazon S3 can automatically encrypt all new objects placed into a bucket, even when the user or software doesn’t specify encryption.
You can use batch operations in Amazon S3 to encrypt existing objects that weren’t originally stored with encryption.
You can use the Amazon S3 inventory report to generate a list of all S3 objects in a bucket, including their encryption status.
AWS services that track encryption configurations to comply with your requirements
Anyone who has pasted a screenshot of a configuration into a word processor at the end of the year to memorialize compliance knows how brittle traditional on-premises forms of compliance attestation can be. Everything looked right the day it was installed and still looked right at the end of the year—but how can you be certain that everything was correctly configured at all times?
AWS provides several different services to help you configure your environment correctly and monitor its configuration over time. AWS services can also be configured to perform automated remediation to correct any deviations from your desired configuration state. AWS helps automate the collection of compliance evidence and provides nearly continuous, rather than point in time, compliance snapshots.
AWS Config is a service that enables you to assess, audit, and evaluate the configurations of your AWS resources. AWS Config continuously monitors and records your AWS resource configurations and helps you to automate the evaluation of recorded configurations against desired configurations. One of the most powerful features of AWS Config is AWS Config Rules. While AWS Config continuously tracks the configuration changes that occur among your resources, it checks whether these changes violate any of the conditions in your rules. If a resource violates a rule, AWS Config flags the resource and the rule as noncompliant. AWS Config comes with a wide range of prewritten managed rules to help you maintain compliance for many different AWS services. The managed rules include checks for encryption status on a variety of resources, ACM certificate expiration, IAM policy configurations, and many more.
For additional monitoring capabilities, consider Amazon Macie and AWS Security Hub. Amazon Macie is a service that helps you understand the contents of your S3 buckets by analyzing and classifying the data contained within your S3 objects. It can also be used to report on the encryption status of your S3 buckets, giving you a central view into the configurations of all buckets in your account, including default encryption settings. Amazon Macie also integrates with AWS Security Hub, which can perform automated checks of your configurations, including several checks that focus on encryption settings.
Another critical service for compliance outcomes is AWS CloudTrail. CloudTrail enables governance, compliance, operational auditing, and risk auditing of your AWS account. With CloudTrail, you can log, continuously monitor, and retain account activity related to actions across your AWS infrastructure. AWS KMS records all of its activity in CloudTrail, allowing you to identify who used the encryption keys, in what context, and with which resources. This information is useful for operational purposes and to help you meet your compliance needs.
How do I demonstrate compliance with company policy to my stakeholders in the cloud?
You probably have internal and external stakeholders that care about compliance and require that you document your system’s compliance posture. These stakeholders include a range of possible entities and roles, including internal and external auditors, risk management departments, industry and government regulators, diligence teams related to funding or acquisition, and more.
Unfortunately, the relationship between technical staff and audit and compliance staff is sometimes contentious. AWS believes strongly that these two groups should work together—they want the same things. The same services and facilities that engineering teams use to support operational excellence can also provide output that answers stakeholders’ questions about security compliance.
You can provide access to the console for AWS Config and CloudTrail to your counterparts in audit and risk management roles. Use AWS Config to continuously monitor your configurations and produce periodic reports that can be delivered to the right stakeholders. The evolution towards continuous compliance makes compliance with your company policies on AWS not just possible, but often better than is possible in traditional on-premises environments. AWS Config includes several managed rules that check for encryption settings in your environment. CloudTrail contains an ongoing record of every time AWS KMS keys are used to either encrypt or decrypt your resources. The contents of the CloudTrail entry include the KMS key ID, letting your stakeholders review and connect the activity recorded in CloudTrail with the configurations and permissions set in your environment. You can also use the reports produced by Security Hub automated compliance checks to verify and validate your encryption settings and other controls.
Your stakeholders might have further requirements for compliance that are beyond your scope of control because AWS is operating those controls for you. AWS provides System and Organization Controls (SOC) Reports that are independent, third-party examination reports that demonstrate how AWS achieves key compliance controls and objectives. The purpose of these reports is to help you and your auditors understand the AWS controls established to support operations and compliance. You can consult the AWS SOC2 report, available through AWS Artifact, for more information about how AWS operates in the cloud and provides assurance around AWS security procedures. The SOC2 report includes several AWS KMS-specific controls that might be of interest to your audit-minded colleagues.
Summary
Encryption in the cloud is easier than encryption on-premises, powerful, and can help you meet the highest standards for controls and compliance. The cloud provides more comprehensive data protection capabilities for customers looking to rapidly scale and innovate than are available for on-premises systems. This post provides guidance for how to think about encryption in AWS. You can use IAM, AWS KMS, and ACM to provide granular access control to your most sensitive data, and support protection of your data in transit and at rest. Once you’ve identified your compliance requirements, you can use AWS Config and CloudTrail to review your compliance with company policy over time, rather than point-in-time snapshots obtained through traditional audit methods. AWS can provide on-demand compliance evidence, with tools such as reporting from CloudTrail and AWS Config, and attestations such as SOC reports.
I encourage you to review your current encryption approach against the steps I’ve outlined in this post. While every industry and company is different, I believe the core concepts presented here apply to all scenarios. I want to hear from you. If you have any comments or feedback on the approach discussed here, or how you’ve used it for your use case, leave a comment on this post.
And for more information on encryption in the cloud and on AWS, check out the following resources, in addition to our collection of encryption blog posts.
As you look to manage network security on Amazon Web Services (AWS), there are multiple tools you can use to protect your resources and keep your data safe. Amazon Virtual Private Cloud (Amazon VPC), security groups (SGs), network access control lists (network ACLs), AWS WAF, and the recently launched AWS Network Firewall all offer points of protection for your AWS workload. Managing these security controls directly works well when everything is in a single or small number of accounts. However, if you’re part of a security team managing controls on a larger number of accounts, or part of a compliance team whose responsibility includes auditing and remediating application configurations owned by other teams, managing these controls at scale could become cumbersome. To make sure that it doesn’t become so for you, we’re going to walk you through how to manage the new AWS Network Firewall at scale using AWS Firewall Manager.
First, a primer on the new Network Firewall. Network Firewall is a stateful, managed, network firewall and intrusion detection and prevention service for traffic in Amazon VPC. With Network Firewall, you can filter traffic going to and coming from an internet gateway, NAT gateway, or over VPN or AWS Direct Connect using both stateful and stateless rules. The network firewall inspects individual packets by using a stateless rule processing engine and inspects packets in the context of their workflows by using a stateful rule processing engine. The stateless rules engine takes rules with standard 5-tuple connection criteria. The stateful engine takes rules compatible with Suricata. These capabilities enable you to add more advanced, packet payload–level protections for your VPC resources.
In this post, you will learn how to create, configure, and maintain Network Firewall firewalls with common security policies across appropriate accounts and VPCs in your AWS Organizations structure by leveraging Firewall Manager.
Firewall Manager prerequisites
You must complete the following prerequisites before you create and apply a Firewall Manager policy:
A Firewall Manager administrator account: You must designate one of the AWS accounts in your organization as the Firewall Manager administrator. This gives the account permission to deploy security policies across the organization.
AWS Config: You must enable AWS Config for all of the accounts in your organization so that Firewall Manager can detect newly created resources. To enable AWS Config for all of the accounts in your organization, use the Enable AWS Config template from the StackSets sample templates.
AWS Resource Access Manager (AWS RAM): You must enable AWS RAM for all accounts in your organization so that Firewall Manager can modify the Network Firewall configurations.
Architecture diagram
Figure 1 shows an example organizational structure in AWS Organizations, with several organizational units (OUs) that we’ll use in the example policy sets in this blog post.
Figure 1: Best practices OU structure for AWS Organizations
Firewall Manager can be associated to either the AWS primary payer account or one of the member AWS accounts that has appropriate permissions as a delegated administrator. Following the best practices for organizational units, we use a dedicated Security Tooling AWS account (named Security in the diagram) to serve as the Firewall Manager administrator from within the Security OU. The Security OU is used for hosting security-related access and services. The Security OU, its child OUs, and the associated AWS accounts should be owned and managed by your security organization.
This post will focus on two of the accounts in this organization. The first account is the Security Account, since this is where the Firewall Manager Administrator is defined. The second account we will focus on is Tenant 5 in the Staging OU. If you are following these steps, make sure the first account you are signed in to is the Firewall Manager administrator for your organization. You can do this by verifying the Administrator account ID in the Firewall Manager console under Settings. If you don’t have an administrator set, you can find the steps to set one in the Firewall Manager documentation.
Deployment of network firewalls and security policies
Managing security policies begins inside the WAF & Shield console under the AWS Firewall Manager heading. When you navigate from the console and select Firewall Manager, it will bring you to the Getting Started page. You can confirm that you’ve completed the prerequisites mentioned earlier in this post. If the prerequisites aren’t met, use the links in the Prerequisites section to complete the necessary steps. It’s important to note that Network Firewall is the first integration to require the AWS Organizations management account to have AWS RAM enabled. You can find more information about how to do that in the AWS RAM Sharing Your Resources documentation.
AWS Firewall Manager offers multiple security policy types for each service that it manages. A Firewall Manager security policy is a set of configurations that a security administrator defines, including relevant rules, protections, and actions that must be deployed and the accounts and resources (indicated by tags) to include or exclude. With the ability to create a different security policy for each AWS managed service, you can create granular and flexible configurations while still being able to scale control out to large numbers of accounts and VPCs. These policies automatically and consistently enforce the rules you configure even when new accounts and resources are created. For this post, we will focus on the Network Firewall policy type in the Firewall Manager console.
Security policy part 1: Defining a security policy’s rules
The Network Firewall policy type is a regional construct (meaning it applies to one Region only) comprised of stateless rule groups, a policy scope, and policy tags. When you first pick the type of policy in the Firewall Manager security policy console, you also choose the Region you want the policy to apply to. Once you’ve picked your Region, you can configure your policy with a policy name and a Network Firewall policy. This is where you pick the stateless and stateful rule groups and default actions for packets that don’t match any rules, as shown in Figure 2. If you try to add rule groups but none populate the window, this can either mean that you didn’t define any rules and rule groups for the network firewall, or you created them in a different Region. You can choose the link in the window to go to the Network Firewall page to create or import rules.
Additionally, the capacity units for each rule is shown in the interface. Capacity units refer to the total amount of capacity each individual rule allocates towards a total limit for a rule group, and are subject to service quotas. You can find more information on capacity units in the Network Firewall capacity documentation. If you want the same policy to apply to multiple Regions, using AWS CloudFormation StackSets and an infrastructure-as-code approach helps you deploy a policy in each Region. Your CloudFormation template would include the Network Firewall rules, rule group definitions, and security policies.
Figure 2: Defining rule groups for Network Firewall security policy
The next section of the console relates to the configuration of the network firewall. There are two different configuration areas, shown in Figure 3, and once they’re configured they cannot be changed. The first configuration relates to the number of firewall endpoints. This impacts both the cost and availability of the network firewall. Situations where a single network firewall in a single Availability Zone provides adequate availability for the environment could include test or demo environments, applications or workloads that are built solely in a single Availability Zone, or environments where low cost is the driving factor. For environments where high availability is required, applications or workloads are built across multiple Availability Zones, or designers want to reduce cross Availability Zone traffic or dependencies, it’s recommended to use multiple firewall endpoints. To better understand this tradeoff for your workloads, the AWS Well-Architected Framework is the best place to learn more about designing for reliability and cost optimization as well as security, operational excellence, and performance.
The second configuration element is the available Classless Inter-Domain Routing (CIDR) blocks to use for the Network Firewall subnets when they are being created. This optional field should have the /28 subnet you intend to have pulled from the VPC CIDR block as part of the creation of the network firewall. This comes in handy if VPCs in an organizational account follow consistent IP addressing practices, and it will allow more intuitive design guidelines and implementations. You can find more information on how the CIDR blocks are used in the Firewall Manager documentation for security policies. If this field is left blank, Firewall Manager will take a best-effort approach to find unassigned CIDR blocks in your VPCs to create a subnet for Network Firewall. If no CIDR blocks are available, Firewall Manager will display a non-compliant error on its dashboard.
At this point, you’ve defined the Network Firewall security policy’s rules; the next step is to define what the policy should apply to.
Security policy part 2: Defining the security policy scope
Now that you’ve defined the security policy rules, the policy should be scoped to apply only to the appropriate accounts and VPCs. It’s important to note that for each security policy, there will be one Network Firewall instantiation. Therefore, if you apply multiple security policies to an account or to a VPC, multiple firewalls will be created, leading to inefficient routing, cost, and complexity. Firewall Manager doesn’t merge proposed configurations into network firewalls created outside the Firewall Manager framework. Firewall Manager can, however, update or change the configuration of firewalls it manages at any time. Therefore, it’s best to architect your policies with your organizational structure in mind.
Firewall Manager enables you to modify all accounts and resources in an organization, or tailor a policy scope to specific OUs and resources. The architecture diagram in this blog post outlined a practical scenario for how you can structure OUs. Considering security policies, it would be reasonable for network firewalls to have different policies in a Production OU that impacts Tenants 1 and 2, compared to the Sandbox OU for Tenants 7 and 8. However, you might have some commonality between the Pre-prod and Staging OUs. So, for example, you might want to apply the same Network Firewall rules groups across an organization, as shown in Figure 4.
Figure 4: Applying rule groups to an AWS Organizations OU structure
To do this, you would create three different Firewall Manager security policies inside the Security Account in the Security OU:
Prod Environment Policy
Contains rule group “Block-Known-Bad-IPs” and “Block-BadDNS”
Applies to Prod OU
Dev Environment Policy
Contains rule group “Block-Known-Bad-IPs” and “Block-Corporate-Prod”
Applies to Pre-prod OU and Staging OU
Sandbox Environment Policy
Contains rule group “Block-Known-Bad-IPs”
Applies to Sandbox OU
This policy application is shown in Figure 5.
Figure 5: The corresponding security policy application
In addition to applying the security policy to accounts in OUs, it is also possible to filter based on the tags associated with VPCs in the accounts, as shown in Figure 6. For example, if your accounts contain a VPC with bastion hosts, enforcing the same routing and outbound traffic policies could break other security elements. In these cases, tagging the VPC with a consistent identifying key pair such as “Bastion-VPC:True” would enable Firewall Manager to exclude that subnet from requiring a path through the network firewall.
Figure 6: Defining the security policy scope by organization unit and tagging
Security policy part 3: Defining the security policy tags
As part of your Firewall Manager security policy, you should also define policy tags. These tags can be used for multiple purposes, including adding context, defining ownership, or even authorizing changes by using attribute-based authentication with IAM. This step is optional, but recommended to improve the operations. Some recommended tags include:
Policy description: A longer description to capture the purpose of the policy
Policy owner: A contact person for when changes to the policy must be made
Cost Center: Where costs associated with the security policies should be incurred
Last date edited: Enables you to keep track of changes to the policy and map the changes back to a change log or ticketing system
Last date reviewed: Helps maintain an audit schedule to verify that appropriate policy is set and audits mandated by compliance regimes are easily captured
Your organization might have other tags that are also mandated, and these can be configured upon policy creation as well.
Once you’ve defined the appropriate tags, you can review the policy before Firewall Manager puts your policy into effect. It’s important to also note that when you choose Create Policy, Firewall Manager creates AWS RAM configurations and AWS Config rules to enable management and visibility for the Firewall Manager Administrator account, and the member accounts will incur the associated costs.
After the Network Firewall deployment
Now the Firewall Manager policy has been created. On the AWS Firewall Manager policies screen, shown in Figure 7, you’ll see the total number of accounts that are encompassed by the OU selection and tag filters you created, and the number of accounts that are fully compliant with the policy. Because this is a new policy, Firewall Manager must evaluate the status of the accounts before deeming them compliant or noncompliant. Another benefit of this view is the ability to report on ongoing compliance with any given policy. Remember how AWS Config is a prerequisite for Firewall Manager? That’s because AWS Config enables Firewall Manager to access information about the current state of the firewalls and VPCs in each account and report back and/or enforce compliance with the policy on an ongoing basis.
Figure 7: Validating compliance of accounts by policy
In the background, Firewall Manager is building the components required for the network firewalls in each account. This includes the dedicated firewall subnet, the associated route tables in each specific VPC, and then the firewall itself. Once these tasks are completed, Firewall Manager pushes the rule groups defined in the security policy. If a network firewall already exists, Firewall Manager will still follow the same steps and create additional subnets, route tables, and firewalls in the VPCs. Remember, as we mentioned earlier, that Firewall Manager doesn’t update or change the configuration for network firewalls it didn’t create.
Once the resources are built, it can take a couple of minutes for the accounts to be evaluated and appropriately classified in the Firewall Manager console. After the accounts have been evaluated, selecting the name of the Firewall Manager security policy shows which accounts are within the policy scope, their status, and any relevant details. If Firewall Manager identifies any noncompliant events, statuses, or policies, this area of the console is also where those alerts will appear. For a detailed list of possible event types, the Firewall Manager documentation can provide more information.
If you look under Policy Action there is an important informational box, shown in Figure 8.
Figure 8: Information box that identifies necessary route table update actions
Firewall Manager creates the network firewall in the defined accounts, but it doesn’t automatically modify the route tables inside the VPC. This ensures that changes being made by the central security team don’t impact other activities that may be going on in the accounts. Consider a situation where the account is owned by the DevOps team and security is owned by the central Security team. This situation makes it possible for the Security team to roll out the new network firewall without impacting the network path of the application. Once the firewall is deployed, the Security team can engage the DevOps team to push the routes into production through the appropriate code pipeline. Steps to modify the route tables can be found in the blog post that covers the deployment models for AWS Network Firewall.
Conclusion
In this blog post, you learned how security administrators can use Firewall Manager to create security policies for the new Network Firewall service and push them out at scale to their organization. As part of that walkthrough, you also learned how compliance auditors can use Firewall Manager to see, in a single place, the compliance of each account with that policy. In the end, by having AWS do the undifferentiated heavy lifting of deploying resources and collecting state at scale, security teams can focus less on operational burdens and more on strategic opportunities. For further reading and updates, see the Firewall Manager Developer Guide. To learn about pricing for solutions using AWS Firewall Manager, check the AWS Firewall Manager pricing page for examples.
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the AWS Firewall Manager forum or contact AWS Support.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
There are a number of ways to grant access to different types of resources, as some resources support both resource-based policies and IAM policies. This blog post will focus on demonstrating how you can use IAM policies to grant restrictive permissions to IAM principals to meet least privilege standards.
In AWS, an IAM principal can be a user, role, or group. These identities start with no permissions and you add permissions using a policy. In AWS, there are different types of policies that are used for different reasons. In this blog, I only give examples for identity-based policies that attach to IAM principals to grant permissions to an identity. You can create and attach multiple identity-based policies to your IAM principals, and you can reuse them across your AWS accounts. There are two types of managed policies. Customer managed policies are created and managed by you, the customer. AWS managed policies are provided as examples, cannot be modified, but can be copied, enhanced, and saved as Customer managed policies. The main elements of a policy statement are:
Effect: Specifies whether the statement will Allow or Deny an action.
Action: Describes a specific action or actions that will either be allowed or denied to run based on the Effect entered. API actions are unique to each service. For example, s3:ListBuckets is an Amazon S3 service API action that enables an IAM Principal to list all S3 buckets in the same account.
NotAction: Can be used as an alternative to using Action. This element will allow an IAM principal to invoke all API actions to a specific AWS service except those actions specified in this list.
Resource: Specifies the resources—for example, an S3 bucket or objects—that the policy applies to in Amazon Resource Name (ARN) format.
NotResource: Can be used instead of the Resource element to explicitly match every AWS resource except those specified.
Condition: Allows you to build expressions to match the condition keys and values in the policy against keys and values in the request context sent by the IAM principal. Condition keys can be service-specific or global. A global condition key can be used with any service. For example, a key of aws:CurrentTime can be used to allow access based on date and time.
Starting with the visual editor
The visual editor is my default starting place for building policies as I like the wizard and seeing all available services, actions, and conditions without looking at the documentation. If there is a complex policy with many services, I often look at the AWS managed policies as a starting place for the actions that are required, then use the visual editor to fine tune and check the resources and conditions.
The policy I’m going to walk you through creating is to grant an AWS Lambda function permission to get specific objects from Amazon S3, and put items in a specific table in Amazon DynamoDB. You can access the visual editor when you choose Create policy under policies in the IAM console, or add policies when viewing a role, group, or user as shown in Figure 1. If you’re not familiar with creating policies, you can follow the full instructions in the IAM documentation.
Figure 1: Use the visual editor to create a policy
Begin by choosing the first service—S3—to grant access to as shown in Figure 2. You can only choose one service at a time, so you’ll need to add DynamoDB after.
Figure 2: Select S3 service
Now you will see a list of access levels with the option to manually add actions. Expand the read access level to show all read actions that are supported by the Amazon S3 service. You can now see all read access level actions. For getting an object, check the box for GetObject. Selecting the ? next to an action expands information including a description, supported resource types, and supported condition keys as shown in Figure 3.
Figure 3: Expand Read in Access level, select GetObject, and select the ? next to GetObject
Expand Resources, you will see that the visual editor has listed object as that is the only resource supported by the GetObject action as shown in Figure 4.
Figure 4: Expand Resources
Select Add ARN, which opens a dialogue to help you specify the ARN for the objects. Enter a bucket name—such as doc-example-bucket—and then the object name. For the object name you can use a wildcard (*) as a suffix. For example, to allow objects beginning with alpha you would enter alpha*. This is an important step. For this least privileged policy, you are restricting to a specific bucket, and an object prefix. You could even specify an individual object depending on your use case.
Figure 5: Enter bucket name and object name
If you have multiple ARNs (bucket and objects) to allow, you can repeat the step.
Figure 6: ARN added for S3 object
The final step is to expand the request conditions, and choose Add condition. The Add request condition dialogue will open. Select the drop down next to Condition key to list the global condition keys, then the service level condition keys are listed after. You’ll see that there’s an s3:ExistingObjectTag condition that—as the name suggests—matches an existing object tag. You can use this condition key to allow the GetObject request only when the object tag meets your condition. That means you can tag your objects with a specific tag key and value pair, and your policy condition must match this key-value pair to allow the action to execute. When you’re using condition keys with multiple keys or values, you can use condition operators and evaluation logic. As shown in Figure 7, tag-key is entered directly below the condition key. This is the key of the tag to match. For the Operator, select StringEquals to match the tag exactly. Checking If exists tests at least one member of the set of request values, and at least one member of the set of condition key values. The Value to enter is the actual tag value: tag-value as shown in figure 7.
Figure 7: ARN added for S3 object
That’s it for adding the S3 action, as shown in figure 8.
Figure 8: S3 GetObject action with resource and conditions configured
Now you need to add the DynamoDB permissions by selecting Add additional permissions. Select Choose a service and then select DynamoDB. For actions, expand the Write access level, then choose PutItem.
Figure 9: Choose write access level
Expand Resources and then select Add ARN. The dialogue that appears will help you build the ARN just like it did for the Amazon S3 service. Enter the Region, for example the ap-southeast-2 (Sydney) Region, the account ID, and the table name. Choosing Add will add the resource ARN to your policy.
Figure 10: Enter Region, account, and table name
Now it’s time to add conditions. Expand Request conditions and then choose Add condition.
There are many DynamoDB conditions that you could use, however you can choose dynamodb:LeadingKeys to represent the first key, or partition keys in a table. You can see from the documentation that a qualifier of For all values in request is recommend. For the Operator you can use StringEquals as your string is going to exactly match, then a Value can use a prefix with wildcard, such as alpha* as shown in figure 11.
Figure 11: Add request conditions
Choosing Add will take you back to the main visual editor where you can choose Review policy to continue. Enter a name and description for the policy, and then choose Create policy.
You can see in this example that a policy can use least privilege by using specific resources and conditions. Note that sometimes when you use the AWS Management Console, it requires additional permissions to provide information for the console experience.
Starting with AWS managed policies
AWS managed policies can be a good starting place to see the actions typically associated with a particular service or job function. For example, you can attach the AmazonS3ReadOnlyAccess policy to a role used by an Amazon EC2 instance that allows read-only access to all Amazon S3 buckets. It has an effect of Allow to allow access, and there are two actions that use wildcards (*) to allow all Get and List actions for S3—for example, s3:GetObject and s3:ListBuckets. The resource is a wildcard to allow all S3 buckets the account has access to. A useful feature of this policy is that it only allows read and list access to S3, but not to any other services or types of actions.
Let’s make our own custom IAM policy to make it least privilege. Starting with the action element, you can use the reference for Amazon S3 to see all actions, a description of what each action does, the resource type for each action, and condition keys for each action. Now let’s imagine this policy is used by an Amazon EC2 instance to fetch an application configuration object from within an S3 bucket. Looking at the descriptions for actions starting with Get you can see that the only action that we really need is GetObject. You can then use the resource element to restrict an action to a set of objects prefixed with config within a specific bucket.
Now that you’ve reduced the scope of what this policy can do for service actions and resources, you can add a condition element that uses attribute based access control (ABAC) to define conditions based on attributes—in this case, a resource tag. In this example, when you’re reading objects from a single bucket, you can set specific conditions to further reduce the scope of permissions given to an IAM principal. There’s an s3:ExistingObjectTag condition that you can use to allow the GetObject request only when the object tag meets your condition. That means you can tag your objects with a specific tag key and value pair, and your IAM policy condition must match this key-value pair to allow the API action to successfully run. When you’re using condition keys with multiple keys or values, you can use condition operators and evaluation logic. You can see that ForAnyValue tests at least one member of the set of request values, and at least one member of the set of condition key values. Alternatively, you can use global condition keys that apply to all services:
In the preceding policy example, the condition element only allows s3:GetObject permissions if the object is tagged with a key of tag-key and a value of tag-value. While you’re experimenting, you can identify errors in your custom policies by using the IAM policy simulator or reviewing the errors messages recorded in AWS CloudTrail logs.
Conclusion
In this post, I’ve shown two different techniques that you can use to create least privilege policies for IAM. You can adapt these methods to create AWS Single Sign-Onpermission sets and AWS Organizations service control policies (SCPs). Starting with managed policies is a useful strategy when an AWS supplied managed policy already exists for your use case, and then to reduce the scope of what it can do through permissions. I tend to use the visual editor the most for editing policies because it saves looking up the resource and conditions for each action. I suggest that you start by reviewing the policies you’re already using. Start with policies that grant excessive permissions—like the example Administrator policy—and tie them back to the use case of the users or things that need the access. Use the last accessed information, IAM best practices, and look at the AWS Well-Architected best practices and AWS Well-Architected tool.
If you have feedback about this post, submit comments in the Comments section below.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
In this post, we explain the United Kingdom (UK) National Cyber Security Centre (NCSC)’s guidance on VPN profiles configuration, and how the configuration parameters for the AWS Virtual Private Network (AWS VPN) align with the NCSC guidance. At the end of the post, there are links to code to deploy the AWS VPN in line with those parameters.
Many public sector organizations in the UK need to connect their existing on-premises facilities, data centers, or offices to the Amazon Web Services (AWS) cloud so they can take advantage of the broad set of services AWS provides to help them deliver against their mission.
This can be achieved using the AWS VPN service. However some customers find it difficult to know the exact configuration parameters that they should choose when establishing the VPN connection in-line with guidance for the UK public sector.
AWS VPN services enable organizations to establish secure connections between their on-premises networks, remote offices, and client devices and the AWS global network. AWS VPN comprises two services: AWS Site-to-Site VPN and AWS Client VPN. Together, they deliver a highly available, fully managed, elastic cloud VPN solution to protect your network traffic.
For the purposes of this post, we focus on the Site-to-Site VPN configuration, not Client VPN because the NCSC guidance we’re discussing is specifically related to site-to-site VPNs. This post covers two areas:
An overview of the current guidance for VPN configurations for the public sector.
Recommendations on how to configure AWS VPN to meet or exceed the current guidance.
VPN guidance for UK public sector organizations
The starting point for security guidance for the UK public sector is often the NCSC. The role of the NCSC includes:
Protecting government systems and information.
Planning for and responding to cyber incidents.
Working with providers of critical national infrastructure to improve the protection and computer security of such infrastructure against cyber-borne threats.
Specifically, for guidance on the configuration of VPNs for the UK public sector to support data at OFFICIAL, the NCSC has created detailed guidance on the technical configurations to support two different profiles: PRIME and Foundation. These two profiles provide different technical implementations to support different equipment and are both suitable for use with OFFCICIAL data. Beyond these technical differences, NCSC also documents that Foundation is expected to provide suitable protection for OFFICIAL information until at least December 31, 2023, while PRIME has no review date specified at the time of writing.
Myth 1: I have to adhere exactly to the NCSC technical configuration or I cannot use a VPN for OFFICIAL data
It’s a common misconception that a public sector organization must adhere exactly to the configuration of either PRIME or Foundation in order to use a VPN for OFFICIAL data, even if other configuration options available—such as a longer key length—offer a higher security baseline.
Note that the NCSC isn’t mandating the use of the configuration in their guidance. They’re offering a configuration that provides a useful baseline, but you must assess your use of the NCSC guidance in context of the risks. To help with these risk-based decisions, the NCSC has developed a series of guidance documents to help organizations make risk-based decisions. A common consideration that might require deviating from the guidance would be supporting interoperability with legacy systems where the suggested algorithms aren’t supported. In this case, a risk-based decision should be made—including accounting for other factors such as cost.
It’s also worth noting that the NCSC creates guidance designed to be useful to as many organizations as possible. The NCSC balances adopting the latest possible configurations with backwards compatibility and vendor support. For example, the NCSC suggests AES-128 where—in theory—AES-256 could also be a good choice. Organizations need to be aware that if they choose to adopt devices that support only AES-256 and later need to connect in devices capable of only AES-128, there could be significant investment to replace the legacy devices with ones that support AES-256. However, AWS provides both AES-128 and AES-256, so if the remote device supports it, AWS would recommend opting for AES-256.
The NCSC also tries to develop advice that has some longevity. For example, the guidance suggesting use of AES-128 was created in 2012 with a view to providing solid guidance over a number of years. This means customers can choose different configuration parameters that offer increased levels of security if both sides of the VPN can support it.
It’s possible for a customer to choose options that might lower the security of the connection, provided that risks are identified and appropriately managed by the customers assurance team. This might be needed to support interoperability between existing systems where the cost of an upgrade outweighs the risk.
Myth 2: Foundation has been deprecated and I must use PRIME
Another common misconception is that Foundation has been deprecated in favor of PRIME. This is not the case. The NCSC has stated that Foundation is expected to provide suitable protection for OFFICIAL information until at least December 31, 2023. The security provided by both solutions provides commensurate security for accessing data classified as OFFICIAL. One of the main differences between PRIME and Foundation is the choice of signature algorithm: RSA or ECDSA. This difference can be helpful in enabling an organization to choose which profile to adopt. For example, if the organization already has a private key infrastructure (PKI), then the decision regarding which signature algorithm to use is based on what existing systems support.
Myth 3: I can’t use Foundation for accessing OFFICIAL SENSITIVE data
A final point that often causes confusion is the classifying of data at OFFICIAL SENSITIVE because it isn’t a classification, but a handling caveat. The data would be classified as OFFICIAL and marked as OFFICIAL SENSITIVE, meaning that systems handling the data need risk-appropriate security measures. A system that can handle OFFICIAL data might be appropriate to handle sensitive information. Hence Foundation could be suitable for accessing OFFICIAL SENSITIVE data, depending on the risks identified.
Deep-dive into the technical specifications
Now that you know a little more about how the guidance should be viewed, let’s look more closely at the technical configurations for each VPN profile.
The following table shows the configuration parameters suggested by the NCSC VPN guidance discussed previously.
Technical detail
Foundation
PRIME
IKEv* – Encryption
IKEv1 – AES with 128-bit keys in CBC mode (RFC3602)
IKEv2 – AES-128 in GCM-128 (and optionally CBC)
IKEv2 – Pseudo-random function
HMAC-SHA256
HMAC-SHA256
IKEv2 – Diffie-Hellman group
Group 14 (2048-bit MODP group) (RFC3526)
256 bit random ECP (RFC5903) Group 19
IKEv2 – Authentication
X.509 certificates with RSA signatures (2048 bits) and SHA-256 (RFC4945 and RFC4055)
X.509 certificates with ECDSA-256 with SHA256 on P-256 curve
ESP – Encryption
AES with 128-bit keys in CBC mode (RFC3602) SHA-256 (RFC4868)
AES-128 in GCM-128 SHA-256 (RFC4868)
Recommended AWS VPN configuration for public sector
Bearing in mind these policies, and remembering that the configuration is only guidance, you must make a risk-based decision. AWS recommends the following configuration as a starting point for the configuration of the AWS VPN.
Technical detail
AWS configuration
Adherence
IKEv* – Encryption
IKEv2 – AES-256-GCM
Suitable for Foundation and PRIME
IKEv2 – Pseudo-random function
HMAC-SHA256
Meets Foundation and PRIME
IKEv2 – Diffie-Hellman group
Group 19
Suitable for Foundation and matches PRIME
IKEv2 – Authentication
RSA 2048 SHA2-512
Suitable for Foundation
ESP – Encryption
AES-256-GCM
Suitable for PRIME and Foundation
In the table above, we use the term suitable for where the protocol doesn’t match the guidance exactly but the AWS configuration options provide equivalent or stronger security—for example, by using a longer key length.
With the configuration defined above, the AWS VPN service is suitable for use under the Foundation profile in all areas. It can also be made suitable for PRIME in all areas apart from IKEv2 encryption. The use of RSA or ECDSA is the main difference between the AWS VPN and PRIME configurations. This makes the current AWS VPN solution closer to Foundation than PRIME.
When considering which options are available to you, the starting point should be the capabilities of your current—and possible future—VPN devices. Based on its capabilities, you can use the NCSC guidance and preceding tables to choose the protocols that match or are suitable for the NCSC guidance.
Summary
To review:
The NCSC provides guidance for the VPN configuration, not a mandate.
An organization is free to decide not to use the guidance, but should consider risks when they make that decision.
The AWS VPN meets or is suitable for the configuration options for Foundation.
After reviewing the details contained in this blog, UK public sector organizations should have the confidence to use the AWS VPN service with systems running at OFFICIAL.
Today, I’m pleased to announce the launch of Cloud Audit Academy AWS-specific (CAA AWS-specific). This is a new, accelerated training program for auditing AWS Cloud implementations, and is designed for auditors, regulators, or anyone working within a control framework.
Over the past few years, auditing security in the cloud has become one of the fastest growing questions among Amazon Web Services (AWS) customers, across multiple industries and all around the world. Here are the two pain points that I hear about most often:
Engineering teams want to move regulatory frameworks compliant workloads to AWS to take advantage of its innovation capabilities, but security and risk teams are uncertain how AWS can help them meet their compliance requirements through audits.
Compliance teams want to effectively audit the cloud environments and take advantage of the available security control options that are built into the cloud, but the legacy audit processes and control frameworks are built for an on-premises environment. The differences require some reconciliation and improvement work to be done on compliance programs, audit processes, and auditor training.
To help address these issues for not only AWS customers but for any auditor or compliance team facing cloud migration, we announced Cloud Audit Academy Cloud Agnostic (CAA Cloud Agnostic) at re:Inforce 2019. This foundational, first-of-its-kind, course provides baseline knowledge on auditing in the cloud and in understanding the differences in control operation, design, and auditing. It is cloud agnostic and can benefit security and compliance professionals in any industry—including independent third-party auditors. Since its launch in June 2019, 1,400 students have followed this cloud audit learning path, with 91 percent of participants saying that they would recommend the workshop to others.
So today we’re releasing the next phase of that education program, Cloud Audit Academy AWS-specific. Offered virtually or in-person, CAA AWS-specific is an instructor-led workshop on addressing risks and auditing security in the AWS Cloud, with a focus on the security and audit tools provided by AWS. All instructors have professional audit industry experience, current audit credentials, and maintain AWS Solutions Architect credentials.
Here are four things to know about CAA AWS-specific and what it has to offer audit and compliance teams:
Content was created with PricewaterhouseCoopers (PwC) PricewaterhouseCoopers worked with us to develop the curriculum content, bringing their expertise in independent risk and control auditing.
“With so many of our customers already in the cloud—or ready to be—we’ve seen a huge increase in the need to meet regulatory and compliance requirements. We’re excited to have combined our risk and controls experience with the power of AWS to create a curriculum in which customers can not only [leverage AWS to help them] meet their compliance needs, but unlock the total value of their cloud investment.” – Paige Hayes, Global Account Leader at PwC
Attendees earn continuing professional education credits Based on feedback from CAA Cloud Agnostic, we now offer continuing professional education (CPE) credits to attendees. Completion of CAA AWS-specific will allow attendees to earn 28 CPE credits towards any of the International Information System Security Certification Consortium, or (ISC)², certifications, and 18 CPE credits towards any Global Information Assurance Certification (GIAC).
Training helps boost confidence when auditing the AWS cloud Our customers have proven repeatedly that running sensitive workloads in AWS can be more secure than in on-premises environments. However, a lack of knowledge and updated processes for implementing, monitoring, and proving compliance in the cloud has caused some difficulty. Through CAA AWS-specific, you will get critical training to become more comfortable and confident knowing how to audit the AWS environment with precision.
“Our FSI customer conversations are often focused on security and compliance controls. Leveraging the Cloud Audit Academy enables our team to educate the internal and external auditors of our customers. CAA provides them the necessary tools and knowledge to evaluate and gain comfort with their AWS control environment firsthand. The varying depth and levels focus on everything from basic cloud auditing to diving deeper into the domains which align with our governance and control domains. We reference key AWS services that customers can utilize to create an effective control environment that [helps to meet their] regulatory and audit expectations.” – Jeff (Axe) Axelrad, Compliance Manager, AWS Financial Services
Training enables the governance, risk, and compliance professional In four days of CAA AWS-specific, you’ll become more comfortable with topics like control domains, network management, vulnerability management, logging and monitoring, incident response, and general knowledge about compliance controls in the cloud.
“In addition to [using AWS to help support and maintain their compliance], our customers need to be able to clearly communicate with their external auditors and regulators HOW compliance is achieved. CAA doesn’t teach auditors how to audit, but rather accelerates the learning necessary to understand specifically how the control landscape changes.” – Jesse Skibbe, Sr. Practice Manager, AWS Professional Services
CAA Cloud Agnostic provides some foundational concepts and is a prerequisite to CAA AWS-specific. It is available for free online at our AWS Training and Certification learning library, or you can contact your account manager to have a one-day instructor-led training session in person.
If it sounds like Cloud Audit Academy training would benefit you and your team, contact our AWS Security Assurance Services team or contact your AWS account manager. For more information, check out the newly updated Security Audit Learning Path.
If you have feedback about this post, submit comments in the Comments section below.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
AWS re:Invent will certainly be different in 2020! Instead of seeing you all in Las Vegas, this year re:Invent will be a free, three-week virtual conference. One thing that will remain the same is the variety of sessions, including many Security, Identity, and Compliance sessions. As we developed sessions, we looked to customers—asking where they would like to expand their knowledge. One way we did this was shared in a recent Security blog post, where we introduced a new customer polling feature that provides us with feedback directly from customers. The initial results of the poll showed that Identity and Access Management and Data Protection are top-ranking topics for customers. We wanted to highlight some of the re:Invent sessions for these two important topics so that you can start building your re:Invent schedule. Each session is offered at multiple times, so you can sign up for the time that works best for your location and schedule.
Managing your Identities and Access in AWS
AWS identity: Secure account and application access with AWS SSO Ron Cully, Principal Product Manager, AWS
AWS SSO provides an easy way to centrally manage access at scale across all your AWS Organizations accounts, using identities you create and manage in AWS SSO, Microsoft Active Directory, or external identity providers (such as Okta Universal Directory or Azure AD). This session explains how you can use AWS SSO to manage your AWS environment, and it covers key new features to help you secure and automate account access authorization.
Getting started with AWS identity services Becky Weiss, Senior Principal Engineer, AWS
The number, range, and breadth of AWS services are large, but the set of techniques that you need to secure them is not. Your journey as a builder in the cloud starts with this session, in which practical examples help you quickly get up to speed on the fundamentals of becoming authenticated and authorized in the cloud, as well as on securing your resources and data correctly.
AWS identity: Ten identity health checks to improve security in the cloud Cassia Martin, Senior Security Solutions Architect, AWS
Get practical advice and code to help you achieve the principle of least privilege in your existing AWS environment. From enabling logs to disabling root, the provided checklist helps you find and fix permissions issues in your resources, your accounts, and throughout your organization. With these ten health checks, you can improve your AWS identity and achieve better security every day.
AWS identity: Choosing the right mix of AWS IAM policies for scale Josh Du Lac, Principal Security Solutions Architect, AWS
This session provides both a strategic and tactical overview of various AWS Identity and Access Management (IAM) policies that provide a range of capabilities for the security of your AWS accounts. You probably already use a number of these policies today, but this session will dive into the tactical reasons for choosing one capability over another. This session zooms out to help you understand how to manage these IAM policies across a multi-account environment, covering their purpose, deployment, validation, limitations, monitoring, and more.
Zero Trust: An AWS perspective Quint Van Deman, Principal WW Identity Specialist, AWS
AWS customers have continuously asked, “What are the optimal patterns for ensuring the right levels of security and availability for my systems and data?” Increasingly, they are asking how patterns that fall under the banner of Zero Trust might apply to this question. In this session, you learn about the AWS guiding principles for Zero Trust and explore the larger subdomains that have emerged within this space. Then the session dives deep into how AWS has incorporated some of these concepts, and how AWS can help you on your own Zero Trust journey.
This session is for central security teams and developers who manage application permissions. This session reviews a permissions model that enables you to scale your permissions management with confidence. Learn how to set your organization up for access management success with permission guardrails. Then, learn about granting workforce permissions based on attributes, so they scale as your users and teams adjust. Finally, learn about the access analysis tools and how to use them to identify and reduce broad permissions and give users and systems access to only what they need.
Goldman Sachs takes security and access to AWS accounts seriously. While empowering teams with the freedom to build applications autonomously is critical for scaling cloud usage across the firm, guardrails and controls need to be set in place to enable secure administrative access. In this session, learn how the company built its credential brokering workflow and administrator access for its users. Learn how, with its simple application that uses proprietary and AWS services, including Amazon DynamoDB, AWS Lambda, AWS CloudTrail, Amazon S3, and Amazon Athena, Goldman Sachs is able to control administrator credentials and monitor and report on actions taken for audits and compliance.
Data Protection
Do you need an AWS KMS custom key store? Tracy Pierce, Senior Consultant, AWS
AWS Key Management Service (AWS KMS) has integrated with AWS CloudHSM, giving you the option to create your own AWS KMS custom key store. In this session, you learn more about how a KMS custom key store is backed by an AWS CloudHSM cluster and how it enables you to generate, store, and use your KMS keys in the hardware security modules that you control. You also learn when and if you really need a custom key store. Join this session to learn why you might choose not to use a custom key store and instead use the AWS KMS default.
Using certificate-based authentication on containers & web servers on AWS Josh Rosenthol, Senior Product Manager, AWS Kevin Rioles, Manager, Infrastructure & Security, BlackSky
In this session, BlackSky talks about its experience using AWS Certificate Manager (ACM) end-entity certificates for the processing and distribution of real-time satellite geospatial intelligence and monitoring. Learn how BlackSky uses certificate-based authentication on containers and web servers within its AWS environment to help make TLS ubiquitous in its deployments. The session details the implementation, architecture, and operations best practices that the company chose and how it was able to operate ACM at scale across multiple accounts and regions.
The busy manager’s guide to encryption Spencer Janyk, Senior Product Manager, AWS
In this session, explore the functionality of AWS cryptography services and learn when and where to deploy each of the following: AWS Key Management Service, AWS Encryption SDK, AWS Certificate Manager, AWS CloudHSM, and AWS Secrets Manager. You also learn about defense-in-depth strategies including asymmetric permissions models, client-side encryption, and permission segmentation by role.
Building post-quantum cryptography for the cloud Alex Weibel, Senior Software Development Engineer, AWS
This session introduces post-quantum cryptography and how you can use it today to secure TLS communication. Learn about recent updates on standards and existing deployments, including the AWS post-quantum TLS implementation (pq-s2n). A description of the hybrid key agreement method shows how you can combine a new post-quantum key encapsulation method with a classical key exchange to secure network traffic today.
Data protection at scale using Amazon Macie Neel Sendas, Senior Technical Account Manager, AWS
Data Loss Prevention (DLP) is a common topic among companies that work with sensitive data. If an organization can’t identify its sensitive data, it can’t protect it. Amazon Macie is a fully managed data security and data privacy service that uses machine learning and pattern matching to discover and protect your sensitive data in AWS. In this session, we will share details of the design and architecture you can use to deploy Macie at large scale.
While sessions are virtual this year, they will be offered at multiple times with live moderators and “Ask the Expert” sessions available to help answer any questions that you may have. We look forward to “seeing” you in these sessions. Please see the re:Invent agenda for more details and to build your schedule.
If you have feedback about this post, submit comments in the Comments section below.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
In the weeks leading up to re:Invent, we’ll share conversations we’ve had with people at AWS who will be presenting, and get a sneak peek at their work.
How long have you been at AWS?
I’ve been at AWS for 4 years.
What’s your favorite part of your job?
The ability to channel the technologist, sales person, developer, and creative marketer and fuse them all into one in my current role as a security solutions architect at AWS. It’s deeply satisfying to know that multiple AWS services put together can help solve a security problem for a customer.
How did you get started in Security?
I was a product manager in one of my previous jobs where I started working deeper with crypto algorithms used in the financial services industry. This led me to understand how, in certain industry verticals, security is a core part of product building and how important it was to infuse security features into the various functionalities that a product provides. Since then, I have pursued my interest further in this field.
How do you explain what you do to non-technical friends or family?
My 8-year-old daughter once asked me, “Why aren’t you delivering packages although you work for Amazon?” Since then, I always thought about how I would explain to her what I do and this is what I came up with: The Netflix shows that you watch, they are streamed from computers that are hosted on Amazon Web Services. My job is to provide advice to customers, such as Netflix and others, on how they can continuously innovate and enrich their end customers’ experience, while making sure that it’s done in a secure manner.
What are you currently working on that you’re excited about?
Customers are trying to use AWS security services at scale to solve for security problems that span multiple regions and multiple AWS accounts. Currently, I am working on providing prescriptive guidance to customers on trade-offs that they need to think about while building and protecting their data on AWS across their multi-account and multi-region architectural deployments.
You’re presenting at re:Invent this year – can you give readers a sneak peek of what you’re covering?
Protecting data in transit is an important security control that AWS customers want to implement. In this talk, we are working with one of our customers, BlackSky, and talking about their initiative to achieve TLS Everywhere. We will cover architectural trade-offs, automation at scale, and architectural best practices while using AWS Certificate Manager (ACM).
What are you hoping your audience will do differently after your session?
After attending this session, customers will become more comfortable in knowing that AWS Certificate Manager (ACM) can help them achieve TLS Everywhere for the applications and architectures that they build on AWS.
From your perspective, what’s the biggest thing happening in security right now?
In my opinion, a lot of startups that build security products are now being born in the cloud, and, with AWS Marketplace, it’s very easy for customers to take advantage of these security services that these startups build and integrate it within their AWS accounts. This is big for the security startup ecosystem and can spur a lot of innovation in security.
Think Big is one of the leadership principles I really like. The reason is that the ability to think big about any problem that one is trying to solve will allow you to look at the problem across multiple dimensions, and the end result can produce significant impact and a superior customer experience.
What’s the best career advice you’ve ever received?
One of my mentors told me to never give up if the first iteration of a product fails. I have seen that persisting through failures can lead to lot of learning about what customers actually want and, in the long term, helps build valuable customer experiences.
If you could go back, what would you tell yourself at the beginning of your career?
I would have told myself to seek out and work with teams with a growth mindset, along with a strong builder’s culture.
From what I understand, you enjoy table tennis in your free time, correct?
This is a sport I have played since high school and I got into it then. I like the competition and the pace of the game. The margin of error is very low in this game, and I love how the probability of winning changes every minute, making it super competitive and fun.
If you have feedback about this post, submit comments in the Comments section below.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
Starting today, you can pass user attributes in the AWS session when your workforce sign-in into the cloud using AWS Single Sign-On. This gives you the centralized account access management of AWS Single Sign-On and ABAC, with the flexibility to use AWS SSO, Active Directory, or an external identity provider as your identity source. To learn more about the advantages of ABAC policies on AWS, you may read my previous blog post on the subject.
Overview On one side, system administrators configure user attributes on the AWS Single Sign-On identity repository, or the managed Active Directory. System administrators may also configure an external identity provider, such as Okta, OneLogin or PingFederate to pass existing user attributes in the AWS sessions when their workforce federates into AWS. These attributes are known as session tags in AWS. On the other side, cloud administrators create fine-grained permissions policies such that your workforce get only access to cloud resources with matching resource tags.
Creating policies based on matching attributes instead of functional roles helps to reduce the number of distinct permissions and roles you must create and manage in your AWS environment. For example, when developers Bob from team red and Alice from team blue sign-in into AWS and assume the same AWS Identity and Access Management (IAM) role, they get distinct permissions to project resources tagged for their team. The identity system sends the team name attribute in the AWS session when Bob and Alice sign-in into AWS. The role’s permissions grant access to project resources with matching team name tags. Now, if Bob moves to team blue and system administrators update his team name in their identity provider directory, Bob automatically gets access to team blue’s project resources without requiring permissions updates in IAM.
How to Configure AWS SSO to Map User Attributes Before to configure AWS SSO, there are two important points to highlight. First, ABAC will work with attributes from any identity source configured in AWS SSO : AWS SSO itself, a managed Active Directory, or an external identity provider. Second, there are two ways to pass attributes for access control to AWS SSO. Either you can pass attributes directly in the SAML assertion using the prefix https://aws.amazon.com/SAML/Attributes/AccessControl, or you can use attributes that are in the AWS SSO identity store. Those attributes are configured by your AWS SSO administrator for users created in AWS SSO, synchronized in from an Active Directory, or synchronized in from an external identity provider using automatic provisioning (SCIM).
For this demo, I choose to use an external identity provider and SCIM.
I can enable ABAC in AWS using AWS SSO with three steps:
Step 1: I configure my identity source with the associated user identities and attributes in the external identity provider. As of today, AWS SSO supports identity synchronization via SCIM with Azure AD, Okta, OneLogin, and PingFederate. Check this page to get an up-to-date list. The specifics depend on each identity provider.
Step 2: I configure the SCIM attributes I want to use for access control using the new Access Control Attributes global setting in the AWS SSO console or API. This screen allows me to select attributes for access control from the identity source I configured in step 1.
Step 3: I author ABAC rules through permission sets and resource-based policies using the attributes I configured in Step 2. More about this in a minute.
Now, when my workforce federates into an AWS account using SSO, they get access to their AWS resources based on matching attributes.
Attributes are passed as session tags. They are passed as comma-separated key:value pairs. The total character length of all the attributes together must be less than or equal to 460 characters.
What Does a Policy Look Like? I now can use user attributes in my permission sets using the aws:PrincipalTag condition key when creating access control rules. For example, I can tag all the resources in my organization with their respective department name, and use a single permission set that grants developers access only to their department resources. Now, whenever developers federate into the AWS account, AWS SSO creates a department session tag with the value received from the identity provider. The security policies allow them to only get access to the resources in their respective department. As the team adds more developers and resources to their project, I only have to tag resources with the correct department name. As a result, as the organization adds new resources and developers to departments, developers can only manage resources aligned to their department without needing any permission updates.
An ABAC SSO permission set policy might look like this:
This policy allows anybody to DescribeInstances, but only users with a aws:PrincipalTag/Department tag’s value matching the EC2 instance ec2:ResourceTag/Department tag’s value are authorized to stop or to start instances.
I attach this policy to an AWS Account’s Permission Set. On the left part of the AWS Single Sign-On console, I click AWS Accounts and select the Permission sets tab. Then I click Create permission set. On the next screen, I select Create a customer permission set.
I enter a name and description, I make sure Create a custom permissions policy is selected. Then I can copy/paste the previous policy allowing to start and stop EC2 instances when the department name tag value is equal to the person’s department name tag value.
On the next screen, I enter some tags, then I review my configuration before clicking Create. Et voila, I am ready to go.
If you have existing federation configured with AWS Security Token Service, remember that external identity providers consider AWS SSO as a new application configuration. This means when you move from direct IAM federation to AWS SSO, you have to update your external identity provider configuration to connect with AWS SSO and to introduce attributes as session tags for this configuration.
Available Today There is no additional charge to configure user attributes with AWS Single Sign-On. You can start to use it today in all AWS Regions where AWS SSO is available.
In the weeks leading up to re:Invent, we’ll share conversations we’ve had with people at AWS who will be presenting, and get a sneak peek at their work.
How long have you been at AWS and what do you do in your current role?
Working with smart and awesome people who I get to learn a lot from.
How did you get started in Security?
Around 2000, I became a system administrator for a multiuser university shell service called RedBrick. RedBrick is an old-school Unix terminal service run by students, for students. Thousands of curious people had access to log in, which makes it a very interesting security challenge. We had to keep everything extremely up-to-date and deal with all sorts of nuisances and abuse. I learned how to find and report new kernel vulnerabilities, deal with denial-of-service attacks, and manage campaigns like getting everyone to move to the encrypted SSH protocol rather than Telnet (which was more common at the time). We tried educating users, but in the end I built a client with a one-click SSH to RedBrick button and that did the trick.
How do you explain what you do to non-technical friends or family?
“I work on the internet” is probably the most common, or these days I can say, “I work on the cloud.” Most of my friends and family are non-technical; we hang out and play music, and catch up and socialize. I try to avoid talking about work.
What are you currently working on that you’re excited about?
Nitro Enclaves is going to make it cheaper and easier for customers to isolate sensitive data. That’s a big deal. Anything we can do that is going to improve the security of people’s data is a big deal. We’re all tired and weary of hearing about “yet another data breach.” Not everyone has the depth of expertise and experience that Amazon has. When we can take the lessons we’ve learned, and the techniques we’ve applied, for securing businesses like Amazon.com and then give those lessons and techniques to customers in an easy to consume form—that excites me.
You’re presenting at re:Invent this year—can you give readers a sneak peek of what you’re covering?
I’ll be talking about Nitro Enclaves, but also presenting some more insights into how we build at AWS. We recently launched the Amazon Builders’ Library, which is an ongoing series of articles and deep dives into lessons we’ve learned from building Amazon.com, Alexa, AWS, and other large services. I’m going to cover what simplicity means for us, and also talk about things we do that most customers would never need to do themselves, so that should be fun.
What are you hoping that your audience will do differently after your session?
I’ll be happy if people pick up a few tips and tricks and get a sense of how we break down problems in a customer-obsessed way.
My favorite leadership principle is Ownership. I love that we’re empowered (and expected) to be owners at Amazon. Part of that is not having to seek a lot of permission, which helps with moving quickly, and part of that is a feeling of team pride that comes from a job well done.
What’s the best career advice you’ve ever received?
Be fully committed or get out of the way, but don’t do anything in between.
If you could go back, what would you tell yourself at the beginning of your career?
I’ve caught enough lucky breaks that I feel like I’ve done really well in my career, definitely wildly beyond what I could have dreamed of when I was a teenager, so I wouldn’t want to change anything. Who knows how things would go then! If I could go back in time, I’d give some hints and help to amazingly talented people I know who got stung by bad luck.
What are you most proud of in your career?
Becoming a Project Management Committee (PMC) member for the Apache httpd webserver was a huge milestone for me. I got to contribute to and maintain Apache, and was trusted to be release manager. That was all volunteer work, but it started everything for me.
I hear you play Irish music. What instruments do you play?
Yes, I play and sing Irish traditional music. Mainly guitar, but also piano, Irish whistle, banjo, cittern, and bouzouki. Those last instruments are double-stringed and used mainly for accompaniment. I’ve played in stage shows, bands, and I get to record and tour often enough, when we’re not on lockdown. It is very hard to beat how fun it is to play music with other people, there’s something very special about it. Now that I live in the U.S., it also connects me to Ireland, where I grew up, and it gives me an opportunity to sing in Irish, the language I spoke at home and at school growing up.
If you have feedback about this post, submit comments in the Comments section below.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
Our mission at Amazon Web Services (AWS) is to innovate on behalf of our customers so they have less and less work to do when building, deploying, and rapidly iterating on secure systems. From a security perspective, our customers seek answers to the ongoing question What are the optimal patterns to ensure the right level of confidentiality, integrity, and availability of my systems and data while increasing speed and agility? Increasingly, customers are asking specifically about how security architectural patterns that fall under the banner of Zero Trust architecture or Zero Trust networking might help answer this question.
Given the surge in interest in technology that uses the Zero Trust label, as well as the variety of concepts and models that come under the Zero Trust umbrella, we’d like to provide our perspective. We’ll share our definition and guiding principles for Zero Trust, and then explore the larger subdomains that have emerged under that banner. We’ll also talk about how AWS has woven these principles into the fabric of the AWS cloud since its earliest days, as well as into many recent developments. Finally, we’ll review how AWS can help you on your own Zero Trust journey, focusing on the underlying security objectives that matter most to our customers. Technological approaches rise and fall, but underlying security objectives tend to be relatively stable over time. (A good summary of some of those can be found in the Design Principles of the AWS Well-Architected Framework.)
Definition and guiding principles for Zero Trust
Let’s start out with a general definition. Zero Trust is a conceptual model and an associated set of mechanisms that focus on providing security controls around digital assets that do not solely or fundamentally depend on traditional network controls or network perimeters. The zero in Zero Trust fundamentally refers to diminishing—possibly to zero!—the trust historically created by an actor’s location within a traditional network, whether we think of the actor as a person or a software component. In a Zero Trust world, network-centric trust models are augmented or replaced by other techniques—which we can describe generally as identity-centric controls—to provide equal or better security mechanisms than we had in place previously. Better security mechanisms should be understood broadly to include attributes such as greater usability and flexibility, even if the overall security posture remains the same. Let’s consider more details and possible approaches along the two dimensions.
One dimension is the network. Do we achieve Zero Trust by allowing all network packets to flow between all hosts or endpoints, but implement all security controls above the network layer? Or do we break our systems down into smaller logical components and implement much tighter network segments or packet-level controls—so-called micro-segments or micro-perimeters? Do we add some kind of gateway or proxy technology that enforces a new kind of trust boundary? Do we still use VPN technology for network isolation but make it more dynamic and hidden from the user experience, so that users don’t even notice that network boundaries are being created and torn down as needed? Or some combination of these techniques?
The other dimension is identity and access management. Are we talking about human actors with their PCs, tablets, and phones trying to access web applications? Or are we talking about machine-to-machine, software-to-software communication, where all requests are authenticated and authorized using other kinds of techniques? Or perhaps we’re thinking of some combination of the two. For example, certain security-relevant properties or attributes of the user’s situation—strength of authentication, device type, ownership, posture assessment, health, network location, and others—are propagated to and through the software systems with which the user is interacting, and alter their access dynamically.
Thus, as we start to look more closely at Zero Trust, we can immediately see the possibility of confusion—because many different topics and concepts are implicated—but also a clear indication of opportunities to build better, more flexible, and more secure software systems. What are some of the principles that can help guide us through both the confusion and the opportunities?
Our first guiding principle for Zero Trust is that while the conceptual model decreases reliance on network location, the role of network controls and perimeters remains important to the overall security architecture. In other words, the best security doesn’t come from making a binary choice between identity-centric and network-centric tools, but rather by using both effectively in combination with each other. Identity-centric controls, such as the AWS SigV4 request signing process, which is used to interact with AWS API endpoints, uniquely authenticate and authorize each and every signed API request, and provide very fine-grained access controls. However, network-centric tools such as Amazon Virtual Private Cloud (Amazon VPC), security groups, AWS PrivateLink, and VPC endpoints are straightforward to understand and use, filter unnecessary noise out of the system, and provide excellent guardrails within which identity-centric controls can operate. Ideally, these two kinds of controls should not only coexist, they should be aware of and augment one another. For example, VPC endpoints provide the ability to attach a policy that allows you to write and enforce identity-centric rules at a logical network boundary—in that case, the private network exit from your Amazon VPC on the way to a nearby AWS service endpoint.
Our second guiding principle for Zero Trust is that it can mean different things in different contexts. Arguably one of the key reasons for the ambiguity surrounding Zero Trust is that the term encompasses many different use cases which share only the fundamental technical concept of diminishing the security relevance of a network location or boundary. Yet those use cases differ substantially in what they’re trying to achieve for the organization. As we noted above, common examples of Zero Trust goals range from ensuring workforce agility and mobility—using browsers and mobile apps and the internet to access business systems and applications—to the creation of carefully segmented micro-service architectures inside of new cloud-based applications. By focusing on a specific problem that we’re trying to solve, and approaching it with fresh eyes and new tools, we can avoid getting mired in low-value discussions around whether a new approach to a security challenge is really—or to what degree it is—an application of the Zero Trust concept.
Our third guiding principle is that Zero Trust concepts must be applied in accordance with the organizational value of the system and data being protected. Over time, the application of the Zero Trust conceptual model and associated mechanisms will continue to improve defense in depth, and continue to make security controls we already have work better through the increased visibility and software-defined nature of the cloud. Applied well, the tenets of Zero Trust can significantly raise the security bar, especially for critical workloads. However, if applied in strict orthodoxy, Zero Trust methods can limit the incorporation of more traditional technologies into upgraded or new systems, and stifle innovation by overly taxing organizations where the benefits aren’t commensurate with the effort. For many business systems, network controls and network perimeters will continue to be important and usually adequate controls for a long time, perhaps forever. We believe it’s best to think of Zero Trust concepts as additive to existing security controls and concepts, rather than as replacements.
Examples of Zero Trust principles and capabilities at work today within the AWS cloud
The most prominent example of Zero Trust in AWS is how millions of customers typically interact with AWS every day using the AWS Management Console or securely calling AWS APIs over a diverse set of public and private networks. Whether called via the console, the AWS Command Line Interface (AWS CLI), or software written to the AWS APIs, ultimately all of these methods of interaction reach a set of web services with endpoints that are reachable from the internet. There is absolutely nothing about the security of the AWS API infrastructure that depends on network reachability. Each one of these signed API requests is authenticated and authorized every single time at rates of millions upon millions of requests per second globally. Our customers do so confidently; knowing that the cryptographic strength of the underlying Transport Layer Security (TLS) protocol—augmented by the AWS Signature v4 signing process—properly secures these requests without any regard to the trustworthiness of the underlying network. Interestingly, the use of cloud-based APIs is rarely—if ever—mentioned in Zero Trust discussions. Perhaps this is because AWS led the way with this approach to securing APIs from the start, such that it is now assumed to be a basic part of every cloud security story.
Similarly, but perhaps not as well understood, when individual AWS services need to call each other to operate and deliver their service capabilities, they rely on the same mechanisms that you use as a customer. You can see this in action in the form of service-linked roles. For example, when AWS Auto Scaling determines that it needs to call the Amazon Elastic Compute Cloud (Amazon EC2) API to create or terminate an EC2 instance in your account, the AWS Auto Scaling service assumes the service-linked role you’ve provided in your account, receives the resulting AWS short-term credentials, and uses these credentials to sign requests using the SigV4 process to the appropriate EC2 APIs. On the receiving end, AWS Identity and Access Management (IAM) authenticates and authorizes the incoming calls for EC2. In other words, even though they’re both AWS services, AWS Auto Scaling and EC2 have no inherent trust, network or otherwise, of one another and use strong identity-centric controls as the basis of the security model between the two services as they operate on your behalf. You, the customer, have full visibility into both the privileges that you’re granting to one service, as well as an AWS CloudTrail record of the use of those privileges.
Other great examples of Zero Trust capabilities in the AWS portfolio can be found in the IoT Service. When we launched AWS IoT Core we made a strategic decision—against the prevailing industry norms at the time—to always require TLS network encryption and modern client authentication, including certificate-based mutual TLS, when connecting IoT devices to service endpoints. We subsequently added TLS support to FreeRTOS, enabling modern, secure communication to an entire class of small CPU and small memory devices that were previously assumed to not be capable of it. With AWS IoT Greengrass, we pioneered a way of working with existing no-security devices using a remote gateway that relied on local network presence but also was able to run AWS Lambda functions to validate security and provide a secure proxy to the cloud. These examples highlight where adherence to AWS security standards brought key foundational components of Zero Trust to a technology domain where vast amounts of unauthenticated, unencrypted network messaging over the open internet was previously the norm.
How AWS can help you on your Zero Trust journey
To help you on your own Zero Trust journey, there are a number of AWS cloud-specific identity and networking capabilities that provide core Zero Trust building blocks as standard features. AWS services provide this functionality via simple API calls, without you needing to build, maintain, or operate any infrastructure or additional software components. To help best frame the conversation, we’ll consider these capabilities against the backdrop of three distinct use cases:
Authorizing specific flows between components to eliminate unneeded lateral network mobility.
Enabling friction-free access to internal applications for your workforce.
Securing digital transformation projects such as IoT.
Our first use case focuses mainly on machine-to-machine communications—authorizing specific flows between components to help eliminate lateral network mobility risk. Otherwise put, if two components don’t need to talk to one another across the network, they shouldn’t be able to, even if these systems happen to exist within the same network or network segment. This greatly reduces the overall surface area of the connected systems and eliminates unneeded pathways, particularly those that lead to sensitive data. Within this use case, our discussion should begin with security groups, which have been a part of Amazon EC2 since its earliest days. Security groups provide highly dynamic, software-defined network micro-perimeters for both north-south and east-west traffic. Security group assignments occur automatically as resources come and go, and rules in one security group can reference one another by ID, either within the same Amazon VPC or across larger peered networks in the same or different regions. These properties allow security groups to act as a kind of identity system in which group membership becomes a relevant property for determining whether or not to permit particular network flows. This helps enable you to author extremely granular rules without the associated operational burden of keeping them up-to-date as membership in a group ebbs and flows. Similarly, PrivateLink provides an extremely useful building block in the general space of micro-perimeters and micro-segmentation. Using PrivateLink, a load-balanced endpoint can be exposed as a narrow, one-way gateway between two VPCs, with tight identity-based controls determining who can access the gateway and where incoming packets can land. Initiating network connections in the other direction isn’t allowed at all, and the VPCs don’t even need to have routes between one another. Thousands of customers use PrivateLink today as a fundamental building block of a secure micro-services architecture, as well as secure and private access to PaaS and SaaS services from their suppliers.
Going back to our discussion about AWS APIs, the AWS SigV4 signature process for authenticating and authorizing API requests is no longer just for AWS services. You can achieve the same kind of hardened interface approach using the Amazon API Gateway service, which allows software interfaces to be securely available on the open internet. API Gateway provides distributed denial of service (DDoS) protection, rate limiting, and AWS IAM support as one of several authorization options. When you choose AWS IAM authorization, you author standard IAM policies that define who can call your API and where they can call it from, using the full expressiveness of the IAM policy language. Callers sign their requests using their AWS credentials, typically delivered in the form of IAM roles attached to compute resources, and IAM uniquely authenticates and authorizes every single call to your API according to those policies. With one step, your API is protected behind the massively scaled, super performant, globally available IAM service that protects AWS APIs—with nothing for you to manage or maintain. Calls from the API Gateway front-end to your back-end implementation are secured by mutual TLS, so you’re assured that only API Gateway is able to invoke the back-end implementation. With this strong identity-centric control in place, you have two choices. You can safely place your back-end implementation on the public network, or add the VPC integration model such that the API Gateway call to your back-end implementation running inside of your VPC is protected by an identity-centric control (mutual TLS) and a network-centric control (private connectivity from API Gateway to your code). The security achieved by these feature combinations, arguably only possible in the cloud, makes discussions of east-west concerns seem underwhelming and rooted in constraints of the past.
Our second use case, enabling friction-free access to internal applications for your workforce, is all about improving workforce mobility without compromising security. Traditionally these applications have existed behind a strong VPN front door. However, VPNs can be expensive to scale and aren’t necessarily compatible with the full array of mobile devices that the modern workforce demands. The objective in this case is to make the locks on the individual applications so good that you can eliminate the VPN-based front door. To achieve this, our customers have told us that they want a range of technical solutions to choose from according to their industry, risk tolerance, developer maturity, and other factors. At one end of the spectrum, we have many customers who prefer to use desktop as a service—Amazon Workspaces—or application as a service—Amazon AppStream 2.0—models to provide a powerful and flexible pixel proxy approach to Zero Trust. Traditional security controls are applied to those intermediary virtual devices, and then any user with a PC, tablet, or HTML5 client can reach those virtualized desktops or applications over the internet—or behind additional network controls and perimeters, if they so desire—to provide a rich, desktop-like experience without having to worry about the security of the final device in the hands of the user. Similarly, customers have asked for a better way to access their enterprise applications securely from mobile phones without deploying mobile device management or other such often cumbersome and expensive technologies. To meet that requirement, we launched Amazon WorkLink, providing a secure proxy service that renders complex web applications in the AWS cloud. Amazon WorkLink streams only pixels—and a very minimal amount of JavaScript for interactivity—to mobile phones. No sensitive enterprise data is ever stored or cached on the mobile device.
At the other end of the spectrum, we have customers who want to connect their internal web applications directly to the internet. For these customers, the combination of AWS Shield, AWS WAF, and Application Load Balancer with OpenID Connect (OIDC) authentication provides a fully managed identity-aware network protection stack. Shield provides managed DDoS protection services that provide always-on detection and automatic inline mitigations that minimize application downtime and latency. AWS WAF is a web application firewall that lets you monitor and protect web requests before they reach your infrastructure using your desired combination of rule groups provided by AWS, the AWS Marketplace, or your own custom ones. By enabling authentication in Application Load Balancer—beyond the normal load balancing capabilities—you can directly integrate with your existing identity provider (IdP) to offload the work of authenticating users, and to leverage the existing capabilities within your IdP—such as strong authentication, device posture assessment, conditional access, and policy enforcement. Using this combination, your internal custom applications quickly become just as flexible as SaaS applications, allowing your workforce to enjoy the same work-anywhere flexibility as SaaS while unifying your application portfolio under a common security model powered by modern identity standards.
Our third use case—securing digital transformation projects such as IoT—is markedly different from the first two. Consider a connected vehicle, relaying a critical stream of instrumentation over mobile networks and the internet into a cloud based analytics environment for processing and insights. These workloads have always existed entirely outside the traditional enterprise network, and require a security model that accounts for that situation. The family of AWS IoT services provides scalable solutions for issuing unique device identities to every device in your fleet, and then using those identities and their associated access control policies to securely control how they communicate and interact with the cloud. The security of these devices can be easily monitored and maintained with AWS IoT Device Defender, over-the-air software updates, and even entire operating system upgrades—now built in to FreeRTOS—to keep devices safe and secure over time. Moving forward, as more and more IT workloads move closer to the edge to minimize latency and improve user experiences, the prevalence of this use case will continue to expand, even if it isn’t applicable to your business today.
It’s still Day 1
We hope this post has helped communicate our vision for Zero Trust, and highlighted how we believe that our underlying security principles and advancing capabilities represent a bar-raising security model both for the AWS cloud and for the environments that our customers build on top of our services.
At Amazon we obsess over customers and their needs, so our job is never done. We have lots more capabilities we want to build, and lots more guidance still to offer. We look forward to your feedback and to continuing the journey together—reflecting the words and core vision of our founder, Jeff Bezos: “It’s still Day 1.”
If you have feedback about this post, submit comments in the Comments section below.
Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.
The collective thoughts of the interwebz
By continuing to use the site, you agree to the use of cookies. more information
The cookie settings on this website are set to "allow cookies" to give you the best browsing experience possible. If you continue to use this website without changing your cookie settings or you click "Accept" below then you are consenting to this.
