Tag Archives: Multiple accounts

How to automate SAML federation to multiple AWS accounts from Microsoft Azure Active Directory

Post Syndicated from Sepehr Samiei original https://aws.amazon.com/blogs/security/how-to-automate-saml-federation-to-multiple-aws-accounts-from-microsoft-azure-active-directory/

You can use federation to centrally manage access to multiple AWS accounts using credentials from your corporate directory. Federation is the practice of establishing trust between a system acting as an identity provider and other systems, often called service providers, that accept authentication tokens from that identity provider. Amazon Web Services (AWS) supports open federation standards, including Security Assertion Markup Language (SAML) 2.0, to make it easier for the systems and service providers to interact. Here, I’m going to explain how to automate federation between AWS Identity and Access Management (IAM) in multiple AWS accounts and Microsoft Azure Active Directory (Azure AD). I’ll be following the same general patterns that allow SAML federation to AWS from any other identity provider that supports SAML 2.0, but I’m also adding some automation that is specific to Azure AD. I’ll show you how to perform the initial configuration, and then how to automatically keep Azure AD in sync with your AWS IAM roles.

AWS supports any SAML 2.0-compliant identity provider. If you’re interested in configuring federated access using an identity provider other than Azure AD, these links might be useful:

In this post, I’m going to focus on the nuances of using Azure AD as a SAML identity provider for AWS. The approach covered here gives you a solution that makes this option easier and adheres to AWS best practices. The primary objectives of this step-by-step walkthrough, along with the accompanying packaged solution, are:

  • Support any number of AWS accounts and roles, making it easier to scale.
  • Keep configuration of both sides updated automatically.
  • Use AWS short-term credentials so you don’t have to store your credentials with your application. This enhances your security posture because these credentials are dynamically generated, securely delivered, naturally expire after their limited lifetime, and are automatically rotated for you.

Solution overview

I’ll discuss:

  • How to configure Microsoft Azure Active Directory and show the steps needed to prepare it for federation with AWS.
  • How to configure AWS IAM Identity Providers and Roles, and explain the steps you need to carry out in your AWS accounts.
  • How to automatically import your AWS configuration into the Azure AD SSO app for AWS.

The following diagram shows the high-level flow of SAML authentication and how your users will be federated into the AWS Management console:
 

Figure 1: SAML federation between Azure AD and AWS

Figure 1: SAML federation between Azure AD and AWS

Key to the interactions in the diagram

  1. User opens a browser and navigates to Azure AD MyApps access panel (myapps.microsoft.com).
  2. If the user isn’t authenticated, she’ll be redirected to the login endpoint for authentication.
  3. User enters her credentials and the login endpoint will verify them against Azure AD tenant.
  4. Upon successful login, user will be redirected back to the access panel.
  5. User will see the list of available applications, including the AWS Console app, and will select the AWS Console app icon.
  6. The access panel redirects the user to the federated application endpoint, passing the application ID of the AWS SSO app.
  7. The AWS SSO application queries Azure AD and generates a SAML assertion, including all the AWS IAM roles assigned to the user.
  8. SAML assertion is sent back to the user.
  9. User is redirected to AWS federation endpoint, presenting the SAML assertion. The AWS federation endpoint verifies the SAML assertion. The user will choose which of their authorized roles they currently want to operate in. Note: If there’s only one role included, the selection is automatic.
  10. The AWS federation endpoint invokes the AssumeRoleWithSAML API of AWS Security Token Service (STS) and exchanges the SAML token with temporary AWS IAM credentials.
  11. Temporary IAM credentials are used to formulate a specific AWS Console URL that’s passed back to the client browser.
  12. User is redirected to AWS Management Console with permissions of the assumed role.

Automated solution components and flow

At the core of this automated solution, there’s a Docker container that runs inside an AWS ECS Fargate task. The container includes a number of PowerShell scripts that iterate through your IAM Roles, find roles that are associated with the Identity Provider of Azure AD, and update the Azure AD SSO app manifest with the necessary values.

The Fargate task is invoked through an AWS Lambda function that’s scheduled through a CloudWatch Rule to run with the frequency you specify during setup.

All of these components require a number of parameters to run correctly, and you provide these parameters through the setup.ps1 script. The setup.ps1 script is run once and acquires all required parameters from you. It then stores these parameters with encryption inside the SSM Parameter Store. Azure credentials are stored in AWS Secrets Manager. This means you could even go another step further and use Secrets Manager lifecycle management capabilities to automatically rotate your Azure credentials. For encryption of Azure credentials, the template creates a new KMS key, exclusive to this application. If you prefer to use an existing key or a Customer Managed Key (CMK), you can modify the CloudFormation template, or simply pass your own key name to the setup.ps1 script.

The following diagram shows all components of the solution:
 

Figure 2: Solution architecture

Figure 2: Solution architecture

  1. You’ll want any ongoing changes in AWS IAM roles to be replicated into Azure AD. Therefore, you need to have the update task run periodically. A CloudWatch Rule triggers an event and an AWS Lambda Function starts running as a result of this event.
  2. The Lambda Function runs an ECS Fargate Task.
  3. The ECS Task is associated with a Task Role with permission to fetch parameters from Systems Manager (SSM) Parameter Store and Secrets Manager. The task will request parameters from SSM PS, and SSM PS decrypts parameter values using the associated key in AWS Key Management Service (KMS). Azure credentials are securely stored in AWS Secrets Manager.
  4. Fargate Task queries AWS Organizations and gets a list of child accounts. It then constructs cross-account role ARNs. The ECS Task then assumes those cross-account roles and iterates through all IAM roles in each account to find those associated with your IdP for Azure AD.
  5. The ECS Task connects to the Azure AD SSO application and retrieves the existing manifest. Notice that, although you manually retrieved the manifest file during setup, it still needs to be fetched again every time to make sure it’s the latest version. The one you manually downloaded is used to retrieve parameters needed for setup, such as the application identifier or entity ID.
  6. ECS Task stores the existing manifest as a backup in a highly-durable S3 bucket. In case anything goes wrong, the last working state of the application manifest is always available in the S3 bucket. These files are stored with the exact time of their retrieval as their file name. You can find the correct version based on the point in time it was retrieved.
  7. The ECS Task generates a new manifest based on your AWS account/roles as inspected in the preceding steps. It uses the Azure AD credentials retrieved from AWS Secrets Manager and uses them to update the Azure AD SSO app with the new manifest. It also creates any required Azure AD Groups according to the specified custom naming convention. This makes it easier for the Azure AD administrator to map Azure AD users to AWS roles and entitle them to assume those roles.

Prerequisites

To start, download a copy of the sample code package.

You must have AWS Organizations enabled on all of your accounts to take advantage of this solution’s automation. Using AWS Organizations, you can configure one of your accounts as the root account and all other accounts will join your organization as child accounts. The root account will be trusted by all child accounts, so you can manage your child account resources from your root account. This trust is enabled using a role in each of your child accounts. AWS Organizations creates a default role with full permissions on child accounts that are directly created using AWS Organizations. Best practice is to delete this default role and create one with privileges restricted to your requirements. A sample role, named AWSCloudFormationStackSetExecutionRole, is included in cross-account-role-cfn.json
of my code package. You should modify this template based on your requirements.

Setup steps

In following sections, I’ll show the steps to setup federation and deploy the automation package. First, I’ll show the steps to prepare Azure Active Directory for federation. After that, you’ll see how you can configure all of your AWS accounts from a central place, regardless of the number of your accounts. The last step is to deploy the automation package in your master AWS account to automatically handle ongoing changes as you go.

Step 1: Configure Microsoft Azure Active Directory

You need to create two resources on your Azure AD tenant: a User and an Enterprise Application.

First thing you need for accessing Azure AD is an Azure AD user. In following the principle of least privilege, you want a user that can only manipulate the SSO application. Azure AD users with the directory role of User will only have access to resources they “own.” Therefore, you can create a new user specifically for this purpose and assign it as the owner of your SSO app. This user will be used by the automation to access Azure AD and update the SSO app.

Here’s how you can create a user with the directory role of User (default):

  1. Open Azure Portal.
  2. Open Azure Active Directory.
  3. In the left pane, select Users.
  4. In the Manage pane, select All users.
  5. Select New user.
  6. Enter values for the Name and User name fields.
  7. Select the Show Password box and note the auto-generated password for this user. You will need it when you change the password.
  8. Select Create.
  9. Open a browser window and go to https://login.microsoftonline.com.
  10. Log in with the new user. You’ll be prompted to change your password. Note the new password so you don’t forget it.

Next, create an Enterprise Application from the Azure AD application gallery:

  1. Open Azure Portal.
  2. Open Azure Active Directory.
  3. In the Manage pane, select Enterprise applications.
  4. Select New application.
  5. In the gallery text box, type AWS.
  6. You’ll see an option with the name Amazon Web Services (AWS). Select that application. Make sure you don’t choose the other option with the name “AWS Console.” That option uses an alternate integration method that isn’t relevant to this post.
  7.  

    Figure 3: Select "Amazon Web Services (AWS)

    Figure 3: Select “Amazon Web Services (AWS)

  8. Select Add. You can change the name to any name you would prefer.
  9. Open the application using this path: Azure Portal > Azure Active Directory > Enterprise Applications > All Applications > your application name (for example, “Amazon Web Services (AWS)”).
  10. From left pane, select Single Sign-on, and then set Single Sign-on mode to SAML-based Sign-on.
  11. The first instance of the app is pre-integrated with Azure AD and requires no mandatory URL settings. However, if you previously created a similar application, you’ll see this:
  12.  

    Figure 4: Azure AD Application Identifier

    Figure 4: Azure AD Application Identifier

  13. If you see the red “Required” value in the Identifier field, select the Edit button and enter a value for it. This can be any value you prefer (the default is https://signin.aws.amazon.com/saml), but it has to be unique within your Azure AD tenant. If you don’t see the Identifier field, it means it’s already prepopulated and you can proceed with the default value. However, if for any reason you prefer to have a custom Identifier value, you can select the Show advanced URL settings checkbox and enter the preferred value.
  14. In the User Attributes section, select the Edit button.
  15. You need to tell Azure AD what SAML attributes and values are expected and accepted on the AWS side. AWS requires two mandatory attributes in any incoming SAML assertion. The Role attribute defines which roles the federated user is allowed to assume. The RoleSessionName attribute defines the specific, traceable attribute for the user that will appear in AWS CloudTrail logs. Role and RoleSessionName are mandatory attributes. You can also use the optional attribute of SessionDuration to specify how long each session will be valid until the user is requested to get a new token. Add the following attributes to the User Attributes & Claims section in the Azure AD SSO application. You can also remove existing default attributes, if you want, because they’ll be ignored by AWS:

    Name (case-sensitive)ValueNamespace (case-sensitive)Required or optional?
    RoleSessionNameuser.userprincipalname
    (this will show logged in user ID in AWS portal, if you want user name, replace it with user.displayName)
    https://aws.amazon.com/SAML/AttributesRequired
    Roleuser.assignedroleshttps://aws.amazon.com/SAML/AttributesRequired
    SessionDurationAn integer between 900 seconds (15 minutes) and 43200 seconds (12 hours).https://aws.amazon.com/SAML/AttributesOptional

    Note: I assume that you use users that are directly created within your Azure AD tenant. If you’re using an external user such as a Hotmail, Live, or Gmail account for proof-of-concept purposes, RoleSessionName should be set to user.mail instead.

  16. As a good practice, when it approaches its expiration date, you can rotate your SAML certificate. For this purpose, Azure AD allows you to create additional certificates, but only one certificate can be active at a time. In the SAML Signing Certificate section, make sure the status of this certificate is Active, and then select Federation Metadata XML to download the XML document.
  17. Download the Metadata XML file and save it in the setup directory of the package you downloaded in the beginning of this walkthrough. Make sure you save it with file extension of .xml.
  18. Open Azure Portal > Azure Active Directory > App Registrations > your application name (for example, “Amazon Web Services (AWS)”). If you don’t see your application in the list on the App Registrations page, select All apps from the drop-down list on top of that page and search for it.
  19. Select Manifest. All Azure AD applications are described as a JavaScript Object Notification (JSON) document called manifest. For AWS, this manifest defines all AWS to Azure AD role mappings. Later, we’ll be using automation to generate updates to this file.
     
    Figure 5: Azure AD Application Manifest

    Figure 5: Azure AD Application Manifest

  20. Select Download to download the app manifest JSON file. Save it in the setup directory of the package you downloaded in the beginning of this walkthrough. Make sure you save it with file extension of .json.
  21. Now, back on your registered app, select Settings.
  22. In the Settings pane, select Owners.
     
    Figure 6: Application Owner

    Figure 6: Application Owner

  23. Select Add owner and add the user you created previously as owner of this application. Adding the Azure AD user as owner enables the user to manipulate this object. Since this application is the only Azure AD resource owned by our user, it means we’re enforcing the principle of least privilege on Azure AD side.

At this point, we’re done with the initial configuration of Azure AD. All remaining steps will be performed in your AWS accounts.

Step 2: Configure AWS IAM Identity Providers and Roles

In the previous section, I showed how to configure the Azure AD side represented in the Solution architecture in Figure 1. This section explains the AWS side.

As seen in Figure 1, enabling SAML federation in any AWS account requires two types of AWS IAM resources:

You’ll have to create these two resources in all of your AWS accounts participating in SAML federation. There are various options for doing this. You can:

  • Manually create IAM IdP and Roles using AWS Management Console. For one or two accounts, this might be the easiest way. But as the number of your AWS accounts and roles increase, this method becomes more difficult.
  • Use AWS CLI or AWS Tools for PowerShell. You can use these tools to write automation scripts and simplify both creation and maintenance of your roles.
  • Use AWS CloudFormation. CloudFormation templates enable structured definition of all resources and minimize the effort required to create and maintain them.

Here, I’m going to use CloudFormation and show how it can help you create up to thousands of roles in your organization, if you need that many.

Managing multiple AWS accounts from a root account

AWS CloudFormation simplifies provisioning and management on AWS. You can create templates for the service or application architectures you want and have AWS CloudFormation use those templates for quick and reliable provisioning of the services or applications (called “stacks“). You can also easily update or replicate the stacks as needed. Each stack is deployed in a single AWS account and a specific AWS Region. For example, you can write a template that defines your organization roles in AWS IAM and deploy it in your first AWS account and US East (N.Virginia) region.

But if you have hundreds of accounts, it wouldn’t be easy, and if you have time or budget constraints, sometimes not even possible to manually deploy your template in all accounts. Ideally, you’d want to manage all your accounts from a central place. AWS Organizations is the service that gives you this capability.

In my GitHub package there is a CloudFormation template named cross-account-roles-cfn.json. It’s located under the cfn directory. This template includes two cross-account roles. The first one is a role for cross-account access with the minimum required privileges for this solution that trusts your AWS Organizations master account. This role is used to deploy AWS IAM Identity Provider (IdP) for Azure AD and all SAML federation roles, trusting that IdP within all of your AWS child accounts. The second one is used by the automation to inspect your AWS accounts (through describe calls) and keep the Azure AD SSO application updated. I’ve created two roles to ensure that each component executes with the least privilege required. To recap, you’ll have two cross account roles for two different purposes:

  1. A role with full IAM access and Lambda execution permissions. This one is used for creation and maintenance of SAML IdP and associated IAM roles in all accounts.
  2. A role with IAM read-only access. This one is used by the update task to read and detect any changes in your federation IAM roles so it can update Azure AD SSO app with those changes.

You can deploy CloudFormation templates in your child accounts using CloudFormation StackSets. Log in to your root account, go to the CloudFormation console, and select StackSets.

Select Template is ready, select Upload a template file, and then select the cross-account-roles-cfn.json template to deploy it in all of your accounts. AWS IAM is a global service, so it makes no difference which region you choose for this template. You can select any region, such as us-east-1.
 

Figure 7: Upload template to StackSets console

Figure 7: Upload template to StackSets console

This template includes a parameter prompting you to enter root account number. For instructions to find your account number, see this page.

If you create your child accounts through AWS Organizations, you’ll be able to directly deploy StackSets in those child accounts. But, if you add existing accounts to you organization, you have to first manually deploy
cross-account-roles-cfn.json in your existing accounts. This template includes the IAM role and policies needed to enable your root account to execute StackSets on it.

Configure the SAML Identity Provider and Roles

A sample template to create your organization roles as SAML federation IAM roles is included in the saml-roles.json file in the same cfn directory. This template includes the SAML IdP and three sample roles trusting that IdP. The IdP is implemented as an AWS Lambda-backed CloudFormation custom resource. Included roles are samples using AWS IAM Job Functions for Administrator, Observer, and DBA. Modify this template by adding or removing roles as needed in your organization.

If you need different roles in some of your accounts, you’ll have to create separate copies of this template and modify them accordingly. From the CloudFormation StackSets console, you can choose the accounts to which your template should be deployed.

The last modification to make is on the IdentityProvider custom resource. It includes a <Metadata> property. Its value is defined as <MetadataDocument>. You’d have to replace the value with the content of the SAML certificate metadata XML document that you previously saved in the setup directory (see the Configure Microsoft Azure Active Directory section above). You’ll need to escape all of the quotation marks (“) in the XML string with a backslash (\). If you don’t want to do this manually, you can copy the saml-roles.json template file in the setup directory and as you follow the remainder of instructions in this post, my setup script will do that for you.

Step 3: Updating Azure AD from the root AWS account

The third and last template in the cfn directory is setup-env-cfn-template.json. You have to deploy this template only in your root account. This template creates all the components in your root account, as shown in Figure 8. These are resources needed to run the update task and keep Azure AD SSO App updated with your IAM roles. In addition, it also creates a temporary EC2 instance for initial configuration of that update task. The update task uses AWS Fargate, a serverless service that allows you to run Docker containers in AWS. You have to deploy the setup-env-cfn-template.json template in a region where Fargate is available. Check the AWS Region Table to make sure Fargate is available in your target region. Follow these steps to deploy the stack:

  1. Log in to your root account and open the CloudFormation console page.
  2. Select Create Stack, upload the setup-env-cfn-template.json file, and then select Next.
  3. Enter a stack name, such as aws-iam-aad. The stack name must be all lowercase letters. The template uses the stack name to create an S3 bucket, and because S3 does not allow capital letters, if you choose a stack name containing capital letters, the stack creation will fail. The stack name is also used as the appName parameter in all scripts, and all Parameter Store parameter names are prefixed with it.
  4. Enter and select values for the following parameters:
    1. azureADTenantName: You can get the Azure Active Directory Tenant Name from Azure Portal. Go to the Azure Active Directory Overview page and the tenant name should appear at the top of the page. During setup, this is used as the value for the parameter.
    2. ExecFrequency is the time period for the update task to run. For example, if you enter 30, every 30 minutes Azure AD will be updated with any changes in IAM roles of your AWS accounts.
    3. KeyName is a key pair that is used for login and accessing the EC2 instance. You’ll need to have a key pair created before deploying this template. To create a key pair, follow these instructions: Amazon EC2 Key Pairs. Also, for more convenience, if you’re using a MAC or Linux, you can copy your private key in the setup directory. Don’t forget to run chmod 600 <key name> to change the permissions on the key.
    4. NamingConvention is used to map AWS IAM roles to Azure AD roles. The default naming convention is: “AWS {0} – {1}”. The value of {0} is your account number. The value of {1} is the name of your IAM Role.
    5. SSHLocation is used in a Security Group that restricts access to the setup EC2 instance. You only need this instance for initial setup; therefore, the best practice and most secure option is to change this value to your specific IP address. In any case, make sure you only allow access to your internal IP address range.
    6. Subnet is the VPC subnet in which you want the update task to run. This subnet must have egress (outgoing) internet connectivity. The update task needs this to reach Azure AD Graph API endpoints.
       
      Figure 8: Enter parameters for automation stack

      Figure 8: Enter parameters for automation stack

Once you deploy this template in CloudFormation and the associated stack is successfully created, you can get the IP address of the setup EC2 instance from the Output tab in CloudFormation. Now, follow the steps below to complete the setup.

Note: At this point, in addition to all the files already included in the original package, you have two additional, modified files in the setup directory:

  • The SAML Certificate XML file from Azure AD
  • The App Manifest JSON file from Azure AD

Make sure you have following information handy. This info is required in some of the steps:

Now, follow these steps to complete the setup:

  1. If you’re using Mac, Linux, or UNIX, run the initiate_setup.sh script in the setup directory and, when prompted, provide the IP address from the previous procedure. It will copy all the required files to the target setup EC2 instance and automatically take you to the setup.ps1 script. Now, skip to step 3 below.
  2. If you’re using Windows on your local computer, use your favorite tool (such as WinSCP) to copy both setup and docker directories from your local computer to the /home/ec2-user/scripts directory on the target EC2 instance.
  3. Once copied, use your favorite SSH tool to log in to the target setup EC2 instance. For example, you can use PuTTY for this purpose. As soon as you log in, Setup.ps1 will automatically run for you.
  4. Setup.ps1 is interactive. It will prompt for the path to the three files you saved in the setup directory, and also for your Azure AD user credentials. Use the credentials of the user you created in step 1 of the Configure Microsoft Azure Active Directory section. The script will perform following tasks:
    1. Store Azure AD credentials securely in AWS Secrets Manager. The script also extracts necessary values out of the three input files and stores them as additional parameters in AWS Systems Manager (SSM) Parameter Store.

      Important: The credentials of your Azure user will be stored in AWS Secrets Manager. You must make sure that access to Secrets Manager is restricted to users who are also authorized to retrieve these credentials.

    2. Create a Docker image and push it into an AWS Elastic Container Registry (ECR) repository that’s created as part of the CloudFormation template.
    3. The script checks if saml-roles.json is available in setup directory. If it’s available, the script will replace the value of the Metadata property in the IdP custom resource with content of the SAML metadata XML file. It also generates a text file containing a comma-separated list of all your child accounts, extracting account numbers from cross-account-roles-cfn.json. Both of these are copied to the S3 bucket that is created as part of the template. You can use these at any time to deploy, maintain, and manage your SAML roles in child accounts using CloudFormation StackSets.
    4. If saml-roles.json is available, the script will prompt whether you want it to deploy your roles on your behalf. If you select yes (“y“), it will immediately deploy the template in all child accounts. You can also select no (“n“), if you prefer to do this at another time, or if you need different templates and roles in some accounts.
  5. Once the script executes and successfully completes, you should terminate the setup EC2 instance.

You’ve now completed setting up federation on both sides. All AWS IAM roles that trust an IdP with the SAML certificate of your Azure AD (the Metadata XML file) will now automatically be replicated into your Azure AD tenant. This will take place with the frequency you have defined. Therefore, if you have set the ExecFrequency parameter to “30“, after 30 minutes you’ll see the roles replicated in Azure AD.

But to enable your users to use this federation, you have to entitle them to assume roles, which is what I’ll cover in the next section.

Entitling Azure AD users to assume AWS Roles

  1. Open Azure Portal > Azure Active Directory >
    Enterprise applications > All applications > (your application name) > Users and groups.
  2. Select Add user.
  3. In the Users and groups pane, select one of your Azure AD users (or groups), and then select Select.
  4. Select the Select role pane and, on the right hand side, you should now see your AWS roles listed.

You can add and map Azure AD users or groups to AWS IAM roles this way. By adding users to these groups, you’re giving them access to those roles in AWS through pre-established trust. In the case of Groups, any Azure AD users inside that Group will have SSO access to the AWS Console and permitted to assume AWS roles/accounts associated with their Azure AD Group. Azure AD users who are authenticated against login.microsoftonline.com can go to their Access Panel (myapps.microsoft.com) and select the AWS app icon.

Application maintenance

Most of the time, you will not need to do anything else because the Fargate task will execute on each interval and keep the Azure AD manifest aligned with your AWS accounts and roles. However, there are two situations that might require you to take action:

  • If you rotate your Azure AD SAML certificate
  • If you rotate the Azure user credentials used for synchronization

You can use AWS Secrets Manager lifecycle management capabilities to automate the process for the second case. Otherwise, in the event of either of these two situations, you can modify the corresponding values using the AWS Systems Manager Parameter Store and Secrets Manager consoles. Open the Parameter Store console and find parameters having names prefixed with your setup-env-cfn-template.json stack name (you entered this name when you were creating the stack). In case you rotate your Azure AD SAML certificate, you should also update all of your IdP resources in AWS accounts to use the new resource. Here again, StackSets can do the heavy-lifting for you. Use the same saml-roles.json template to update all of your Stack Instances through CloudFormation. You’ll have to replace the Metadata property value with content of the new certificate, and replace quotation mark characters (“) with escaped quotes (\”).

Summary

I’ve demonstrated how to set up and configure SAML federation and SSO using Azure Active Directory to AWS Console following these principles and requirements:

  • Using security best practices to keep both sides of federation (AWS and Azure) secure
  • Saving time and effort by automating the manual effort needed to synchronize two sides of federation
  • Keeping operation cost to a minimum through a serverless solution

If you have feedback about this blog post, submit comments in the Comments section below. If you have questions about this blog post, start a new thread in the forums.

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

Sepehr Samiei

Sepehr is currently a Senior Microsoft Tech Specialized Solutions Architect at AWS. He started his professional career as a .Net developer, which continued for more than 10 years. Early on, he quickly became a fan of cloud computing and he loves to help customers utilise the power of Microsoft tech on AWS. His wife and daughter are the most precious parts of his life.

How to centralize DNS management in a multi-account environment

Post Syndicated from Mahmoud Matouk original https://aws.amazon.com/blogs/security/how-to-centralize-dns-management-in-a-multi-account-environment/

In a multi-account environment where you require connectivity between accounts, and perhaps connectivity between cloud and on-premises workloads, the demand for a robust Domain Name Service (DNS) that’s capable of name resolution across all connected environments will be high.

The most common solution is to implement local DNS in each account and use conditional forwarders for DNS resolutions outside of this account. While this solution might be efficient for a single-account environment, it becomes complex in a multi-account environment.

In this post, I will provide a solution to implement central DNS for multiple accounts. This solution reduces the number of DNS servers and forwarders needed to implement cross-account domain resolution. I will show you how to configure this solution in four steps:

  1. Set up your Central DNS account.
  2. Set up each participating account.
  3. Create Route53 associations.
  4. Configure on-premises DNS (if applicable).

Solution overview

In this solution, you use AWS Directory Service for Microsoft Active Directory (AWS Managed Microsoft AD) as a DNS service in a dedicated account in a Virtual Private Cloud (DNS-VPC).

The DNS service included in AWS Managed Microsoft AD uses conditional forwarders to forward domain resolution to either Amazon Route 53 (for domains in the awscloud.com zone) or to on-premises DNS servers (for domains in the example.com zone). You’ll use AWS Managed Microsoft AD as the primary DNS server for other application accounts in the multi-account environment (participating accounts).

A participating account is any application account that hosts a VPC and uses the centralized AWS Managed Microsoft AD as the primary DNS server for that VPC. Each participating account has a private, hosted zone with a unique zone name to represent this account (for example, business_unit.awscloud.com).

You associate the DNS-VPC with the unique hosted zone in each of the participating accounts, this allows AWS Managed Microsoft AD to use Route 53 to resolve all registered domains in private, hosted zones in participating accounts.

The following diagram shows how the various services work together:
 

Diagram showing the relationship between all the various services

Figure 1: Diagram showing the relationship between all the various services

 

In this diagram, all VPCs in participating accounts use Dynamic Host Configuration Protocol (DHCP) option sets. The option sets configure EC2 instances to use the centralized AWS Managed Microsoft AD in DNS-VPC as their default DNS Server. You also configure AWS Managed Microsoft AD to use conditional forwarders to send domain queries to Route53 or on-premises DNS servers based on query zone. For domain resolution across accounts to work, we associate DNS-VPC with each hosted zone in participating accounts.

If, for example, server.pa1.awscloud.com needs to resolve addresses in the pa3.awscloud.com domain, the sequence shown in the following diagram happens:
 

How domain resolution across accounts works

Figure 2: How domain resolution across accounts works

 

  • 1.1: server.pa1.awscloud.com sends domain name lookup to default DNS server for the name server.pa3.awscloud.com. The request is forwarded to the DNS server defined in the DHCP option set (AWS Managed Microsoft AD in DNS-VPC).
  • 1.2: AWS Managed Microsoft AD forwards name resolution to Route53 because it’s in the awscloud.com zone.
  • 1.3: Route53 resolves the name to the IP address of server.pa3.awscloud.com because DNS-VPC is associated with the private hosted zone pa3.awscloud.com.

Similarly, if server.example.com needs to resolve server.pa3.awscloud.com, the following happens:

  • 2.1: server.example.com sends domain name lookup to on-premise DNS server for the name server.pa3.awscloud.com.
  • 2.2: on-premise DNS server using conditional forwarder forwards domain lookup to AWS Managed Microsoft AD in DNS-VPC.
  • 1.2: AWS Managed Microsoft AD forwards name resolution to Route53 because it’s in the awscloud.com zone.
  • 1.3: Route53 resolves the name to the IP address of server.pa3.awscloud.com because DNS-VPC is associated with the private hosted zone pa3.awscloud.com.

Step 1: Set up a centralized DNS account

In previous AWS Security Blog posts, Drew Dennis covered a couple of options for establishing DNS resolution between on-premises networks and Amazon VPC. In this post, he showed how you can use AWS Managed Microsoft AD (provisioned with AWS Directory Service) to provide DNS resolution with forwarding capabilities.

To set up a centralized DNS account, you can follow the same steps in Drew’s post to create AWS Managed Microsoft AD and configure the forwarders to send DNS queries for awscloud.com to default, VPC-provided DNS and to forward example.com queries to the on-premise DNS server.

Here are a few considerations while setting up central DNS:

  • The VPC that hosts AWS Managed Microsoft AD (DNS-VPC) will be associated with all private hosted zones in participating accounts.
  • To be able to resolve domain names across AWS and on-premises, connectivity through Direct Connect or VPN must be in place.

Step 2: Set up participating accounts

The steps I suggest in this section should be applied individually in each application account that’s participating in central DNS resolution.

  1. Create the VPC(s) that will host your resources in participating account.
  2. Create VPC Peering between local VPC(s) in each participating account and DNS-VPC.
  3. Create a private hosted zone in Route 53. Hosted zone domain names must be unique across all accounts. In the diagram above, we used pa1.awscloud.com / pa2.awscloud.com / pa3.awscloud.com. You could also use a combination of environment and business unit: for example, you could use pa1.dev.awscloud.com to achieve uniqueness.
  4. Associate VPC(s) in each participating account with the local private hosted zone.

The next step is to change the default DNS servers on each VPC using DHCP option set:

  1. Follow these steps to create a new DHCP option set. Make sure in the DNS Servers to put the private IP addresses of the two AWS Managed Microsoft AD servers that were created in DNS-VPC:
     
    The "Create DHCP options set" dialog box

    Figure 3: The “Create DHCP options set” dialog box

     

  2. Follow these steps to assign the DHCP option set to your VPC(s) in participating account.

Step 3: Associate DNS-VPC with private hosted zones in each participating account

The next steps will associate DNS-VPC with the private, hosted zone in each participating account. This allows instances in DNS-VPC to resolve domain records created in these hosted zones. If you need them, here are more details on associating a private, hosted zone with VPC on a different account.

  1. In each participating account, create the authorization using the private hosted zone ID from the previous step, the region, and the VPC ID that you want to associate (DNS-VPC).
     
    aws route53 create-vpc-association-authorization –hosted-zone-id <hosted-zone-id> –vpc VPCRegion=<region>,VPCId=<vpc-id>
     
  2. In the centralized DNS account, associate DNS-VPC with the hosted zone in each participating account.
     
    aws route53 associate-vpc-with-hosted-zone –hosted-zone-id <hosted-zone-id> –vpc VPCRegion=<region>,VPCId=<vpc-id>
     

After completing these steps, AWS Managed Microsoft AD in the centralized DNS account should be able to resolve domain records in the private, hosted zone in each participating account.

Step 4: Setting up on-premises DNS servers

This step is necessary if you would like to resolve AWS private domains from on-premises servers and this task comes down to configuring forwarders on-premise to forward DNS queries to AWS Managed Microsoft AD in DNS-VPC for all domains in the awscloud.com zone.

The steps to implement conditional forwarders vary by DNS product. Follow your product’s documentation to complete this configuration.

Summary

I introduced a simplified solution to implement central DNS resolution in a multi-account environment that could be also extended to support DNS resolution between on-premise resources and AWS. This can help reduce operations effort and the number of resources needed to implement cross-account domain resolution.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the AWS Directory Service forum or contact AWS Support.

Want more AWS Security news? Follow us on Twitter.

Central Logging in Multi-Account Environments

Post Syndicated from matouk original https://aws.amazon.com/blogs/architecture/central-logging-in-multi-account-environments/

Centralized logging is often required in large enterprise environments for a number of reasons, ranging from compliance and security to analytics and application-specific needs.

I’ve seen that in a multi-account environment, whether the accounts belong to the same line of business or multiple business units, collecting logs in a central, dedicated logging account is an established best practice. It helps security teams detect malicious activities both in real-time and during incident response. It provides protection to log data in case it is accidentally or intentionally deleted. It also helps application teams correlate and analyze log data across multiple application tiers.

This blog post provides a solution and building blocks to stream Amazon CloudWatch log data across accounts. In a multi-account environment this repeatable solution could be deployed multiple times to stream all relevant Amazon CloudWatch log data from all accounts to a centralized logging account.

Solution Summary 

The solution uses Amazon Kinesis Data Streams and a log destination to set up an endpoint in the logging account to receive streamed logs and uses Amazon Kinesis Data Firehose to deliver log data to the Amazon Simple Storage Solution (S3) bucket. Application accounts will subscribe to stream all (or part) of their Amazon CloudWatch logs to a defined destination in the logging account via subscription filters.

Below is a diagram illustrating how the various services work together.


In logging an account, a Kinesis Data Stream is created to receive streamed log data and a log destination is created to facilitate remote streaming, configured to use the Kinesis Data Stream as its target.

The Amazon Kinesis Data Firehose stream is created to deliver log data from the data stream to S3. The delivery stream uses a generic AWS Lambda function for data validation and transformation.

In each application account, a subscription filter is created between each Amazon CloudWatch log group and the destination created for this log group in the logging account.

The following steps are involved in setting up the central-logging solution:

  1. Create an Amazon S3 bucket for your central logging in the logging account
  2. Create an AWS Lambda function for log data transformation and decoding in logging account
  3. Create a central logging stack as a logging-account destination ready to receive streamed logs and deliver them to S3
  4. Create a subscription in application accounts to deliver logs from a specific CloudWatch log group to the logging account destination
  5. Create Amazon Athena tables to query and analyze log data in your logging account

Creating a log destination in your logging account

In this section, we will setup the logging account side of the solution, providing detail on the list above. The example I use is for the us-east-1 region, however any region where required services are available could be used.

It’s important to note that your logging-account destination and application-account subscription must be in the same region. You can deploy the solution multiple times to create destinations in all required regions if application accounts use multiple regions.

Step 1: Create an S3 bucket

Use the CloudFormation template below to create S3 bucket in logging account. This template also configures the bucket to archive log data to Glacier after 60 days.


{
  "AWSTemplateFormatVersion":"2010-09-09",
  "Description": "CF Template to create S3 bucket for central logging",
  "Parameters":{

    "BucketName":{
      "Type":"String",
      "Default":"",
      "Description":"Central logging bucket name"
    }
  },
  "Resources":{
                        
   "CentralLoggingBucket" : {
      "Type" : "AWS::S3::Bucket",
      "Properties" : {
        "BucketName" : {"Ref": "BucketName"},
        "LifecycleConfiguration": {
            "Rules": [
                {
                  "Id": "ArchiveToGlacier",
                  "Prefix": "",
                  "Status": "Enabled",
                  "Transitions":[{
                      "TransitionInDays": "60",
                      "StorageClass": "GLACIER"
                  }]
                }
            ]
        }
      }
    }

  },
  "Outputs":{
    "CentralLogBucket":{
    	"Description" : "Central log bucket",
    	"Value" : {"Ref": "BucketName"} ,
    	"Export" : { "Name" : "CentralLogBucketName"}
    }
  }
} 

To create your central-logging bucket do the following:

  1. Save the template file to your local developer machine as “central-log-bucket.json”
  2. From the CloudFormation console, select “create new stack” and import the file “central-log-bucket.json”
  3. Fill in the parameters and complete stack creation steps (as indicated in the screenshot below)
  4. Verify the bucket has been created successfully and take a note of the bucket name

Step 2: Create data processing Lambda function

Use the template below to create a Lambda function in your logging account that will be used by Amazon Firehose for data transformation during the delivery process to S3. This function is based on the AWS Lambda kinesis-firehose-cloudwatch-logs-processor blueprint.

The function could be created manually from the blueprint or using the cloudformation template below. To find the blueprint navigate to Lambda -> Create -> Function -> Blueprints

This function will unzip the event message, parse it and verify that it is a valid CloudWatch log event. Additional processing can be added if needed. As this function is generic, it could be reused by all log-delivery streams.

{
  "AWSTemplateFormatVersion":"2010-09-09",
  "Description": "Create cloudwatch data processing lambda function",
  "Resources":{
      
    "LambdaRole": {
        "Type": "AWS::IAM::Role",
        "Properties": {
            "AssumeRolePolicyDocument": {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Effect": "Allow",
                        "Principal": {
                            "Service": "lambda.amazonaws.com"
                        },
                        "Action": "sts:AssumeRole"
                    }
                ]
            },
            "Path": "/",
            "Policies": [
                {
                    "PolicyName": "firehoseCloudWatchDataProcessing",
                    "PolicyDocument": {
                        "Version": "2012-10-17",
                        "Statement": [
                            {
                                "Effect": "Allow",
                                "Action": [
                                    "logs:CreateLogGroup",
                                    "logs:CreateLogStream",
                                    "logs:PutLogEvents"
                                ],
                                "Resource": "arn:aws:logs:*:*:*"
                            }
                        ]
                    }
                }
            ]
        }
    },
      
    "FirehoseDataProcessingFunction": {
        "Type": "AWS::Lambda::Function",
        "Properties": {
            "Handler": "index.handler",
            "Role": {"Fn::GetAtt": ["LambdaRole","Arn"]},
            "Description": "Firehose cloudwatch data processing",
            "Code": {
                "ZipFile" : { "Fn::Join" : ["\n", [
                  "'use strict';",
                  "const zlib = require('zlib');",
                  "function transformLogEvent(logEvent) {",
                  "       return Promise.resolve(`${logEvent.message}\n`);",
                  "}",
                  "exports.handler = (event, context, callback) => {",
                  "    Promise.all(event.records.map(r => {",
                  "        const buffer = new Buffer(r.data, 'base64');",
                  "        const decompressed = zlib.gunzipSync(buffer);",
                  "        const data = JSON.parse(decompressed);",
                  "        if (data.messageType !== 'DATA_MESSAGE') {",
                  "            return Promise.resolve({",
                  "                recordId: r.recordId,",
                  "                result: 'ProcessingFailed',",
                  "            });",
                  "         } else {",
                  "            const promises = data.logEvents.map(transformLogEvent);",
                  "            return Promise.all(promises).then(transformed => {",
                  "                const payload = transformed.reduce((a, v) => a + v, '');",
                  "                const encoded = new Buffer(payload).toString('base64');",
                  "                console.log('---------------payloadv2:'+JSON.stringify(payload, null, 2));",
                  "                return {",
                  "                    recordId: r.recordId,",
                  "                    result: 'Ok',",
                  "                    data: encoded,",
                  "                };",
                  "           });",
                  "        }",
                  "    })).then(recs => callback(null, { records: recs }));",
                    "};"

                ]]}
            },
            "Runtime": "nodejs6.10",
            "Timeout": "60"
        }
    }

  },
  "Outputs":{
   "Function" : {
      "Description": "Function ARN",
      "Value": {"Fn::GetAtt": ["FirehoseDataProcessingFunction","Arn"]},
      "Export" : { "Name" : {"Fn::Sub": "${AWS::StackName}-Function" }}
    }
  }
}

To create the function follow the steps below:

  1. Save the template file as “central-logging-lambda.json”
  2. Login to logging account and, from the CloudFormation console, select “create new stack”
  3. Import the file “central-logging-lambda.json” and click next
  4. Follow the steps to create the stack and verify successful creation
  5. Take a note of Lambda function arn from the output section

Step 3: Create log destination in logging account

Log destination is used as the target of a subscription from application accounts, log destination can be shared between multiple subscriptions however according to the architecture suggested in this solution all logs streamed to the same destination will be stored in the same S3 location, if you would like to store log data in different hierarchy or in a completely different bucket you need to create separate destinations.

As noted previously, your destination and subscription have to be in the same region

Use the template below to create destination stack in logging account.

{
  "AWSTemplateFormatVersion":"2010-09-09",
  "Description": "Create log destination and required resources",
  "Parameters":{

    "LogBucketName":{
      "Type":"String",
      "Default":"central-log-do-not-delete",
      "Description":"Destination logging bucket"
    },
    "LogS3Location":{
      "Type":"String",
      "Default":"<BU>/<ENV>/<SOURCE_ACCOUNT>/<LOG_TYPE>/",
      "Description":"S3 location for the logs streamed to this destination; example marketing/prod/999999999999/flow-logs/"
    },
    "ProcessingLambdaARN":{
      "Type":"String",
      "Default":"",
      "Description":"CloudWatch logs data processing function"
    },
    "SourceAccount":{
      "Type":"String",
      "Default":"",
      "Description":"Source application account number"
    }
  },
    
  "Resources":{
    "MyStream": {
      "Type": "AWS::Kinesis::Stream",
      "Properties": {
        "Name": {"Fn::Join" : [ "", [{ "Ref" : "AWS::StackName" },"-Stream"] ]},
        "RetentionPeriodHours" : 48,
        "ShardCount": 1,
        "Tags": [
          {
            "Key": "Solution",
            "Value": "CentralLogging"
          }
       ]
      }
    },
    "LogRole" : {
      "Type"  : "AWS::IAM::Role",
      "Properties" : {
          "AssumeRolePolicyDocument" : {
              "Statement" : [ {
                  "Effect" : "Allow",
                  "Principal" : {
                      "Service" : [ {"Fn::Join": [ "", [ "logs.", { "Ref": "AWS::Region" }, ".amazonaws.com" ] ]} ]
                  },
                  "Action" : [ "sts:AssumeRole" ]
              } ]
          },         
          "Path" : "/service-role/"
      }
    },
      
    "LogRolePolicy" : {
        "Type" : "AWS::IAM::Policy",
        "Properties" : {
            "PolicyName" : {"Fn::Join" : [ "", [{ "Ref" : "AWS::StackName" },"-LogPolicy"] ]},
            "PolicyDocument" : {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": ["kinesis:PutRecord"],
                  "Resource": [{ "Fn::GetAtt" : ["MyStream", "Arn"] }]
                },
                {
                  "Effect": "Allow",
                  "Action": ["iam:PassRole"],
                  "Resource": [{ "Fn::GetAtt" : ["LogRole", "Arn"] }]
                }
              ]
            },
            "Roles" : [ { "Ref" : "LogRole" } ]
        }
    },
      
    "LogDestination" : {
      "Type" : "AWS::Logs::Destination",
      "DependsOn" : ["MyStream","LogRole","LogRolePolicy"],
      "Properties" : {
        "DestinationName": {"Fn::Join" : [ "", [{ "Ref" : "AWS::StackName" },"-Destination"] ]},
        "RoleArn": { "Fn::GetAtt" : ["LogRole", "Arn"] },
        "TargetArn": { "Fn::GetAtt" : ["MyStream", "Arn"] },
        "DestinationPolicy": { "Fn::Join" : ["",[
		
				"{\"Version\" : \"2012-10-17\",\"Statement\" : [{\"Effect\" : \"Allow\",",
                " \"Principal\" : {\"AWS\" : \"", {"Ref":"SourceAccount"} ,"\"},",
                "\"Action\" : \"logs:PutSubscriptionFilter\",",
                " \"Resource\" : \"", 
                {"Fn::Join": [ "", [ "arn:aws:logs:", { "Ref": "AWS::Region" }, ":" ,{ "Ref": "AWS::AccountId" }, ":destination:",{ "Ref" : "AWS::StackName" },"-Destination" ] ]}  ,"\"}]}"

			]]}
          
          
      }
    },
      
    "S3deliveryStream": {
      "DependsOn": ["S3deliveryRole", "S3deliveryPolicy"],
      "Type": "AWS::KinesisFirehose::DeliveryStream",
      "Properties": {
        "DeliveryStreamName": {"Fn::Join" : [ "", [{ "Ref" : "AWS::StackName" },"-DeliveryStream"] ]},
        "DeliveryStreamType": "KinesisStreamAsSource",
        "KinesisStreamSourceConfiguration": {
            "KinesisStreamARN": { "Fn::GetAtt" : ["MyStream", "Arn"] },
            "RoleARN": {"Fn::GetAtt" : ["S3deliveryRole", "Arn"] }
        },
        "ExtendedS3DestinationConfiguration": {
          "BucketARN": {"Fn::Join" : [ "", ["arn:aws:s3:::",{"Ref":"LogBucketName"}] ]},
          "BufferingHints": {
            "IntervalInSeconds": "60",
            "SizeInMBs": "50"
          },
          "CompressionFormat": "UNCOMPRESSED",
          "Prefix": {"Ref": "LogS3Location"},
          "RoleARN": {"Fn::GetAtt" : ["S3deliveryRole", "Arn"] },
          "ProcessingConfiguration" : {
              "Enabled": "true",
              "Processors": [
              {
                "Parameters": [ 
                { 
                    "ParameterName": "LambdaArn",
                    "ParameterValue": {"Ref":"ProcessingLambdaARN"}
                }],
                "Type": "Lambda"
              }]
          }
        }

      }
    },
      
    "S3deliveryRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Sid": "",
              "Effect": "Allow",
              "Principal": {
                "Service": "firehose.amazonaws.com"
              },
              "Action": "sts:AssumeRole",
              "Condition": {
                "StringEquals": {
                  "sts:ExternalId": {"Ref":"AWS::AccountId"}
                }
              }
            }
          ]
        }
      }
    },
      
    "S3deliveryPolicy": {
      "Type": "AWS::IAM::Policy",
      "Properties": {
        "PolicyName": {"Fn::Join" : [ "", [{ "Ref" : "AWS::StackName" },"-FirehosePolicy"] ]},
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "s3:AbortMultipartUpload",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:PutObject"
              ],
              "Resource": [
                {"Fn::Join": ["", [ {"Fn::Join" : [ "", ["arn:aws:s3:::",{"Ref":"LogBucketName"}] ]}]]},
                {"Fn::Join": ["", [ {"Fn::Join" : [ "", ["arn:aws:s3:::",{"Ref":"LogBucketName"}] ]}, "*"]]}
              ]
            },
            {
              "Effect": "Allow",
              "Action": [
                "lambda:InvokeFunction",
                "lambda:GetFunctionConfiguration",
                "logs:PutLogEvents",
                "kinesis:DescribeStream",
                "kinesis:GetShardIterator",
                "kinesis:GetRecords",
                "kms:Decrypt"
              ],
              "Resource": "*"
            }
          ]
        },
        "Roles": [{"Ref": "S3deliveryRole"}]
      }
    }

  },
  "Outputs":{
      
   "Destination" : {
      "Description": "Destination",
      "Value": {"Fn::Join": [ "", [ "arn:aws:logs:", { "Ref": "AWS::Region" }, ":" ,{ "Ref": "AWS::AccountId" }, ":destination:",{ "Ref" : "AWS::StackName" },"-Destination" ] ]},
      "Export" : { "Name" : {"Fn::Sub": "${AWS::StackName}-Destination" }}
    }

  }
} 

To create log your destination and all required resources, follow these steps:

  1. Save your template as “central-logging-destination.json”
  2. Login to your logging account and, from the CloudFormation console, select “create new stack”
  3. Import the file “central-logging-destination.json” and click next
  4. Fill in the parameters to configure the log destination and click Next
  5. Follow the default steps to create the stack and verify successful creation
    1. Bucket name is the same as in the “create central logging bucket” step
    2. LogS3Location is the directory hierarchy for saving log data that will be delivered to this destination
    3. ProcessingLambdaARN is as created in “create data processing Lambda function” step
    4. SourceAccount is the application account number where the subscription will be created
  6. Take a note of destination ARN as it appears in outputs section as you did above.

Step 4: Create the log subscription in your application account

In this section, we will create the subscription filter in one of the application accounts to stream logs from the CloudWatch log group to the log destination that was created in your logging account.

Create log subscription filter

The subscription filter is created between the CloudWatch log group and a destination endpoint. Asubscription could be filtered to send part (or all) of the logs in the log group. For example,you can create a subscription filter to stream only flow logs with status REJECT.

Use the CloudFormation template below to create subscription filter. Subscription filter and log destination must be in the same region.

{
  "AWSTemplateFormatVersion":"2010-09-09",
  "Description": "Create log subscription filter for a specific Log Group",
  "Parameters":{

    "DestinationARN":{
      "Type":"String",
      "Default":"",
      "Description":"ARN of logs destination"
    },
    "LogGroupName":{
      "Type":"String",
      "Default":"",
      "Description":"Name of LogGroup to forward logs from"
    },
    "FilterPattern":{
      "Type":"String",
      "Default":"",
      "Description":"Filter pattern to filter events to be sent to log destination; Leave empty to send all logs"
    }
  },
    
  "Resources":{
    "SubscriptionFilter" : {
      "Type" : "AWS::Logs::SubscriptionFilter",
      "Properties" : {
        "LogGroupName" : { "Ref" : "LogGroupName" },
        "FilterPattern" : { "Ref" : "FilterPattern" },
        "DestinationArn" : { "Ref" : "DestinationARN" }
      }
    }
  }
}

To create a subscription filter for one of CloudWatch log groups in your application account, follow the steps below:

  1. Save the template as “central-logging-subscription.json”
  2. Login to your application account and, from the CloudFormation console, select “create new stack”
  3. Select the file “central-logging-subscription.json” and click next
  4. Fill in the parameters as appropriate to your environment as you did above
    a.  DestinationARN is the value of obtained in “create log destination in logging account” step
    b.  FilterPatterns is the filter value for log data to be streamed to your logging account (leave empty to stream all logs in the selected log group)
    c.  LogGroupName is the log group as it appears under CloudWatch Logs
  5. Verify successful creation of the subscription

This completes the deployment process in both the logging- and application-account side. After a few minutes, log data will be streamed to the central-logging destination defined in your logging account.

Step 5: Analyzing log data

Once log data is centralized, it opens the door to run analytics on the consolidated data for business or security reasons. One of the powerful services that AWS offers is Amazon Athena.

Amazon Athena allows you to query data in S3 using standard SQL.

Follow the steps below to create a simple table and run queries on the flow logs data that has been collected from your application accounts

  1. Login to your logging account and from the Amazon Athena console, use the DDL below in your query  editor to create a new table

CREATE EXTERNAL TABLE IF NOT EXISTS prod_vpc_flow_logs (

Version INT,

Account STRING,

InterfaceId STRING,

SourceAddress STRING,

DestinationAddress STRING,

SourcePort INT,

DestinationPort INT,

Protocol INT,

Packets INT,

Bytes INT,

StartTime INT,

EndTime INT,

Action STRING,

LogStatus STRING

)

ROW FORMAT SERDE ‘org.apache.hadoop.hive.serde2.RegexSerDe’

WITH SERDEPROPERTIES (

“input.regex” = “^([^ ]+)\\s+([0-9]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([0-9]+)\\s+([0-9]+)\\s+([^ ]+)\\s+([^ ]+)$”)

LOCATION ‘s3://central-logging-company-do-not-delete/’;

2. Click ”run query” and verify a successful run/ This creates the table “prod_vpc_flow_logs”

3. You can then run queries against the table data as below:

Conclusion

By following the steps I’ve outlined, you will build a central logging solution to stream CloudWatch logs from one application account to a central logging account. This solution is repeatable and could be deployed multiple times for multiple accounts and logging requirements.

 

About the Author

Mahmoud Matouk is a Senior Cloud Infrastructure Architect. He works with our customers to help accelerate migration and cloud adoption at the enterprise level.

 

How to Manage Amazon GuardDuty Security Findings Across Multiple Accounts

Post Syndicated from Tom Stickle original https://aws.amazon.com/blogs/security/how-to-manage-amazon-guardduty-security-findings-across-multiple-accounts/

Introduced at AWS re:Invent 2017, Amazon GuardDuty is a managed threat detection service that continuously monitors for malicious or unauthorized behavior to help you protect your AWS accounts and workloads. In an AWS Blog post, Jeff Barr shows you how to enable GuardDuty to monitor your AWS resources continuously. That blog post shows how to get started with a single GuardDuty account and provides an overview of the features of the service. Your security team, though, will probably want to use GuardDuty to monitor a group of AWS accounts continuously.

In this post, I demonstrate how to use GuardDuty to monitor a group of AWS accounts and have their findings routed to another AWS account—the master account—that is owned by a security team. The method I demonstrate in this post is especially useful if your security team is responsible for monitoring a group of AWS accounts over which it does not have direct access—known as member accounts. In this solution, I simplify the work needed to enable GuardDuty in member accounts and configure findings by simplifying the process, which I do by enabling GuardDuty in the master account and inviting member accounts.

Enable GuardDuty in a master account and invite member accounts

To get started, you must enable GuardDuty in the master account, which will receive GuardDuty findings. The master account should be managed by your security team, and it will display the findings from all member accounts. The master account can be reverted later by removing any member accounts you add to it. Adding member accounts is a two-way handshake mechanism to ensure that administrators from both the master and member accounts formally agree to establish the relationship.

To enable GuardDuty in the master account and add member accounts:

  1. Navigate to the GuardDuty console.
  2. In the navigation pane, choose Accounts.
    Screenshot of the Accounts choice in the navigation pane
  1. To designate this account as the GuardDuty master account, start adding member accounts:
    • You can add individual accounts by choosing Add Account, or you can add a list of accounts by choosing Upload List (.csv).
  1. Now, add the account ID and email address of the member account, and choose Add. (If you are uploading a list of accounts, choose Browse, choose the .csv file with the member accounts [one email address and account ID per line], and choose Add accounts.)
    Screenshot of adding an account

For security reasons, AWS checks to make sure each account ID is valid and that you’ve entered each member account’s email address that was used to create the account. If a member account’s account ID and email address do not match, GuardDuty does not send an invitation.
Screenshot showing the Status of Invite

  1. After you add all the member accounts you want to add, you will see them listed in the Member accounts table with a Status of Invite. You don’t have to individually invite each account—you can choose a group of accounts and when you choose to invite one account in the group, all accounts are invited.
  2. When you choose Invite for each member account:
    1. AWS checks to make sure the account ID is valid and the email address provided is the email address of the member account.
    2. AWS sends an email to the member account email address with a link to the GuardDuty console, where the member account owner can accept the invitation. You can add a customized message from your security team. Account owners who receive the invitation must sign in to their AWS account to accept the invitation. The service also sends an invitation through the AWS Personal Health Dashboard in case the member email address is not monitored. This invitation appears in the member account under the AWS Personal Health Dashboard alert bell on the AWS Management Console.
    3. A pending-invitation indicator is shown on the GuardDuty console of the member account, as shown in the following screenshot.
      Screenshot showing the pending-invitation indicator

When the invitation is sent by email, it is sent to the account owner of the GuardDuty member account.
Screenshot of the invitation sent by email

The account owner can click the link in the email invitation or the AWS Personal Health Dashboard message, or the account owner can sign in to their account and navigate to the GuardDuty console. In all cases, the member account displays the pending invitation in the member account’s GuardDuty console with instructions for accepting the invitation. The GuardDuty console walks the account owner through accepting the invitation, including enabling GuardDuty if it is not already enabled.

If you prefer to work in the AWS CLI, you can enable GuardDuty and accept the invitation. To do this, call CreateDetector to enable GuardDuty, and then call AcceptInvitation, which serves the same purpose as accepting the invitation in the GuardDuty console.

  1. After the member account owner accepts the invitation, the Status in the master account is changed to Monitored. The status helps you track the status of each AWS account that you invite.
    Screenshot showing the Status change to Monitored

You have enabled GuardDuty on the member account, and all findings will be forwarded to the master account. You can now monitor the findings about GuardDuty member accounts from the GuardDuty console in the master account.

The member account owner can see GuardDuty findings by default and can control all aspects of the experience in the member account with AWS Identity and Access Management (IAM) permissions. Users with the appropriate permissions can end the multi-account relationship at any time by toggling the Accept button on the Accounts page. Note that ending the relationship changes the Status of the account to Resigned and also triggers a security finding on the side of the master account so that the security team knows the member account is no longer linked to the master account.

Working with GuardDuty findings

Most security teams have ticketing systems, chat operations, security information event management (SIEM) systems, or other security automation systems to which they would like to push GuardDuty findings. For this purpose, GuardDuty sends all findings as JSON-based messages through Amazon CloudWatch Events, a scalable service to which you can subscribe and to which AWS services can stream system events. To access these events, navigate to the CloudWatch Events console and create a rule that subscribes to the GuardDuty-related findings. You then can assign a target such as Amazon Kinesis Data Firehose that can place the findings in a number of services such as Amazon S3. The following screenshot is of the CloudWatch Events console, where I have a rule that pulls all events from GuardDuty and pushes them to a preconfigured AWS Lambda function.

Screenshot of a CloudWatch Events rule

The following example is a subset of GuardDuty findings that includes relevant context and information about the nature of a threat that was detected. In this example, the instanceId, i-00bb62b69b7004a4c, is performing Secure Shell (SSH) brute-force attacks against IP address 172.16.0.28. From a Lambda function, you can access any of the following fields such as the title of the finding and its description, and send those directly to your ticketing system.

Example GuardDuty findings

You can use other AWS services to build custom analytics and visualizations of your security findings. For example, you can connect Kinesis Data Firehose to CloudWatch Events and write events to an S3 bucket in a standard format, which can be encrypted with AWS Key Management Service and then compressed. You also can use Amazon QuickSight to build ad hoc dashboards by using AWS Glue and Amazon Athena. Similarly, you can place the data from Kinesis Data Firehose in Amazon Elasticsearch Service, with which you can use tools such as Kibana to build your own visualizations and dashboards.

Like most other AWS services, GuardDuty is a regional service. This means that when you enable GuardDuty in an AWS Region, all findings are generated and delivered in that region. If you are regulated by a compliance regime, this is often an important requirement to ensure that security findings remain in a specific jurisdiction. Because customers have let us know they would prefer to be able to enable GuardDuty globally and have all findings aggregated in one place, we intend to give the choice of regional or global isolation as we evolve this new service.

Summary

In this blog post, I have demonstrated how to use GuardDuty to monitor a group of GuardDuty member accounts and aggregate security findings in a central master GuardDuty account. You can use this solution whether or not you have direct control over the member accounts.

If you have comments about this blog post, submit them in the “Comments” section below. If you have questions about using GuardDuty, start a thread in the GuardDuty forum or contact AWS Support.

-Tom