Tag Archives: AWS Secrets Manager

How to rotate a WordPress MySQL database secret using AWS Secrets Manager in Amazon EKS

Post Syndicated from Paavan Mistry original https://aws.amazon.com/blogs/security/how-to-rotate-a-wordpress-mysql-database-secret-using-aws-secrets-manager-in-amazon-eks/

AWS Secrets Manager recently announced a feature update to rotate credentials for all Amazon RDS database types. This allows you to automatically rotate credentials for all types of databases hosted on Amazon RDS. In this post, I show you how to rotate database secrets for a non-RDS database using AWS Secrets Manager. I use a containerized WordPress application with a MySQL database to demonstrate the secret rotation.

Enabling regular rotation of database secrets helps secure application databases, protects customer data, and helps meet compliance requirements. You’ll use Amazon Elastic Container Service for Kubernetes (Amazon EKS) to help deploy, manage, and scale the WordPress application, and Secrets Manager to perform secret rotation on a containerized MySQL database.

Prerequisites

You’ll need an Amazon EKS or a Kubernetes cluster running and accessible for this post. Getting Started with Amazon EKS provides instructions on setting up the cluster and node environment. I recommend that you have a basic understanding of Kubernetes concepts, but Kubernetes reference document links are provided throughout this walk-through. For this post, I use the placeholder EKSClusterName to denote the existing EKS cluster. Remember to replace this with the name of your EKS cluster.

You’ll also need AWS Command Line Interface (AWS CLI) installed and configured on your machine. For this blog, I assume that the default AWS CLI region is set to Oregon (us-west-2) and that you have access to the AWS services described in this post. If you use other regions, you should check the availability of AWS services in those regions.

Architecture overview

 

Figure 1: Architecture and data flow diagram within Amazon EKS nodes

Figure 1: Architecture and data flow diagram within Amazon EKS nodes

The architecture diagram shows the overall deployment architecture with two data flows, a user data flow and a Secrets Manager data flow within the EKS three-node cluster VPC (virtual private cloud).

To access the WordPress site, user data flows through an internet gateway and an external load balancer as part of the WordPress frontend Kubernetes deployment. The AWS Secrets Manager data flow uses the recently announced Secrets Manager VPC Endpoint and an AWS Lambda function within the VPC that rotates the MySQL database secret through an internal load balancer. MySQL database and the internal load balancer are provisioned as part of the WordPress MySQL Kubernetes deployment. The internal load balancer allows the database service to be exposed internally for Secrets Manager to perform the rotation within a VPC. It removes the need for the database to be exposed to the Internet for secret rotation, which is not a good security practice.

Solution overview

The blog post consists of the following steps:

  1. Host WordPress and MySQL services on Amazon EKS.
  2. Store the database secret in AWS Secrets Manager.
  3. Set up the rotation of the database secret using Secrets Manager VPC Endpoint.
  4. Rotate the database secret and update the WordPress frontend deployment.

Note: This post helps you implement MySQL secret rotation for non-RDS databases. It should be used as a reference guide on database secret rotation. For guidance on using WordPress on AWS, please refer to the WordPress: Best Practices on AWS whitepaper.

Step 1: Host WordPress and MySQL services on Amazon EKS

To deploy WordPress on an EKS cluster, you’ll use three YAML templates.

  1. First, create a StorageClass in Amazon EKS that uses Amazon Elastic Block Store (Amazon EBS) for persistent volumes, using the command below if the volumes don’t exist. Then, create a Kubernetes namespace. The Kubernetes namespace creates a separate environment within your Kubernetes cluster to create objects specific to this walk-through. This helps with object management and logical separation.
    
    kubectl create -f https://raw.githubusercontent.com/paavan98pm/eks-secret-rotation/master/templates/gp2-storage-class.yaml
    
    kubectl create namespace wp
    

  2. Next, use Kubernetes secrets to store your MySQL database password with an Amazon EKS cluster. The password is generated by the AWS Secret Manager get-random-password API using the command below. This allows a random password to be created and stored in the Kubernetes secret object through the Secrets Manager API feature without the need to manually create it. The get-random-password API allows various password length and type restrictions to be enforced based on your organizational security policy.
    
    kubectl create secret generic mysql-pass --from-literal=password=$(aws secretsmanager get-random-password --password-length 20 --no-include-space | jq -r .RandomPassword) --namespace=wp
    

You’ll use the Kubernetes secret mysql-pass to create your MySQL and WordPress deployments using the YAML manifests in the next step.

Deploying MySQL and WordPress in Amazon EKS

Now, you’ll run the MySQL and WordPress templates provided by the Kubernetes community to deploy your backend MySQL services, deployments, and persistent volume claims.

  1. To deploy an internal load balancer that AWS Secrets Manager will use to perform its secret rotation, you’ll use the Kubernetes service annotation service.beta.kubernetes.io/aws-load-balancer-internal within the MySQL service YAML. For WordPress deployment, you’ll add three replicas for availability across the three availability zones.
    
    kubectl create -f https://raw.githubusercontent.com/paavan98pm/eks-secret-rotation/master/templates/mysql-deployment.yaml --namespace=wp
    
    kubectl create -f https://raw.githubusercontent.com/paavan98pm/eks-secret-rotation/master/templates/wordpress-deployment.yaml --namespace=wp
    

    Run the commands above with the Kubernetes YAML manifests to create MySQL and WordPress services, deployments, and persistent volume claims. As shown in the architecture diagram, these services also provision an external and an internal load balancer. The external load balancer is accessible over the internet for your WordPress frontend, while you’ll use the internal load balancer to rotate database secrets. It takes a few minutes for the load balancers to be provisioned.

  2. Once the load balancers are provisioned, you should be able to access your WordPress site in your browser by running the following command. The browser will open to show the WordPress setup prompt in Figure 2.
    
    open http://$(kubectl get svc -l app=wordpress --namespace=wp -o=jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}')
    

     

    Figure 2: WordPress setup page for the Amazon EKS deployment

    Figure 2: WordPress setup page for the Amazon EKS deployment

  3. From here, follow the WordPress instructions to complete installation and set up credentials to access your WordPress administration page. This completes the hosting and setup of your WordPress blog on Amazon EKS.

Step 2: Store the database secret in AWS Secrets Manager

Next, you’re going to store your database secret in AWS Secrets Manager. AWS Secrets Manager is a fully managed AWS service that enables you to rotate, manage, and retrieve secrets such as database credentials and application programming interface keys (API keys) throughout their lifecycle. Follow the steps below to use Secrets Manager to store the MySQL database secret and the rotation configuration details.
 

Figure 3: Store a new secret in AWS Secrets Manager

Figure 3: Store a new secret in AWS Secrets Manager

From the Secrets Manager console, select Store a new secret to open the page shown in Figure 3 above. Follow the instructions below to store the MySQL database secret parameters.

  1. Select Credentials for other database and provide the username and password you created in the previous step. The username value will be root, and you should paste the password value from the output of the command below. This command copies the MySQL password stored in the Kubernetes secret object, allowing you to store it in Secrets Manager.

    Password:

    
    kubectl get secret mysql-pass -o=jsonpath='{.data.password}' --namespace=wp | base64 --decode | pbcopy
    

  2. Secrets Manager encrypts secrets by default using the service-specific encryption key for your AWS account. Since you’re storing the secret for MySQL database, choose the MySQL tile and provide the server details. For Server address, you can paste the output of the command below. The database name and port values are mysql and 3306, respectively.

    Server address:

    
    kubectl get svc -l app=wordpress-mysql --namespace=wp -o=jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}' | pbcopy
    

  3. Select Next and give the secret a name and a description that enables you to reference and manage the secret easily. I use the placeholder yourSecretName to reference your secret throughout the rest of my post; be sure to swap this placeholder with your own secret name.
  4. On the next screen, accept the default setting: Disable automatic rotation.
  5. Finally, select Store to store your MySQL secret in AWS Secrets Manager.

Step 3: Set MySQL secret rotation using Secrets Manager VPC Endpoint

Next, you’ll create a Lambda Function within the Amazon EKS node cluster VPC to rotate this secret. Within the node cluster VPC, you’re also going to configure the AWS Secrets Manager recently announced support for Amazon Virtual Private Cloud (VPC) endpoints powered by AWS PrivateLink.

Note: If you have a database hosted on Amazon RDS, Secrets Manager natively supports rotating secrets. However, if you’re using a container-based MySQL database, you must create and configure the Lambda rotation function for AWS Secrets Manager, and then provide the Amazon Resource Name (ARN) of the completed function to the secret.

Creating a Lambda rotation function

The following commands apply a WordPress MySQL rotation template to a new Lambda function. CloudFormation uses an AWS Serverless Application Repository template to automate most of the steps for you. The values you need to replace for your environment are denoted in red.

  1. The command below will return a ChangeSetId. It uses the Secrets Manager MySQL single user template from the AWS Serverless Application Repository and creates a CloudFormation Change Set by providing the Secrets Manager endpoint and Lambda function name as parameters. You need to change the Secrets Manager region within the endpoint parameter and provide a name for the Lambda function that you’re creating.
    
    aws serverlessrepo create-cloud-formation-change-set --application-id arn:aws:serverlessrepo:us-east-1:297356227824:applications/SecretsManagerRDSMySQLRotationSingleUser --stack-name MyLambdaCreationStack --parameter-overrides '[{"Name":"endpoint","Value":"https://secretsmanager.<REGION>.amazonaws.com"},{"Name":"functionName","Value":"<EKSMySQLRotationFunction>"}]' | jq -r '.ChangeSetId'
    

  2. The next command runs the change set that you just created to create a Lambda function. The change-set-name parameter comes from the ChangeSetId output of the previous command. Copy the ChangeSetId value into the instruction below.
    
    aws cloudformation execute-change-set --change-set-name <ChangeSetId>
    

  3. The following command grants Secrets Manager permission to call the Lambda function on your behalf. You should provide the Lambda function name that you set earlier.
    
    aws lambda add-permission --function-name <EKSMySQLRotationFunction> --principal secretsmanager.amazonaws.com --action lambda:InvokeFunction --statement-id SecretsManagerAccess
    

Configuring the Lambda rotation function to use Secrets Manager VPC Endpoint

Instead of connecting your VPC to the internet, you can connect directly to Secrets Manager through a private endpoint that you configure within your VPC. When you use a VPC service endpoint, communication between your VPC and Secrets Manager occurs entirely within the AWS network and requires no public internet access.

  1. To enable AWS Secrets Manager VPC endpoint, first store the Node Instance security group in a NODE_INSTANCE_SG environment variable to use it as an argument for creating the Secrets Manager VPC Endpoint. Using the VPC ID from the EKS cluster, this command filters the Security Groups attached to the nodes and stores them in the NODE_INSTANCE_SG environment variable. This variable will be used as an argument in the next step to create a Secrets Manager VPC endpoint.
    
    export NODE_INSTANCE_SG=$(aws ec2 describe-security-groups --filters Name=vpc-id,Values=$(aws eks describe-cluster --name <EKSClusterName> | jq -r '.cluster.resourcesVpcConfig.vpcId') Name=tag:aws:cloudformation:logical-id,Values=NodeSecurityGroup | jq -r '.SecurityGroups[].GroupId')
    

  2. Next, create a Secrets Manager VPC endpoint attached to the EKS node cluster subnets and the related security group. This command retrieves the VPC ID and Subnet IDs from the EKS cluster to create a Secrets Manager VPC endpoint.
    
    aws ec2 create-vpc-endpoint --vpc-id $(aws eks describe-cluster --name <EKSClusterName> | jq -r '.cluster.resourcesVpcConfig.vpcId') --vpc-endpoint-type Interface --service-name com.amazonaws.<region>.secretsmanager --subnet-ids $(aws eks describe-cluster --name <EKSClusterName> | jq -r '.cluster.resourcesVpcConfig.subnetIds | join(" ")') --security-group-id $NODE_INSTANCE_SG --private-dns-enabled
    

  3. Next, update the Lambda function to attach it to the EKS node cluster subnets and the related security group. This command updates the Lambda function’s configuration by retrieving its ARN and EKS cluster Subnet IDs and Security Group ID.
    
    aws lambda update-function-configuration --function-name $(aws lambda list-functions | jq -r '.Functions[] | select(.FunctionName == "<EKSMySQLRotationFunction>") | .FunctionArn') --vpc-config SubnetIds=$(aws eks describe-cluster --name <EKSClusterName> | jq -r '.cluster.resourcesVpcConfig.subnetIds | join(",")'),SecurityGroupIds=$NODE_INSTANCE_SG
    

Step 4: Rotate the database secret and update your WordPress frontend deployment

Now that you’ve configured the Lambda rotation function, use the rotate-secret command to schedule a rotation of this secret. The command below uses the previously stored secret name with the Lambda function ARN to rotate your secret automatically every 30 days. You can adjust the rotation frequency value, if you want. The minimum rotation frequency is 1 day.


aws secretsmanager rotate-secret --secret-id <yourSecretName> --rotation-lambda-arn $(aws lambda list-functions | jq -r '.Functions[] | select(.FunctionName == "<EKSMySQLRotationFunction>") | .FunctionArn') --rotation-rules AutomaticallyAfterDays=30

The figure below shows the Secrets Manager data flow for secret rotation using the VPC endpoint.
 

Figure 4: MySQL database secret rotation steps

Figure 4: MySQL database secret rotation steps

Based on the frequency of the rotation, Secrets Manager will follow the steps below to perform automatic rotations:

  1. Secrets Manager will invoke <EKSMySQLRotationFunction> within the EKS node VPC using the Secrets Manager VPC endpoint.
  2. Using the credentials provided for MySQL database, <EKSMySQLRotationFunction> will rotate the database secret and update the secret value in Secrets Manager.

Once the Lambda function has executed, you can test the updated database secret value that’s stored in AWS Secrets Manager by using the command below.


aws secretsmanager get-secret-value --secret-id <yourSecretName>

Next, update the Kubernetes secret (mysql-pass) with the rotated secret stored in AWS Secrets Manager, then update the WordPress frontend deployment with the rotated MySQL password environment variable using the command below. These instructions should be automated using Kubernetes and AWS API once you’ve completed your secret rotation.

The command below retrieves the rotated secret from Secrets Manager using the GetSecretValue API call and passes the updated value to the Kubernetes secret. The next command patches the WordPress deployment for the frontend WordPress containers to use the updated Kubernetes secret. It also adds a rotation date annotation for the deployment.


kubectl create secret generic mysql-pass --namespace=wp --from-literal=password=$(aws secretsmanager get-secret-value --secret-id <yourSecretName> | jq --raw-output '.SecretString' | jq -r .password) -o yaml --dry-run | k replace -f -

kubectl patch deploy wordpress -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"rotation_date\":\"`date +'%s'`\"}}}}}" --namespace=wp

You should now be able to access the updated WordPress site with database secret rotation enabled.


open http://$(kubectl get svc -l app=wordpress --namespace=wp -o=jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}')

Clean-up

To delete the environment used in this walk-through in EKS, the secret in Secrets Manager, and the accompanying Lambda function, run the following commands. You only need to follow these steps if you intend to remove the secret, Lambda function, VPC endpoint, and EKS objects created in this walk-through.


kubectl delete namespace wp

aws secretsmanager delete-secret --secret-id <yourSecretName> --force-delete-without-recovery

aws lambda delete-function --function-name <EKSMySQLRotationFunction>

aws ec2 delete-vpc-endpoints --vpc-endpoint-ids $(aws ec2 describe-vpc-endpoints | jq -r '.VpcEndpoints[] | select(.ServiceName == "com.amazonaws.<REGION>.secretsmanager") | .VpcEndpointId')

Pricing

Next, I review the pricing and estimated cost of this example. AWS Secrets Manager offers a 30-day trial period that starts when you store your first secret. Storage of each secret costs $0.40 per secret per month. For secrets that are stored for less than a month, the price is prorated based on the number of hours. There is an additional cost of $0.05 per 10,000 API calls. You can learn more by visiting the AWS Secrets Manager pricing service details page.

You pay $0.20 per hour for each Amazon EKS cluster that you create. You can use a single Amazon EKS cluster to run multiple applications by taking advantage of Kubernetes namespaces and IAM security policies. You pay for AWS resources (for example, EC2 instances or EBS volumes) that you create to run your Kubernetes worker nodes. You only pay for what you use, as you use it; there are no minimum fees and no upfront commitments. See detailed pricing information on the Amazon EC2 pricing page.

Assuming you allocate two hours to follow the blog instructions, the cost will be approximately $1.

  1. Amazon EKS – 2 hours x ($0.20 per EKS cluster + 3 nodes x $0.096 m5.large instance)
  2. AWS Secrets Manager – 2 hours x ($0.40 per secret per month / 30 days / 24 hours + $0.05 per 10,000 API calls)

Summary

In this post, I showed you how to rotate WordPress database credentials in Amazon EKS using AWS Secrets Manager.

For more details on secrets management within Amazon EKS, check out the Github workshop for Kubernetes, particularly the section on ConfigMaps and Secrets, to understand different secret management alternatives available. To get started using Kubernetes on AWS, open the Amazon EKS console. To learn more, read the EKS documentation. To get started managing secrets, open the Secrets Manager console. To learn more, read the Secrets Manager documentation.

If you have comments about this post, submit them in the Comments section below. If you have questions about anything in this post, start a new thread on the EKS forum or Secrets Manager forum.

Want more AWS Security news? Follow us on Twitter.

Author

Paavan Mistry

Paavan is a Security Specialist Solutions Architect at AWS where he enjoys solving customers’ cloud security, risk, and compliance challenges. Outside of work, he enjoys reading about leadership, politics, law, and human rights.

How to use AWS Secrets Manager to rotate credentials for all Amazon RDS database types, including Oracle

Post Syndicated from Apurv Awasthi original https://aws.amazon.com/blogs/security/how-to-use-aws-secrets-manager-rotate-credentials-amazon-rds-database-types-oracle/

You can now use AWS Secrets Manager to rotate credentials for Oracle, Microsoft SQL Server, or MariaDB databases hosted on Amazon Relational Database Service (Amazon RDS) automatically. Previously, I showed how to rotate credentials for a MySQL database hosted on Amazon RDS automatically with AWS Secrets Manager. With today’s launch, you can use Secrets Manager to automatically rotate credentials for all types of databases hosted on Amazon RDS.

In this post, I review the key features of Secrets Manager. You’ll then learn:

  1. How to store the database credential for the superuser of an Oracle database hosted on Amazon RDS
  2. How to store the Oracle database credential used by an application
  3. How to configure Secrets Manager to rotate both Oracle credentials automatically on a schedule that you define

Key features of Secrets Manager

AWS Secrets Manager makes it easier to rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. The key features of this service include the ability to:

  1. Secure and manage secrets centrally. You can store, view, and manage all your secrets centrally. By default, Secrets Manager encrypts these secrets with encryption keys that you own and control. You can use fine-grained IAM policies or resource-based policies to control access to your secrets. You can also tag secrets to help you discover, organize, and control access to secrets used throughout your organization.
  2. Rotate secrets safely. You can configure Secrets Manager to rotate secrets automatically without disrupting your applications. Secrets Manager offers built-in integrations for rotating credentials for all Amazon RDS databases (MySQL, PostgreSQL, Oracle, Microsoft SQL Server, MariaDB, and Amazon Aurora.) You can also extend Secrets Manager to meet your custom rotation requirements by creating an AWS Lambda function to rotate other types of secrets.
  3. Transmit securely. Secrets are transmitted securely over Transport Layer Security (TLS) protocol 1.2. You can also use Secrets Manager with Amazon Virtual Private Cloud (Amazon VPC) endpoints powered by AWS Privatelink to keep this communication within the AWS network and help meet your compliance and regulatory requirements to limit public internet connectivity.
  4. Pay as you go. Pay for the secrets you store in Secrets Manager and for the use of these secrets; there are no long-term contracts, licensing fees, or infrastructure and personnel costs. For example, a typical production-scale web application will generate an estimated monthly bill of $6. If you follow along the instructions in this blog post, your estimated monthly bill for Secrets Manager will be $1. Note: you may incur additional charges for using Amazon RDS and Amazon Lambda, if you’ve already consumed the free tier for these services.

Now that you’re familiar with Secrets Manager features, I’ll show you how to store and automatically rotate credentials for an Oracle database hosted on Amazon RDS. I divided these instructions into three phases:

  1. Phase 1: Store and configure rotation for the superuser credential
  2. Phase 2: Store and configure rotation for the application credential
  3. Phase 3: Retrieve the credential from Secrets Manager programmatically

Prerequisites

To follow along, your AWS Identity and Access Management (IAM) principal (user or role) requires the SecretsManagerReadWrite AWS managed policy to store the secrets. Your principal also requires the IAMFullAccess AWS managed policy to create and configure permissions for the IAM role used by Lambda for executing rotations. You can use IAM permissions boundaries to grant an employee the ability to configure rotation without also granting them full administrative access to your account.

Phase 1: Store and configure rotation for the superuser credential

From the Secrets Manager console, on the right side, select Store a new secret.

Since I’m storing credentials for database hosted on Amazon RDS, I select Credentials for RDS database. Next, I input the user name and password for the superuser. I start by securing the superuser because it’s the most powerful database credential and has full access to the database.
 

Figure 1: For "Select secret type," choose "Credentials for RDS database"

Figure 1: For “Select secret type,” choose “Credentials for RDS database”

For this example, I choose to use the default encryption settings. Secrets Manager will encrypt this secret using the Secrets Manager DefaultEncryptionKey in this account. Alternatively, I can choose to encrypt using a customer master key (CMK) that I have stored in AWS Key Management Service (AWS KMS). To learn more, read the Using Your AWS KMS CMK documentation.
 

Figure 2: Choose either DefaultEncryptionKey or use a CMK

Figure 2: Choose either DefaultEncryptionKey or use a CMK

Next, I view the list of Amazon RDS instances in my account and select the database this credential accesses. For this example, I select the DB instance oracle-rds-database from the list, and then I select Next.

I then specify values for Secret name and Description. For this example, I use Database/Development/Oracle-Superuser as the name and enter a description of this secret, and then select Next.
 

Figure 3: Provide values for "Secret name" and "Description"

Figure 3: Provide values for “Secret name” and “Description”

Since this database is not yet being used, I choose to enable rotation. To do so, I select Enable automatic rotation, and then set the rotation interval to 60 days. Remember, if this database credential is currently being used, first update the application (see phase 3) to use Secrets Manager APIs to retrieve secrets before enabling rotation.
 

Figure 4: Select "Enable automatic rotation"

Figure 4: Select “Enable automatic rotation”

Next, Secrets Manager requires permissions to rotate this secret on my behalf. Because I’m storing the credentials for the superuser, Secrets Manager can use this credential to perform rotations. Therefore, on the same screen, I select Use a secret that I have previously stored in AWS Secrets Manager, and then select Next.

Finally, I review the information on the next screen. Everything looks correct, so I select Store. I have now successfully stored a secret in Secrets Manager.

Note: Secrets Manager will now create a Lambda function in the same VPC as my Oracle database and trigger this function periodically to change the password for the superuser. I can view the name of the Lambda function on the Rotation configuration section of the Secret Details page.

The banner on the next screen confirms that I’ve successfully configured rotation and the first rotation is in progress, which enables me to verify that rotation is functioning as expected. Secrets Manager will rotate this credential automatically every 60 days.
 

Figure 5: The confirmation notification

Figure 5: The confirmation notification

Phase 2: Store and configure rotation for the application credential

The superuser is a powerful credential that should be used only for administrative tasks. To enable your applications to access a database, create a unique database credential per application and grant these credentials limited permissions. You can use these database credentials to read or write to database tables required by the application. As a security best practice, deny the ability to perform management actions, such as creating new credentials.

In this phase, I will store the credential that my application will use to connect to the Oracle database. To get started, from the Secrets Manager console, on the right side, select Store a new secret.

Next, I select Credentials for RDS database, and input the user name and password for the application credential.

I continue to use the default encryption key. I select the DB instance oracle-rds-database, and then select Next.

I specify values for Secret Name and Description. For this example, I use Database/Development/Oracle-Application-User as the name and enter a description of this secret, and then select Next.

I now configure rotation. Once again, since my application is not using this database credential yet, I’ll configure rotation as part of storing this secret. I select Enable automatic rotation, and set the rotation interval to 60 days.

Next, Secrets Manager requires permissions to rotate this secret on behalf of my application. Earlier in the post, I mentioned that applications credentials have limited permissions and are unable to change their password. Therefore, I will use the superuser credential, Database/Development/Oracle-Superuser, that I stored in Phase 1 to rotate the application credential. With this configuration, Secrets Manager creates a clone application user.
 

Figure 6: Select the superuser credential

Figure 6: Select the superuser credential

Note: Creating a clone application user is the preferred mechanism of rotation because the old version of the secret continues to operate and handle service requests while the new version is prepared and tested. There’s no application downtime while changing between versions.

I review the information on the next screen. Everything looks correct, so I select Store. I have now successfully stored the application credential in Secrets Manager.

As mentioned in Phase 1, AWS Secrets Manager creates a Lambda function in the same VPC as the database and then triggers this function periodically to rotate the secret. Since I chose to use the existing superuser secret to rotate the application secret, I will grant the rotation Lambda function permissions to retrieve the superuser secret. To grant this permission, I first select role from the confirmation banner.
 

Figure 7: Select the "role" link that's in the confirmation notification

Figure 7: Select the “role” link that’s in the confirmation notification

Next, in the Permissions tab, I select SecretsManagerRDSMySQLRotationMultiUserRolePolicy0. Then I select Edit policy.
 

Figure 8: Edit the policy on the "Permissions" tab

Figure 8: Edit the policy on the “Permissions” tab

In this step, I update the policy (see below) and select Review policy. When following along, remember to replace the placeholder ARN-OF-SUPERUSER-SECRET with the ARN of the secret you stored in Phase 1.


{
  "Statement": [
    {
        "Effect": "Allow",
        "Action": [
            "ec2:CreateNetworkInterface",
			"ec2:DeleteNetworkInterface",
			"ec2:DescribeNetworkInterfaces",
			"ec2:DetachNetworkInterface"
		],
		"Resource": "*"
	},
	{
	    "Sid": "GrantPermissionToUse",
		"Effect": "Allow",
		"Action": [
            "secretsmanager:GetSecretValue"
        ],
		"Resource": "ARN-OF-SUPERUSER-SECRET"
	}
  ]
}

Here’s what it will look like:
 

Figure 9: Edit the policy

Figure 9: Edit the policy

Next, I select Save changes. I have now completed all the steps required to configure rotation for the application credential, Database/Development/Oracle-Application-User.

Phase 3: Retrieve the credential from Secrets Manager programmatically

Now that I have stored the secret in Secrets Manager, I add code to my application to retrieve the database credential from Secrets Manager. I use the sample code from Phase 2 above. This code sets up the client and retrieves and decrypts the secret Database/Development/Oracle-Application-User.

Remember, applications require permissions to retrieve the secret, Database/Development/Oracle-Application-User, from Secrets Manager. My application runs on Amazon EC2 and uses an IAM role to obtain access to AWS services. I attach the following policy to my IAM role. This policy uses the GetSecretValue action to grant my application permissions to read secret from Secrets Manager. This policy also uses the resource element to limit my application to read only the Database/Development/Oracle-Application-User secret from Secrets Manager. You can refer to the Secrets Manager Documentation to understand the minimum IAM permissions required to retrieve a secret.


{
 "Version": "2012-10-17",
 "Statement": {
    "Sid": "RetrieveDbCredentialFromSecretsManager",
    "Effect": "Allow",
    "Action": "secretsmanager:GetSecretValue",
    "Resource": "arn:aws:secretsmanager:<AWS-REGION>:<ACCOUNT-NUMBER>:secret: Database/Development/Oracle-Application-User     
 }
}

In the above policy, remember to replace the placeholder <AWS-REGION> with the AWS region that you’re using and the placeholder <ACCOUNT-NUMBER> with the number of your AWS account.

Summary

I explained the key benefits of Secrets Manager as they relate to RDS and showed you how to help meet your compliance requirements by configuring Secrets Manager to rotate database credentials automatically on your behalf. Secrets Manager helps you protect access to your applications, services, and IT resources without the upfront investment and on-going maintenance costs of operating your own secrets management infrastructure. To get started, visit the Secrets Manager console. To learn more, visit Secrets Manager documentation.

If you have comments about this post, submit them in the Comments section below. If you have questions about anything in this post, start a new thread on the Secrets Manager forum.

Want more AWS Security news? Follow us on Twitter.

Apurv Awasthi

Apurv is the product manager for credentials management services at AWS, including AWS Secrets Manager and IAM Roles. He enjoys the “Day 1” culture at Amazon because it aligns with his experience building startups in the sports and recruiting industries. Outside of work, Apurv enjoys hiking. He holds an MBA from UCLA and an MS in computer science from University of Kentucky.

How to connect to AWS Secrets Manager service within a Virtual Private Cloud

Post Syndicated from Divya Sridhar original https://aws.amazon.com/blogs/security/how-to-connect-to-aws-secrets-manager-service-within-a-virtual-private-cloud/

You can now use AWS Secrets Manager with Amazon Virtual Private Cloud (Amazon VPC) endpoints powered by AWS Privatelink and keep traffic between your VPC and Secrets Manager within the AWS network.

AWS Secrets Manager is a secrets management service that helps you protect access to your applications, services, and IT resources. This service enables you to rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. When your application running within an Amazon VPC communicates with Secrets Manager, this communication traverses the public internet. By using Secrets Manager with Amazon VPC endpoints, you can now keep this communication within the AWS network and help meet your compliance and regulatory requirements to limit public internet connectivity. You can start using Secrets Manager with Amazon VPC endpoints by creating an Amazon VPC endpoint for Secrets Manager with a few clicks on the VPC console or via AWS CLI. Once you create the VPC endpoint, you can start using it without making any code or configuration changes in your application.

The diagram demonstrates how Secrets Manager works with Amazon VPC endpoints. It shows how I retrieve a secret stored in Secrets Manager from an Amazon EC2 instance. When the request is sent to Secrets Manager, the entire data flow is contained within the VPC and the AWS network.

Figure 1: How Secrets Manager works with Amazon VPC endpoints

Figure 1: How Secrets Manager works with Amazon VPC endpoints

Solution overview

In this post, I show you how to use Secrets Manager with an Amazon VPC endpoint. In this example, we have an application running on an EC2 instance in VPC named vpc-5ad42b3c. This application requires a database password to an RDS instance running in the same VPC. I have stored the database password in Secrets Manager. I will now show how to:

  1. Create an Amazon VPC endpoint for Secrets Manager using the VPC console.
  2. Use the Amazon VPC endpoint via AWS CLI to retrieve the RDS database secret stored in Secrets Manager from an application running on an EC2 instance.

Step 1: Create an Amazon VPC endpoint for Secrets Manager

  1. Open the Amazon VPC console, select Endpoints, and then select Create Endpoint.
  2. Select AWS Services as the Service category, and then, in the Service Name list, select the Secrets Manager endpoint service named com.amazonaws.us-west-2.secrets-manager.
     
    Figure 2: Options to select when creating an endpoint

    Figure 2: Options to select when creating an endpoint

  3. Specify the VPC you want to create the endpoint in. For this post, I chose the VPC named vpc-5ad42b3c where my RDS instance and application are running.
  4. To create a VPC endpoint, you need to specify the private IP address range in which the endpoint will be accessible. To do this, select the subnet for each Availability Zone (AZ). This restricts the VPC endpoint to the private IP address range specific to each AZ and also creates an AZ-specific VPC endpoint. Specifying more than one subnet-AZ combination helps improve fault tolerance and make the endpoint accessible from a different AZ in case of an AZ failure. Here, I specify subnet IDs for availability zones us-west-2a, us-west-2b, and us-west-2c:
     
    Figure 3: Specifying subnet IDs

    Figure 3: Specifying subnet IDs

  5. Select the Enable Private DNS Name checkbox for the VPC endpoint. Private DNS resolves the standard Secrets Manager DNS hostname https://secretsmanager.<region>.amazonaws.com. to the private IP addresses associated with the VPC endpoint specific DNS hostname. As a result, you can access the Secrets Manager VPC Endpoint via the AWS Command Line Interface (AWS CLI) or AWS SDKs without making any code or configuration changes to update the Secrets Manager endpoint URL.
     
    Figure 4: The "Enable Private DNS Name" checkbox

    Figure 4: The “Enable Private DNS Name” checkbox

  6. Associate a security group with this endpoint. The security group enables you to control the traffic to the endpoint from resources in your VPC. For this post, I chose to associate the security group named sg-07e4197d that I created earlier. This security group has been set up to allow all instances running within VPC vpc-5ad42b3c to access the Secrets Manager VPC endpoint. Select Create endpoint to finish creating the endpoint.
     
    Figure 5: Associate a security group and create the endpoint

    Figure 5: Associate a security group and create the endpoint

  7. To view the details of the endpoint you created, select the link on the console.
     
    Figure 6: Viewing the endpoint details

    Figure 6: Viewing the endpoint details

  8. The Details tab shows all the DNS hostnames generated while creating the Amazon VPC endpoint that can be used to connect to Secrets Manager. I can now use the standard endpoint secretsmanager.us-west-2.amazonaws.com or one of the VPC-specific endpoints to connect to Secrets Manager within vpc-5ad42b3c where my RDS instance and application also resides.
     
    Figure 7: The "Details" tab

    Figure 7: The “Details” tab

Step 2: Access Secrets Manager through the VPC endpoint

Now that I have created the VPC endpoint, all traffic between my application running on an EC2 instance hosted within VPC named vpc-5ad42b3c and Secrets Manager will be within the AWS network. This connection will use the VPC endpoint and I can use it to retrieve my RDS database secret stored in Secrets Manager. I can retrieve the secret via the AWS SDK or CLI. As an example, I can use the CLI command shown below to retrieve the current version of my RDS database secret:

$aws secretsmanager get-secret-value –secret-id MyDatabaseSecret –version-stage AWSCURRENT

Since my AWS CLI is configured for us-west-2 region, it uses the standard Secrets Manager endpoint URL https://secretsmanager.us-west-2.amazonaws.com. This standard endpoint automatically routes to the VPC endpoint since I enabled support for Private DNS hostname while creating the VPC endpoint. The above command will result in the following output:


{
  "ARN": "arn:aws:secretsmanager:us-west-2:123456789012:secret:MyDatabaseSecret-a1b2c3",
  "Name": "MyDatabaseSecret",
  "VersionId": "EXAMPLE1-90ab-cdef-fedc-ba987EXAMPLE",
  "SecretString": "{\n  \"username\":\"david\",\n  \"password\":\"BnQw&XDWgaEeT9XGTT29\"\n}\n",
  "VersionStages": [
    "AWSCURRENT"
  ],
  "CreatedDate": 1523477145.713
} 

Summary

I’ve shown you how to create a VPC endpoint for AWS Secrets Manager and retrieve an RDS database secret using the VPC endpoint. Secrets Manager VPC Endpoints help you meet compliance and regulatory requirements about limiting public internet connectivity within your VPC. It enables your applications running within a VPC to use Secrets Manager while keeping traffic between the VPC and Secrets Manager within the AWS network. You can start using Amazon VPC Endpoints for Secrets Manager by creating endpoints in the VPC console or AWS CLI. Once created, your applications that interact with Secrets Manager do not require any code or configuration changes.

To learn more about connecting to Secrets Manager through a VPC endpoint, read the Secrets Manager documentation. For guidance about your overall VPC network structure, see Practical VPC Design.

If you have questions about this feature or anything else related to Secrets Manager, start a new thread in the Secrets Manager forum.

Want more AWS Security news? Follow us on Twitter.