Post Syndicated from Derik Wang original https://aws.amazon.com/blogs/security/how-to-use-the-aws-workload-credentials-provider-for-cross-account-secret-retrieval-and-prefetching-secrets/

If you manage secrets across multiple AWS accounts or need faster secret access for latency-sensitive applications, this post shows you how to meet those requirements using two new features of the AWS Workload Credentials Provider (provider). You will learn how to configure role chaining for cross-account secret retrieval and prefetching of secrets to reduce cold-start latency.

By using role chaining, you can access secrets across AWS accounts through a single provider instance by assuming AWS Identity and Access Management (IAM) roles. Prefetching populates the provider’s in-memory cache with secrets at startup so your application can retrieve secrets without waiting for the first request to trigger a network call at runtime.

AWS Secrets Manager stores and rotates credentials, API keys, and other secrets. The AWS Workload Credentials Provider is a client-side HTTP service that retrieves and caches secrets locally. This reduces latency, improves availability during transient failures, and lowers costs. It supports post-quantum TLS by default, requires no language-specific SDK, and works across Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), and AWS Lambda. For more details, see the Workload Credentials Provider documentation and GitHub repository.

The Server-Side Requst Forger (SSRF) token prevents unauthorized processes from accessing the provider’s HTTP endpoint. Only applications that can read the token file can retrieve secrets through the provider.

Any identity that can access the provider’s endpoint and SSRF token can retrieve secrets through role chaining. This means users with compute environment access can retrieve cross-account secrets when role assumption is configured. Scope the target role’s permissions to only the secrets required by following the principle of least privilege.

For prefetching, secrets are loaded into the provider’s in-memory cache at startup. Any process that can reach the provider’s localhost endpoint and provide a valid SSRF token can retrieve prefetched secrets from the cache.

Organizations might store secrets in a dedicated AWS account, or need to share one secret across applications in different accounts. Until now, cross-account retrieval through the provider required attaching resource-based policies directly to each secret. Some customers prefer IAM role assumption. Before this feature, you had to deploy multiple provider instances with different credentials or build custom credential-switching logic. The provider now supports both approaches: resource-based policies and IAM role assumption. While role assumption is especially useful for cross-account scenarios, it also helps within the same account when secrets are protected by different customer-managed KMS keys.

When you include the roleArn query parameter in a request, the provider uses AWS Security Token Service (AWS STS) AssumeRole to obtain temporary credentials for the specified role and retrieves the secret with those credentials. The provider creates and caches a separate client for each role ARN, so subsequent requests to the same role reuse the existing client. Each role client maintains its own independent cache.

Note: The source account runs the Workload Credentials Provider and your application. The target account contains the secret you want to retrieve. A single provider instance in the source account can assume roles in one or more target accounts.

• A Workload Credentials Provider built and installed in your environment (see the README for build instructions)
• AWS credentials configured in your compute environment with permission to call sts:AssumeRole on the target role ARN
  • If you also retrieve secrets from the source account through the provider, the credentials need secretsmanager:GetSecretValue and secretsmanager:DescribeSecret permissions for those secrets
• A secret in a target AWS account that you want to retrieve
• An IAM role in the target account with a trust policy that allows the provider’s identity to assume it

The provider is written in Rust and compiles to a single executable. The following steps are for an RPM-based system such as Amazon Linux 2023:

1. Install build dependencies:

   sudo yum -y groupinstall "Development Tools"

2. Install Rust:

   curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
   source "$HOME/.cargo/env"

3. Clone the repository and build the provider (use the latest tag available):

   git clone --branch <git tag> https://github.com/aws/aws-workload-credentials-provider.git
   cd aws-workload-credentials-provider
   cargo build --release

The compiled binary is at target/release/aws-workload-credentials-provider.

After building the provider, install it as a system service on your EC2 instance and configure access to the SSRF token.

1. After configuring your config.toml file (see Configuration options section), run the install script to deploy the provider as a systemd service and generate the SSRF token:

   cd aws_workload_credentials_provider_common/configuration
   sudo ./install --config config.toml

2. Add your application user to the aws-wcp-token group. This grants your application permission to read the SSRF token file, which is required for all secret retrieval requests:

   sudo usermod -aG aws-wcp-token <APP_USER>

To install on Amazon ECS, Amazon EKS, or Lambda, see the installation instructions in the GitHub repository.

1. Check that the provider is running:

   curl -v -H \
       "X-Aws-Parameters-Secrets-Token: $(</var/run/awssmatoken)" \
       'http://localhost:2773/secretsmanager/get?secretId=<YOUR_SECRET_ID>'

2. You’ll receive a JSON response with the secret value. If you see a connection refused error, check that the provider process is running. If you see a 401 or 403 error, verify the SSRF token file is readable and that the provider’s IAM credentials have secretsmanager:GetSecretValue and secretsmanager:DescribeSecret permissions.

The provider’s base IAM identity requires:

• sts:AssumeRole on the target role ARN

The target role requires:

• secretsmanager:GetSecretValue
• secretsmanager:DescribeSecret

Create an IAM role in the target account with a trust policy that allows the provider’s identity in the source account to assume it. Then attach a policy that grants access to the required secrets.

1. Create an IAM role in the target account with a trust policy that allows the provider’s identity in the source account to assume it.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": "arn:aws:iam::111111111111:role/WCProviderRole"
               },
               "Action": "sts:AssumeRole"
           }
       ]
   }
   

2. Attach a policy to this role that grants access to the secret:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret"
            ],
            "Resource": "arn:aws:secretsmanager:us-east-1:222222222222:secret:MyDatabaseSecret"
        }
    ]
}

Before the provider can assume the role you created in the target account, grant it permission to call sts:AssumeRole. Attach the following policy to the provider’s IAM role in the source account:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole"
        }
    ]
}

Call the Workload Credentials Provider endpoint with the roleArn parameter. The following curl example shows how to retrieve a secret using a different IAM role:

curl -v -H "X-Aws-Parameters-Secrets-Token: $(</var/run/awssmatoken)" 'http://localhost:2773/secretsmanager/get?secretId=MyDatabaseSecret&roleArn=arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole'

The following Python example shows the same operation:

import requests

def get_secret_cross_account():
    role_arn = "arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole"
    url = f"http://localhost:2773/secretsmanager/get?secretId=MyDatabaseSecret&roleArn={role_arn}"

    with open('/var/run/awssmatoken') as fp:
        token = fp.read()

    headers = {
        "X-Aws-Parameters-Secrets-Token": token.strip()
    }

    response = requests.get(url, headers=headers)

    if response.status_code == 200:
        return response.text
    else:
        raise Exception(f"Status code {response.status_code} - {response.text}")

You can configure the maximum number of simultaneous assumed roles with the max_roles option in the provider’s TOML configuration file. The default is 20, and the range is 1–20.

By default, the Workload Credentials Provider populates its cache lazily—the first request for a secret triggers a network call to Secrets Manager. Prefetching reduces this cold-start latency by loading secrets at startup.

You can configure prefetching by adding a [capabilities.secrets_manager.prefetch] section to the provider’s TOML configuration file. You can specify secrets to prefetch in two ways:

• Explicit secrets – List specific secret IDs or ARNs using [[capabilities.secrets_manager.prefetch.secrets]] entries.
• Tag-based discovery – Discover secrets by tag key using [[capabilities.secrets_manager.prefetch.filter_tags]] entries. The provider calls BatchGetSecretValue with tag key filters to find and cache all matching secrets.

You can use both methods together. Each entry optionally accepts a role_arn field for cross-account prefetching through role chaining.

The following permissions are required on the IAM role that performs the prefetch, depending on whether the secrets are in the source account or a target account.

• secretsmanager:BatchGetSecretValue – Required on the source account role for source-account secrets, or on the target role for cross-account secrets
• secretsmanager:ListSecrets – Required when using tag-based discovery (filter_tags), on whichever role is performing the discovery

You can tune prefetch behavior with the following options in the [capabilities.secrets_manager.prefetch] section of your TOML configuration file:

• cache_buffer_ratio – The maximum fraction of the cache to fill per caching client during prefetch, in the range 0.1–1.0. The default is 0.8. For example, if your cache holds 100 secrets, a ratio of 0.8 prefetches up to 80, leaving room for 20 on-demand secrets to be cached.
• max_jitter_seconds – The maximum random delay in seconds before starting the prefetch task, in the range 0–10. The default is 0 (no jitter). Use this to prevent fleet-wide synchronized API calls when deploying across many instances.

The following configuration prefetches two secrets at startup, one from the source account and one from a different account using role chaining:

[capabilities.secrets_manager.prefetch]
secrets = [
    { secret_id = "arn:aws:secretsmanager:us-east-1:111111111111:secret:MySecret-AbCdEf" },
    { secret_id = "cross-account-secret", role_arn = "arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole" }
]

The following configuration discovers and caches all secrets tagged with the Environment key, and all secrets tagged with the Team key in a different account:

[capabilities.secrets_manager.prefetch]
filter_tags = [
    { key = "Environment" },
    { key = "Team", role_arn = "arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole" },
]

The following example shows a complete provider configuration that combines both features:

[logging]
log_level = "info"

[capabilities.secrets_manager]
http_port = 2773
region = "us-east-1"

[capabilities.secrets_manager.cache]
ttl_seconds = 300

[capabilities.secrets_manager.prefetch]
cache_buffer_ratio = 0.6
max_jitter_seconds = 5
secrets = [
    { secret_id = "arn:aws:secretsmanager:us-east-1:111111111111:secret:MySecret-AbCdEf" },
    { secret_id = "arn:aws:secretsmanager:us-east-1:222222222222:secret:CrossAccount-AbCdEf", role_arn = "arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole" },
]
filter_tags = [
    { key = "Environment" },
    { key = "Team", role_arn = "arn:aws:iam::222222222222:role/CrossAccountSecretAccessRole" },
]

Start the provider with your configuration file:

./aws-workload-credentials-provider sm start --config config.toml

This post showed you how to use role chaining for cross-account secret retrieval and prefetching to reduce cold-start latency. Role chaining simplifies multi-account architectures—a single provider instance can retrieve secrets across accounts using IAM role assumption. Prefetching reduces cold-start latency by populating the provider’s cache before your application makes its first request. Combined, these features let you run the Workload Credentials Provider across multiple accounts with faster secret access.

• AWS Workload Credentials Provider documentation
• Access AWS Secrets Manager secrets from a different account
• GitHub repository for source code and README
• AWS Secrets Manager documentation

Submit feedback in the comments below, or contact AWS Support with questions.

Post Syndicated from Jirka Fajfr original https://aws.amazon.com/blogs/security/how-to-use-the-new-aws-secrets-manager-cost-allocation-tags-feature/

AWS Secrets Manager is a service that you can use to manage, retrieve, and rotate database credentials, application credentials, API keys, and other secrets throughout their lifecycles. You can use Secrets Manager to replace hard-coded credentials in application source code with a runtime call to the Secrets Manager service to retrieve credentials dynamically when you need them. Storing the credentials in Secrets Manager helps avoid unintended access by anyone who inspects your application’s source code, configuration, or components.

Until today, your AWS bill would reflect the total cost of Secrets Manager in any given account, and you had no option to break out the cost per secret to a given cost center or organization.

In this post, we introduce a new feature—Secrets Manager Costs Allocation Tags—and walk through how you can use them for improved visibility into your Secrets Manager costs. Before getting into the details of this new feature, we want to give you primer about cost allocation tags.

A tag is a key-value pair that you assign to an AWS resource. In AWS Cost Explorer, you can activate tags as cost allocation tags. With tags activated, you can categorize and track costs by cost allocation tags. For example, you can create a tag named Group with value Engineering and assign it to resources owned by the engineering team of your company. After activating the Group tag as a cost allocation tag, you can track charges with this tag, filter or group by tags in Cost Explorer, and add tags to reports such as cost and usage reports for further analysis and visualization.

Cost allocation in AWS is a four step process:

1. Create the required cost allocation tags
2. Attach cost allocation tags to your resources
3. Activate your tags in the Cost Allocation Tags section of the AWS Billing console
4. Filter the tags, group by tags in Cost Explorer, and create cost categories

After you create and attach the tags to resources, they appear in the AWS Billing console Cost Allocation Tags section under User-defined cost allocation tags within 24 hours. You must activate these tags for AWS to start tracking them for your resources and for the tags to show up in Cost Explorer. When the tags appear under Tags in the Filter or Group by fields in Cost Explorer, you can start filtering or grouping by tag to view usage and charges by tag.

AWS Secrets Manager now supports cost allocation tags

Secrets Manager now supports cost allocation tags, giving you more granular control and visibility into your Secrets Manager costs. You can use this feature to categorize and track your Secrets Manager usage charges at a more detailed level, helping you to better understand and manage your AWS spending and assign costs per secret back to cost centers or organizations.

Solution overview: Enhanced cost visibility and management

With this new capability, you can:

• Break down Secrets Manager costs by department, project, environment, or other dimensions important to your organization
• View itemized Secrets Manager usage in Cost Explorer as well as in cost and usage reports
• Improve cost allocation and chargeback processes across your business units and organizations

Prerequisites

To walk through the examples in this post, you need to have the following:

1. An AWS account
2. Access to the AWS Management Console or the AWS Command Line Interface (AWS CLI) version 2
3. An existing tagging and cost allocation strategy
4. Existing secrets inside Secrets Manager

Create user-defined tags for cost allocation purposes using the console

In this example, we assume that you want to manage the cost of your secrets by different cost centers in your organization. Here, we create a tag with CostCenter as a key and the value equal to the cost center codes of the teams that are using secrets.

You’ll walk through two examples, the first one with a cost center for Engineering and a second one with a cost center for Finance. You will reuse those examples throughout this post.

In this example, start by creating and assigning a tag called cost allocation center with the key name: CostCenter and assign a cost center value of 7263 for the engineering department to an existing or new secret.

To assign a user-defined tag to a new or existing secret:

1. In the Secrets Manager console, choose Secrets from the navigation pane.
2. In the list of available secrets, select the secret to edit the tags or choose Store A New Secret to create a new secret.
3. When the secret is displayed, select the Tags option and choose Edit Tags to add new or edit existing tags.
   
   

   Figure 1: Assign a user-defined tag to an existing secret

4. Repeat the process, but assign the cost center value of 7263 for the engineering department and 1121 for the finance department to a second secret.

Create user-defined tags for cost allocation purposes using the AWS CLI

Optionally, you can use the AWS CLI to create the same tags as in the preceding example.

To use the AWS CLI to create tags:

1. Use the following AWS CLI command to create the first tag:

   aws secretsmanager tag-resource \
       --secret-id prod/mastersecret \
       --tags Key=Role,Value=admin
   

2. Create the second tag using the following AWS CLI command:

   aws secretsmanager tag-resource \
       --secret-id prod/mastersecret \
       --tags Key=CostCenter,Value=7263
   

   This command produces no output in case of a successful execution.

3. Use the second AWS CLI command with a value of 1121 for the second secret.

Turn tags into cost allocation tags using the AWS Billing and Cost Management console

The next step is to activate the user-defined tags within the AWS Billing and Cost Management console so they can be used.

To activate cost allocation tags:

1. Go to the AWS Billing and Cost Management console and choose Cost allocation tags in the navigation pane.
2. Select the option for user-defined cost allocation tags.
3. Select the tag keys that you want to activate.
4. Choose Activate.

Note: After you create and apply user-defined tags to your resources, it can take up to 24 hours for the tag keys to appear on your cost allocation tags page for activation. It can then take up to 24 hours for tag keys to activate.

Figure 2: Activate cost allocation tags

Turn tags into cost allocation tags using the AWS CLI

You can also activate user-defined tags within the AWS Billing and Cost Management Console using the AWS CLI.

To activate tags using the AWS CLI:

1. For activation of the first user-defined tag use the following AWS CLI command:

   aws ce update-cost-allocation-tags-status \
       --cost-allocation-tags-status TagKey=Role,Status=Active
   

2. To activate the second user-defined tag use the following AWS CLI command:

   aws ce update-cost-allocation-tags-status \
       --cost-allocation-tags-status TagKey=CostCenter,Status=Active
   

View the results in Cost Explorer

The last step is to view the results for secrets in Cost Explorer. When the tag CostCenter shows up under Tags in the Filter or Group By fields in Cost Explorer, you can start filtering or grouping by the tag to view usage and charges by tag.

When applying the tag filter for Secrets Manager, Cost Explorer displays charges only for resources tagged with the selected tag values. And when grouped by a particular tag, the charges are grouped by each value of the selected tag.

To view results:

As an example, use the following parameters to view results.

1. In the Cost Center console, select the right arrow (>) icon to open the report parameters options to the right of the billing dashboard.
2. On the Report parameters window:
   1. For Date Range, enter the desired time range.
   2. Under Group by, for Dimension, select Tag, and for Tag select Cost Center.
   3. For Filters, Service, select Secrets Manager.
      
      

      Figure 3: Configure report parameters

You can use the resulting report to clearly identify the cost and usage of the two secrets, broken down into the two cost centers: engineering 7263 and finance 1121. Now, you can cross-charge secrets to the corresponding cost centers in your organization and provide a report similar to Figure 4.

Figure 4: Cost and usage report

Conclusion

In this post, we introduced the AWS Secrets Manager Cost Allocation Tags feature and showed you how to use AWS Cost Explorer Costs and Usage Reports to gain secrets usage insights. You can now allocate the cost for every secret to one or more cost centers and charge them accordingly. See the AWS Secrets Manager Cost Allocation Tag documentation to learn more about how you can use Secrets Manager Cost Allocation Tags in your accounts.

Go to the AWS Secrets Manager console to get started. For more information, see AWS Secrets Manager.

To learn more about how to build an effective tagging strategy for cost allocation and financial management, see the Tags for cost allocation and financial management whitepaper.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Post Syndicated from Rodrigo Bersa original https://aws.amazon.com/blogs/security/announcing-ascp-integration-with-pod-identity-enhanced-security-for-secrets-management-in-amazon-eks/

In 2021, Amazon Web Services (AWS) introduced the AWS Secrets and Configuration Provider (ASCP) for the Kubernetes Secrets Store Container Storage Interface (CSI) Driver, offering a reliable way to manage secrets in Amazon Elastic Kubernetes Service (Amazon EKS). Today, we’re excited to announce the integration of ASCP with Pod Identity, the new standard for AWS Identity and Access Management (IAM) integration with Amazon EKS. This integration provides enhanced security, simplified configuration, and improved Day-2 operation for applications running on EKS that need access to secrets stored in AWS Secrets Manager and AWS Systems Manager Parameter Store. In this post, we’ll cover use-case scenarios for single and cross-account secrets using Pod Identity integration with ASCP.

Amazon EKS introduced Pod Identity in 2023 as a feature that streamlines the process of configuring IAM permissions for Kubernetes applications. Pod Identity simplifies the identity management of applications running on top of Amazon EKS by allowing you to set up permissions directly through EKS interfaces, reducing the number of steps required and alleviating the need to switch between the EKS and IAM services. Pod Identity enables the use of a single IAM role across multiple clusters without updating trust policies, and it supports role session tags for more granular access control. This approach not only simplifies policy management by allowing the reuse of permission policies across roles, but also enhances security by enabling access to AWS resources based on matching tags.

Background

When your applications that are running on top of Amazon EKS require sensitive information like credentials to access a database, or a key to authenticate through an API, we call this kind of information secrets. You can secure, store, and manage secrets in AWS Secrets Manager. ASCP allows Kubernetes applications to securely retrieve the secrets stored in Secrets Manager and Systems Manager Parameter Store. Previously, ASCP relied on IAM roles for service accounts (IRSA) for authentication. Although IRSA provided improvements over previous methods, Pod Identity offers even greater security and simplicity.

Pod Identity provides a more secure and efficient way to integrate IAM roles with applications running on EKS, granting more granular AWS permissions to individual Pods, so that you don’t need instance-level credentials or IRSA.

This integration provides the following key benefits over using IRSA:

• Enhanced security: Pod Identity provides a more granular and secure way to manage permissions at the Pod level, alleviating the need to expose the IAM role annotation on Kubernetes ServiceAccount objects.
• Simplified configuration:Pod Identity streamlines and simplifies the setup process, reducing the potential for misconfiguration, especially in high-scale environments.
• Improved operation: Pod Identity reduces the operational overhead compared to previous methods, centralizing the management with the AWS API.
• Native EKS integration: Pod Identity is the new standard for IAM integration for applications running on EKS and provides a more cohesive experience.

Solution overview

With this integration, ASCP uses Pod Identity to authenticate and authorize access to AWS services. When a Pod requires access to a secret, the workflow is as shown in Figure 1.

Figure 1: The workflow performed by the Pod Identity and ASCP integration to provide access to a secret stored on AWS Secrets Manager for a Pod running on Amazon EKS

The workflow is as follows:

1. A user creates an IAM role and a Pod Identity association between the IAM role and the Kubernetes ServiceAccount assigned to the Pod.
2. EKS API validates the Pod Identity association, allowing ASCP to use this role to authenticate with AWS services.
3. If authorized, the Pod Identity agent allows ASCP to assume the IAM role assigned to the Pod through the use of a ServiceAccount token.
4. The Pod retrieves the requested secrets values and makes them available to the Pod through the use of a mounted volume.

Prerequisites

You need to have the following prerequisites in place in order to implement this solution:

• An Amazon EKS cluster (version 1.24 or later)
• Pod Identity enabled on your cluster
• Access to two AWS accounts (for cross-account access)
• AWS CLI installed
• kubectl installed
• Helm installed

Implement the solution

This guide presents two scenarios: single-account setup and cross-account setup. Complete the single-account steps in Account A before proceeding to the cross-account configuration, which involves Account B. The cross-account setup builds on the single-account foundation to demonstrate secure secrets management across AWS accounts.

Amazon EKS cluster setup

Before you start, you’ll need to set up an Amazon EKS cluster with the required add-ons in a single account (Account A).

1. (Optional) Use the following commands to set environment variables and create an Amazon EKS cluster:

   export CLUSTER_NAME=<YOUR_CLUSTER_NAME>
   export AWS_REGION=<YOUR_AWS_REGION>
   export NAMESPACE=<YOUR_NAMESPACE>
   export ACCOUNT_A=<ACCOUNT_ID_1>
   export ACCOUNT_B=<ACCOUNT_ID_2>
   export SECRET_NAME=<NAME_OF_YOUR_SECRET>
   export SECRET_NAME_CROSS=<NAME_OF_YOUR_CROSS_SECRET>

   Here are the environment variables for this demonstration:

   export CLUSTER_NAME=ascp-podidentity
   export SECRET_NAME=secret-a
   export AWS_REGION=$(aws configure get region)
   export NAMESPACE=default
   export SECRET_NAME=secret-a
   export SECRET_NAME_CROSS=secret-b

   Create the EKS cluster:

   eksctl create cluster —name $CLUSTER_NAME —version=1.31

2. Update your kubeconfig file to enable Pod Identity on your EKS cluster (if it’s not already enabled):

   eksctl create addon --cluster $CLUSTER_NAME --name eks-pod-identity-agent

3. Install the Secrets Store CSI driver:

   aws eks --region $AWS_REGION update-kubeconfig --name $CLUSTER_NAME
   helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
   helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver

4. Verify the Secrets Store CSI Driver installation:

   kubectl --namespace=kube-system get pods -l "app=secrets-store-csi-driver"

   You should see this expected output:

   NAME READY STATUS RESTARTS AGE
   csi-secrets-store-secrets-store-csi-driver-fcxf6 3/3 Running 0 41s
   csi-secrets-store-secrets-store-csi-driver-j9wqh 3/3 Running 0 41s

5. Install the ASCP plugin:

   kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

Consuming secrets in a single account

Now that you’ve set up your Amazon EKS cluster, in this use case, you’ll create an AWS Secrets Manager secret in the same account where the cluster and the applications reside (Account A).

1. Create a secret in AWS Secrets Manager with tags kubernetes-namespace and eks-cluster-name within the same AWS account as the EKS cluster:

   aws secretsmanager create-secret --region $AWS_REGION \
   --name $SECRET_NAME \
   --tags Key=kubernetes-namespace,Value=$NAMESPACE Key=eks-cluster-name,Value=$CLUSTER_NAME \
   --secret-string "{"user":"user1","password":"passwd1"}"

   You should see this expected output:

   {
   "ARN": "arn:aws:secretsmanager:$AWS_REGION:$ACCOUNT_A:secret:dummy-secret-q1w2e3",
   "Name": "secret-a",
   "VersionId": "abcdefgh-1234-5678-ijklmnopqrst"
   }

   Note the Amazon Resource Name (ARN) of the new secret. You will use it in the next step.

2. Create an IAM role and trust policy with the required permissions to retrieve the newly created secret within the same AWS account:

   SECRET_ARN=< NOTED_SECRET_ARN_FROM_PREVIOUS_STEP>
   cat << EOF > policy.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
               "Resource": ["$SECRET_ARN"],
               "Condition": {
                   "StringEquals": {
                       "secretsmanager:ResourceTag/kubernetes-namespace": "\${aws:PrincipalTag/kubernetes-namespace}",
                       "secretsmanager:ResourceTag/eks-cluster-name": "\${aws:PrincipalTag/eks-cluster-name}"
                   }
               }
           }
       ]
   }
   EOF

   In the preceding example IAM role permission policy, the use of conditions with kubernetes-namespace and eks-cluster-name tags helps enforce fine-grained access control by specifying that secrets can only be accessed by Pods from specific namespace and clusters. This allows secretsmanager actions only if the secret is tagged with the matching kubernetes-namespace and  eks-cluster-name values.

   cat << EOF > trust.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {  
               "Effect": "Allow",
               "Principal": {
                   "Service": "pods.eks.amazonaws.com" 
               },
               "Action": [
                   "sts:AssumeRole",
                   "sts:TagSession"
               ]
           }
       ]
   }
   EOF

   Notice that the preceding example IAM role trust policy allows the Amazon EKS Pod Identity service (pods.eks.amazonaws.com) to assume the role and tag the session. These actions are necessary for Pod Identity to function correctly, enabling Pods to securely access AWS resources.

   Finally, apply the role and trust policy:

   aws iam create-role --role-name ascp-podidentity --assume-role-policy-document file://trust.json
   aws iam put-role-policy --role-name ascp-podidentity --policy-name ascp-podidentity --policy-document file://policy.json

   Note the ARN of the new role. You will use it in the next step.

3. Create a Kubernetes ServiceAccount and add the Pod Identity association between the ServiceAccount and the IAM role:

   ROLE_ARN=<NOTED_ROLE_ARN_FROM_PREVIOUS_STEP>
   kubectl create serviceaccount serviceaccount-a
   
   aws eks create-pod-identity-association \
   --cluster-name "$CLUSTER_NAME" \
   --namespace "default" \
   --service-account serviceaccount-a \
   --role-arn $ROLE_ARN

4. Create a SecretProviderClass to use the newly created secret in your Amazon EKS cluster.

   cat << EOF | kubectl apply -f -
   
   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: aws-secrets-manager
   spec:
     provider: aws
     parameters:
       objects: |
         - objectName: "secret-a" # Secret name or ARN to be mounted to the Pod
           objectType: "secretsmanager"
       usePodIdentity: "true" # Indicator to use Pod Identity instead of IRSA
       
   EOF

5. Create a new deployment to consume your newly created secret using Pod Identity as a mounted volume.

   cat << EOF | kubectl apply -f -
   
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: application-a
     name: application-a
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: application-a
     template:
       metadata:
         labels:
           app: application-a
       spec:
         containers:
         - image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
           name: amazonlinux
           args:
           - infinity
           command:
           - sleep
    volumeMounts:
    - name: secrets-store-inline
    mountPath: "/mnt/secret" # Directory where secret will be mounted to
    readOnly: true
    volumes:
    - name: secrets-store-inline
    csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
    secretProviderClass: "aws-secrets-manager" # Match SecretProviderClass name created
    serviceAccountName: serviceaccount-a # Specify service account created
   
   EOF

6. Validate that the deployment was created successfully and confirm the secret was mounted correctly.

   kubectl get pods -l app=application-a
   kubectl exec -it $(kubectl get pods -l app=application-a -o name) -- cat /mnt/secret/secret-a

   You should see this expected output:

   NAME READY STATUS RESTARTS AGE
   application-a-b98d44bb8-bzvn4 1/1 Running 0 3s
   
   {"user":"user1","password":"passwd1"}%

Working with cross-account secrets through resource policies

For this second use case, you’ll also use the Amazon EKS cluster on Account A, and create a new Secrets Manager secret in a different account (Account B).

In order to access a secret in a different account, you can’t use the default AWS Key Management Service (AWS KMS) key, but will use aws/secretsmanager to encrypt this secret. So you need to first create a new AWS KMS key that allows cross-account access.

On Account B

1. Create a customer managed key on AWS KMS with cross-account permissions:

   cat < key-policy.json
   {
     "Version": "2012-10-17",
     "Id": "cross-account-access",
     "Statement": [
       {
         "Sid": "Enable IAM User Permissions",
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::${ACCOUNT_B}:root"
         },
         "Action": "kms:*",
         "Resource": "arn:aws:kms:${AWS_REGION}:${ACCOUNT_B}:key/*"
       },
       {
         "Sid": "Allow access for Key Administrators",
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::${ACCOUNT_B}:role/Admin"
         },
         "Action": [
           "kms:Create",
           "kms:Describe",
           "kms:Enable",
           "kms:List",
           "kms:Put",
           "kms:Update",
           "kms:Revoke",
           "kms:Disable",
           "kms:Get",
           "kms:Delete",
           "kms:TagResource",
           "kms:UntagResource",
           "kms:ScheduleKeyDeletion",
           "kms:CancelKeyDeletion",
           "kms:RotateKeyOnDemand"
         ],
         "Resource": "arn:aws:kms:${AWS_REGION}:${ACCOUNT_B}:key/*"
       },
       {
         "Sid": "Allow access through AWS Secrets Manager",
         "Effect": "Allow",
         "Principal": {
           "AWS": "*"
         },
         "Action": [
           "kms:Encrypt",
           "kms:Decrypt",
           "kms:ReEncrypt",
           "kms:CreateGrant",
           "kms:DescribeKey"
         ],
         "Resource": "arn:aws:kms:${AWS_REGION}:${ACCOUNT_B}:key/*",
         "Condition": {
           "StringEquals": {
             "kms:ViaService": "secretsmanager.${AWS_REGION}.amazonaws.com",
             "kms:CallerAccount": [
               "$ACCOUNT_A",
               "$ACCOUNT_B"
             ]
           }
         }
       },
       {
         "Sid": "Allow access through AWS Secrets Manager for Generate Data Key",
         "Effect": "Allow",
         "Principal": {
           "AWS": "*"
         },
         "Action": "kms:GenerateDataKey",
         "Resource": "arn:aws:kms:${AWS_REGION}:${ACCOUNT_B}:key/*",
         "Condition": {
           "StringEquals": {
             "kms:CallerAccount": [
               "$ACCOUNT_A",
               "$ACCOUNT_B"
             ]
           },
           "StringLike": {
             "kms:ViaService": "secretsmanager.${AWS_REGION}.amazonaws.com"
           }
         }
       },
       {
         "Sid": "Allow direct access to key metadata to the account",
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "arn:aws:iam::${ACCOUNT_A}:root",
             "arn:aws:iam::${ACCOUNT_B}:root"
           ]
         },
         "Action": [
           "kms:Describe",
           "kms:Get",
           "kms:List",
           "kms:RevokeGrant"
         ],
         "Resource": "arn:aws:kms:${AWS_REGION}:${ACCOUNT_B}:key/*"
       }
     ]
   }
   EOF

   Then create the KMS key:

   $ aws kms create key --region $AWS_REGION --policy file://key-policy.json

   You should see this expected output:

   {
       "KeyMetadata": {
           "AWSAccountId": "$ACCOUNT_B",
           "KeyId": "abcdefgh-1234-5678-90ij-klmnopqrstuv",
           "Arn": "arn:aws:kms:${AWS_REGION}:${ACCOUNT_B}:key/abcdefgh-1234-5678-90ij-klmnopqrstuv",
           "CreationDate": "2025-01-23T18:00:00.756000-05:00",
           "Enabled": true,
           "Description": "",
           "KeyUsage": "ENCRYPT_DECRYPT",
           "KeyState": "Enabled",
           "Origin": "AWS_KMS",
           "KeyManager": "CUSTOMER",
           "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
           "KeySpec": "SYMMETRIC_DEFAULT",
           "EncryptionAlgorithms": [
               "SYMMETRIC_DEFAULT"
           ],
           "MultiRegion": false
       }
   }

   Take note of the KeyID and the ARN of the key. You can use it to create an alias for the newly created key:

   KEY_ID=<NOTED_KEY_ID_FROM_PREVIOUS_STEP>
   aws kms create-alias --region $AWS_REGION —-alias-name alias/secretsmanager-cross —-target-key-id $KEY_ID

2. Create a new secret in Secrets Manager, using the new KMS key to encrypt it:

   aws secretsmanager create-secret --region $AWS_REGION \
   --name $SECRET_NAME_CROSS \
   --tags Key=kubernetes-namespace,Value=$NAMESPACE Key=eks-cluster-name,Value=$CLUSTER_NAME \
   --secret-string "This is a Cross Account Secret" \
   --kms-key-id $KEY_ID

   You should see this expected output:

   {
   "ARN": "arn:aws:secretsmanager:$AWS_REGION:$ACCOUNT_B:secret:secret-b-12345",
   "Name": "secret-b",
   "VersionId": "abcdefgh-1234-5678-ijklmnopqrst"
   }

   Note the ARN of the new secret. You will use it in the next step.

3. Add a resource policy that allows the IAM role on Account A to access the newly created secret:

   SECRET_CROSS_ARN=< NOTED_SECRET_CROSS_ARN_FROM_PREVIOUS_STEP>
   cat << EOF > resource-policy.json
   
   {
     "Version" : "2012-10-17",
     "Statement" : [ {
       "Effect" : "Allow",
       "Principal" : {
         "AWS" : "arn:aws:iam::${ACCOUNT_A}:root"
       },
       "Action" : [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ],
       "Resource" : "$SECRET_CROSS_ARN"
       } 
     ]
   }
   
   EOF
   aws secretsmanager put-resource-policy --secret-id $SECRET_NAME_CROSS --resource-policy file://resource-policy.json

Now let’s go back to Account A.

On Account A

4. Create a new IAM role and trust policy with the required permissions to retrieve the newly created secret within Account B:

   KEY_ARN=< NOTED_KEY_ARN_FROM_PREVIOUS_STEP>
   cat << EOF > cross-policy.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
               "Resource": ["$SECRET_CROSS_ARN"],
               "Condition": {
                   "StringEquals": {
                       "secretsmanager:ResourceTag/kubernetes-namespace": "\${aws:PrincipalTag/kubernetes-namespace}",
                       "secretsmanager:ResourceTag/eks-cluster-name": "\${aws:PrincipalTag/eks-cluster-name}"
                   }
               }
           },
           {
               "Effect": "Allow",
               "Action": [
                   "kms:Decrypt",
                   "kms:DescribeKey"
                   ],
               "Resource": "$KEY_ARN"
           }
       ]
   }
   EOF

   In the preceding example IAM role permission policy, note that the resource ARN is pointing to the secret created on Account B.

   cat << EOF > trust.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {  
               "Effect": "Allow",
               "Principal": {
                   "Service": "pods.eks.amazonaws.com" 
               },
               "Action": [
                   "sts:AssumeRole",
                   "sts:TagSession"
               ]
           }
       ]
   }
   EOF

   The IAM role trust policy needs to have pods.eks.amazonaws.com as a trusted entity to perform the sts:AssumeRole and sts:TagSession actions.

   Next, create the role and apply the policy:

   aws iam create-role --role-name ascp-podidentity-cross --assume-role-policy-document file://trust.json
   aws iam put-role-policy --role-name ascp-podidentity-cross --policy-name ascp-podidentity-cross --policy-document file://cross-policy.json

   Note the ARN of the new role. You will use it in the next step.

5. Create the second Kubernetes ServiceAccount and add the cross-account Pod Identity association:

   kubectl create serviceaccount serviceaccount-b
   
   ROLE_CROSS_ARN=<NOTED_ROLE_CROSS_ARN_FROM_PREVIOUS_STEP>
   aws eks create-pod-identity-association \
   --cluster-name "$CLUSTER_NAME" \
   --namespace "default" \
   --service-account serviceaccount-b \
   --role-arn $ROLE_CROSS_ARN

6. Create a SecretProviderClass to use the cross-account secret created in Account B in your Amazon EKS cluster:

   cat << EOF | kubectl apply -f -
   
   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: aws-secrets-manager-cross
   spec:
     provider: aws
     parameters:
       objects: |
         - objectName: "$SECRET_CROSS_ARN" # Full ARN of the Secret in the ACCOUNT B to be mounted on Pod
           objectType: "secretsmanager"
       usePodIdentity: "true" # Indicator to use Pod Identity instead of IRSA
       
   EOF

7. Create a new deployment to consume your newly created secret using Pod Identity as a mounted volume:

   cat << EOF | kubectl apply -f -
   
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: application-b
     name: application-b
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: application-b
     template:
       metadata:
         labels:
           app: application-b
       spec:
         containers:
         - image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
           name: amazonlinux
           args:
           - infinity
           command:
           - sleep
    volumeMounts:
    - name: secrets-store-cross
    mountPath: "/mnt/secret" # Directory where secret will be mounted to readOnly: true
    volumes:
    - name: secrets-store-cross
    csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
    secretProviderClass: "aws-secrets-manager-cross" # Match SecretProviderClass name created for cross-account access serviceAccountName: serviceaccount-b # Specify service account created for cross-account access
   
   EOF

8. Validate that the deployment was created successfully and confirm the mounted secret:

   kubectl get pods -l app=application-b
   kubectl exec -it $(kubectl get pods -l app=application-b -o name) -- cat /mnt/secret/$SECRET_CROSS_ARN

   Note that the volume name is the secret ARN, because it’s a cross-account secret.

   You should see this expected output:

   NAME READY STATUS RESTARTS AGE
   application-b-67b755444f-ngrhv 1/1 Running 0 8s
   
   "This is a Cross Account Secret"%

Conclusion

The integration of ASCP with Pod Identity marks a significant step forward in secrets management for Amazon EKS. It offers enhanced security, simplified configuration, and improved operations. We encourage all EKS users to explore this new integration and take advantage of its benefits.

The integration of ASCP with Pod Identity offers these benefits over IRSA:

1. Simplified setup: With Pod Identity, you don’t need to create and manage service accounts for each application.
2. Enhanced security: Pod Identity provides more granular control over permissions at the Pod level.
3. Improved scalability: Pod Identity is easier to implement in large-scale environments.
4. Consistent AWS experience: Pod Identity aligns more closely with AWS best practices for IAM management.

For more information, see our documentation for: AWS Secrets Manager, AWS Secrets CSI Store Provider (ASCP), and Amazon EKS Pod Identity.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
 

Post Syndicated from Shagun Beniwal original https://aws.amazon.com/blogs/security/how-to-centrally-manage-secrets-with-aws-secrets-manager/

In today’s digital landscape, managing secrets, such as passwords, API keys, tokens, and other credentials, has become a critical task for organizations. For some Amazon Web Services (AWS) customers, centralized management of secrets can be a robust and efficient solution to address this challenge. In this post, we delve into using AWS data protection services such as AWS Secrets Manager and AWS Key Management Service (AWS KMS) to help make secrets management easier in your environment by centrally managing them from a designated AWS account.

Centralized secrets management involves the consolidation of sensitive information into a single, secure repository. This repository acts as a centralized vault where secrets are stored, accessed, and managed with strict security controls. Centralizing secrets can help organizations enforce uniform security policies, streamline access control, and mitigate the risk of unauthorized access or leakage.

This approach offers several key benefits. First, it can enhance security by reducing the threat surface and providing a single point of control for managing access to sensitive information. Additionally, centralized secrets management can facilitate compliance with regulatory requirements by enforcing strict access controls and audit trails.

Furthermore, centralization promotes efficiency and scalability by enabling automated workflows for secret rotation, provisioning, and revocation. This automation reduces administrative tasks and minimizes the risk of human error, enhancing overall operational excellence.

Overview

In this post, we’ll walk you through how to set up a centralized account for managing your secrets and their lifecycle by using AWS Lambda rotation functions. Furthermore, to facilitate efficient access and management across multiple member accounts, we’ll discuss how to establish tunnelling through VPC peering to enable seamless communication between the Centralized Security Account in this architecture and the associated member accounts.

Notably, applications within the member accounts will directly access the secrets stored in the Centralized Security Account through the use of resource policies, streamlining the retrieval process. Additionally, using AWS provided DNS within the Centralized Security Account’s virtual private cloud (VPC) will automate the resolution of database host addresses to their respective control plane IP addresses. This functionality allows AWS Lambda function traffic to efficiently traverse the peering connection, enhancing overall system performance and reliability.

Figure 1 shows the solution architecture. The architecture has four accounts that are managed through AWS Organizations. Out of these four accounts, there are three workload accounts designated as Account A, Account B, and Account C that host the application and database for serving user requests, and a Centralized Security Account from which the secrets will be maintained and managed. VPC 1 from every workload account (Account A, Account B, and Account C) is peered with VPC 1 (part of the Centralized Security Account) to allow communication between workload accounts and the secrets management account. For high availability, secrets are also replicated to a different AWS Region.

Figure 1: Sample solution architecture for centrally managing secrets

Deploy the solution

Follow the steps in this section to deploy the solution.

Step 1: Create secrets, including database secrets, in your Centralized Security Account

First, create the secrets you want to use for this walkthrough. For example, the database secrets will have a following parameters:

{
    "engine": " sql”,
    "username": " admin ",
    "password": "EXAMPLE-PASSWORD",
    "host": "<cross account DB host URL>",
    "dbInstanceIdentifier": "<cross account DB instance identifier>"
    "port": "3306"
}

Step 2: Deploy the rotation Lambda function where needed

For secrets that require automatic rotation to be turned on, deploy the rotation Lambda function from the serverless application list.

Step 3: Set up networking for Lambda to reach the Secrets Manager service endpoint

To provide connectivity to the Lambda function, you can either deploy a VPC endpoint with Private DNS enabled or a NAT gateway.

Deploy a VPC endpoint with Private DNS enabled

Create a NAT gateway

Alternatively, you can give your function internet access. Place the function in private subnets and route the outbound traffic to a NAT gateway in a public subnet. The NAT gateway has a public IP address and connects to the internet through the VPC’s internet gateway. To create a NAT gateway, follow the steps described in this AWS re:post article.

Step 4: Deploy VPC peering

Next, deploy VPC peering between the Centralized Security Account and the member accounts that hold the database.

Step 5: Set up resource-based policies on each secret

After the secrets are deployed into the Centralized Security Account, to allow application roles or users in other accounts to access the secrets (known as cross-account access), you must allow access in both a resource policy and in an identity policy. This is different than granting access to identities in the same account rather than the secret.

The access policies mentioned here are just for the example in this post. In a production environment, only provide the needed granular permissions by exercising least privilege principles.

What challenges does this solution present, and how can you overcome them?

Along with the advantages discussed in this post, there are a few challenges you should anticipate while deploying this solution:

1. Currently there is a maximum of 20,480 characters allowed in a resource-based permissions policy attached to a secret. For organizations where a large number of external accounts need to be given access to a secret, you will need to keep this quota in mind.
2. There is also a limit on the total number of active VPC peering connections per VPC. By default, the limit is 50 connections, but this is adjustable up to 125. If you require more connections across VPCs, you can use other solutions, like a transit gateway, as an alternative.
3. As the number of applications that require access to secrets from the Centralized Security Account increases, the number of external accesses will also increase, and access control might become difficult over time. To reduce the number of external accounts that have access to the Centralized Security Account, you may choose to use AWS IAM Access Analyzer.

Conclusion

In this post, we provided you with a step-by-step solution to establish a Centralized Security Account that uses the AWS Secrets Manager service for securely storing your secrets in a central place. The post outlined the process of deploying AWS Lambda functions to facilitate automatic rotation of necessary secrets. Furthermore, we delved into the implementation of VPC peering to provide uninterrupted connectivity between the rotation function and your databases or applications housed in different AWS accounts, helping to ensure smooth rotation.

Finally, we discussed the essential policies that are needed to enable applications to use these secrets through resource-based policies. This implementation provides a way for you to conveniently monitor and audit your secrets.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.