Post Syndicated from Rodrigo Bersa original https://aws.amazon.com/blogs/security/announcing-ascp-integration-with-pod-identity-enhanced-security-for-secrets-management-in-amazon-eks/

In 2021, Amazon Web Services (AWS) introduced the AWS Secrets and Configuration Provider (ASCP) for the Kubernetes Secrets Store Container Storage Interface (CSI) Driver, offering a reliable way to manage secrets in Amazon Elastic Kubernetes Service (Amazon EKS). Today, we’re excited to announce the integration of ASCP with Pod Identity, the new standard for AWS Identity and Access Management (IAM) integration with Amazon EKS. This integration provides enhanced security, simplified configuration, and improved Day-2 operation for applications running on EKS that need access to secrets stored in AWS Secrets Manager and AWS Systems Manager Parameter Store. In this post, we’ll cover use-case scenarios for single and cross-account secrets using Pod Identity integration with ASCP.

Amazon EKS introduced Pod Identity in 2023 as a feature that streamlines the process of configuring IAM permissions for Kubernetes applications. Pod Identity simplifies the identity management of applications running on top of Amazon EKS by allowing you to set up permissions directly through EKS interfaces, reducing the number of steps required and alleviating the need to switch between the EKS and IAM services. Pod Identity enables the use of a single IAM role across multiple clusters without updating trust policies, and it supports role session tags for more granular access control. This approach not only simplifies policy management by allowing the reuse of permission policies across roles, but also enhances security by enabling access to AWS resources based on matching tags.

Background

When your applications that are running on top of Amazon EKS require sensitive information like credentials to access a database, or a key to authenticate through an API, we call this kind of information secrets. You can secure, store, and manage secrets in AWS Secrets Manager. ASCP allows Kubernetes applications to securely retrieve the secrets stored in Secrets Manager and Systems Manager Parameter Store. Previously, ASCP relied on IAM roles for service accounts (IRSA) for authentication. Although IRSA provided improvements over previous methods, Pod Identity offers even greater security and simplicity.

Pod Identity provides a more secure and efficient way to integrate IAM roles with applications running on EKS, granting more granular AWS permissions to individual Pods, so that you don’t need instance-level credentials or IRSA.

This integration provides the following key benefits over using IRSA:

• Enhanced security: Pod Identity provides a more granular and secure way to manage permissions at the Pod level, alleviating the need to expose the IAM role annotation on Kubernetes ServiceAccount objects.
• Simplified configuration:Pod Identity streamlines and simplifies the setup process, reducing the potential for misconfiguration, especially in high-scale environments.
• Improved operation: Pod Identity reduces the operational overhead compared to previous methods, centralizing the management with the AWS API.
• Native EKS integration: Pod Identity is the new standard for IAM integration for applications running on EKS and provides a more cohesive experience.

Solution overview

With this integration, ASCP uses Pod Identity to authenticate and authorize access to AWS services. When a Pod requires access to a secret, the workflow is as shown in Figure 1.

Figure 1: The workflow performed by the Pod Identity and ASCP integration to provide access to a secret stored on AWS Secrets Manager for a Pod running on Amazon EKS

The workflow is as follows:

1. A user creates an IAM role and a Pod Identity association between the IAM role and the Kubernetes ServiceAccount assigned to the Pod.
2. EKS API validates the Pod Identity association, allowing ASCP to use this role to authenticate with AWS services.
3. If authorized, the Pod Identity agent allows ASCP to assume the IAM role assigned to the Pod through the use of a ServiceAccount token.
4. The Pod retrieves the requested secrets values and makes them available to the Pod through the use of a mounted volume.

Prerequisites

You need to have the following prerequisites in place in order to implement this solution:

• An Amazon EKS cluster (version 1.24 or later)
• Pod Identity enabled on your cluster
• Access to two AWS accounts (for cross-account access)
• AWS CLI installed
• kubectl installed
• Helm installed

Implement the solution

This guide presents two scenarios: single-account setup and cross-account setup. Complete the single-account steps in Account A before proceeding to the cross-account configuration, which involves Account B. The cross-account setup builds on the single-account foundation to demonstrate secure secrets management across AWS accounts.

Amazon EKS cluster setup

Before you start, you’ll need to set up an Amazon EKS cluster with the required add-ons in a single account (Account A).

1. (Optional) Use the following commands to set environment variables and create an Amazon EKS cluster:

   export CLUSTER_NAME=<YOUR_CLUSTER_NAME>
   export AWS_REGION=<YOUR_AWS_REGION>
   export NAMESPACE=<YOUR_NAMESPACE>
   export ACCOUNT_A=<ACCOUNT_ID_1>
   export ACCOUNT_B=<ACCOUNT_ID_2>
   export SECRET_NAME=<NAME_OF_YOUR_SECRET>
   export SECRET_NAME_CROSS=<NAME_OF_YOUR_CROSS_SECRET>

   Here are the environment variables for this demonstration:

   export CLUSTER_NAME=ascp-podidentity
   export SECRET_NAME=secret-a
   export AWS_REGION=$(aws configure get region)
   export NAMESPACE=default
   export SECRET_NAME=secret-a
   export SECRET_NAME_CROSS=secret-b

   Create the EKS cluster:

   eksctl create cluster —name $CLUSTER_NAME —version=1.31

2. Update your kubeconfig file to enable Pod Identity on your EKS cluster (if it’s not already enabled):

   eksctl create addon --cluster $CLUSTER_NAME --name eks-pod-identity-agent

3. Install the Secrets Store CSI driver:

   aws eks --region $AWS_REGION update-kubeconfig --name $CLUSTER_NAME
   helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
   helm install -n kube-system csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver

4. Verify the Secrets Store CSI Driver installation:

   kubectl --namespace=kube-system get pods -l "app=secrets-store-csi-driver"

   You should see this expected output:

   NAME READY STATUS RESTARTS AGE
   csi-secrets-store-secrets-store-csi-driver-fcxf6 3/3 Running 0 41s
   csi-secrets-store-secrets-store-csi-driver-j9wqh 3/3 Running 0 41s

5. Install the ASCP plugin:

   kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

Consuming secrets in a single account

Now that you’ve set up your Amazon EKS cluster, in this use case, you’ll create an AWS Secrets Manager secret in the same account where the cluster and the applications reside (Account A).

1. Create a secret in AWS Secrets Manager with tags kubernetes-namespace and eks-cluster-name within the same AWS account as the EKS cluster:

   aws secretsmanager create-secret --region $AWS_REGION \
   --name $SECRET_NAME \
   --tags Key=kubernetes-namespace,Value=$NAMESPACE Key=eks-cluster-name,Value=$CLUSTER_NAME \
   --secret-string "{"user":"user1","password":"passwd1"}"

   You should see this expected output:

   {
   "ARN": "arn:aws:secretsmanager:$AWS_REGION:$ACCOUNT_A:secret:dummy-secret-q1w2e3",
   "Name": "secret-a",
   "VersionId": "abcdefgh-1234-5678-ijklmnopqrst"
   }

   Note the Amazon Resource Name (ARN) of the new secret. You will use it in the next step.

2. Create an IAM role and trust policy with the required permissions to retrieve the newly created secret within the same AWS account:

   SECRET_ARN=< NOTED_SECRET_ARN_FROM_PREVIOUS_STEP>
   cat << EOF > policy.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
               "Resource": ["$SECRET_ARN"],
               "Condition": {
                   "StringEquals": {
                       "secretsmanager:ResourceTag/kubernetes-namespace": "\${aws:PrincipalTag/kubernetes-namespace}",
                       "secretsmanager:ResourceTag/eks-cluster-name": "\${aws:PrincipalTag/eks-cluster-name}"
                   }
               }
           }
       ]
   }
   EOF

   In the preceding example IAM role permission policy, the use of conditions with kubernetes-namespace and eks-cluster-name tags helps enforce fine-grained access control by specifying that secrets can only be accessed by Pods from specific namespace and clusters. This allows secretsmanager actions only if the secret is tagged with the matching kubernetes-namespace and  eks-cluster-name values.

   cat << EOF > trust.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {  
               "Effect": "Allow",
               "Principal": {
                   "Service": "pods.eks.amazonaws.com" 
               },
               "Action": [
                   "sts:AssumeRole",
                   "sts:TagSession"
               ]
           }
       ]
   }
   EOF

   Notice that the preceding example IAM role trust policy allows the Amazon EKS Pod Identity service (pods.eks.amazonaws.com) to assume the role and tag the session. These actions are necessary for Pod Identity to function correctly, enabling Pods to securely access AWS resources.

   Finally, apply the role and trust policy:

   aws iam create-role --role-name ascp-podidentity --assume-role-policy-document file://trust.json
   aws iam put-role-policy --role-name ascp-podidentity --policy-name ascp-podidentity --policy-document file://policy.json

   Note the ARN of the new role. You will use it in the next step.

3. Create a Kubernetes ServiceAccount and add the Pod Identity association between the ServiceAccount and the IAM role:

   ROLE_ARN=<NOTED_ROLE_ARN_FROM_PREVIOUS_STEP>
   kubectl create serviceaccount serviceaccount-a
   
   aws eks create-pod-identity-association \
   --cluster-name "$CLUSTER_NAME" \
   --namespace "default" \
   --service-account serviceaccount-a \
   --role-arn $ROLE_ARN

4. Create a SecretProviderClass to use the newly created secret in your Amazon EKS cluster.

   cat << EOF | kubectl apply -f -
   
   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: aws-secrets-manager
   spec:
     provider: aws
     parameters:
       objects: |
         - objectName: "secret-a" # Secret name or ARN to be mounted to the Pod
           objectType: "secretsmanager"
       usePodIdentity: "true" # Indicator to use Pod Identity instead of IRSA
       
   EOF

5. Create a new deployment to consume your newly created secret using Pod Identity as a mounted volume.

   cat << EOF | kubectl apply -f -
   
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: application-a
     name: application-a
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: application-a
     template:
       metadata:
         labels:
           app: application-a
       spec:
         containers:
         - image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
           name: amazonlinux
           args:
           - infinity
           command:
           - sleep
    volumeMounts:
    - name: secrets-store-inline
    mountPath: "/mnt/secret" # Directory where secret will be mounted to
    readOnly: true
    volumes:
    - name: secrets-store-inline
    csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
    secretProviderClass: "aws-secrets-manager" # Match SecretProviderClass name created
    serviceAccountName: serviceaccount-a # Specify service account created
   
   EOF

6. Validate that the deployment was created successfully and confirm the secret was mounted correctly.

   kubectl get pods -l app=application-a
   kubectl exec -it $(kubectl get pods -l app=application-a -o name) -- cat /mnt/secret/secret-a

   You should see this expected output:

   NAME READY STATUS RESTARTS AGE
   application-a-b98d44bb8-bzvn4 1/1 Running 0 3s
   
   {"user":"user1","password":"passwd1"}%

Working with cross-account secrets through resource policies

For this second use case, you’ll also use the Amazon EKS cluster on Account A, and create a new Secrets Manager secret in a different account (Account B).

In order to access a secret in a different account, you can’t use the default AWS Key Management Service (AWS KMS) key, but will use aws/secretsmanager to encrypt this secret. So you need to first create a new AWS KMS key that allows cross-account access.

On Account B

1. Create a customer managed key on AWS KMS with cross-account permissions:

   cat < key-policy.json
   {
     "Version": "2012-10-17",
     "Id": "cross-account-access",
     "Statement": [
       {
         "Sid": "Enable IAM User Permissions",
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::${ACCOUNT_B}:root"
         },
         "Action": "kms:*",
         "Resource": "arn:aws:kms:${AWS_REGION}:${ACCOUNT_B}:key/*"
       },
       {
         "Sid": "Allow access for Key Administrators",
         "Effect": "Allow",
         "Principal": {
           "AWS": "arn:aws:iam::${ACCOUNT_B}:role/Admin"
         },
         "Action": [
           "kms:Create",
           "kms:Describe",
           "kms:Enable",
           "kms:List",
           "kms:Put",
           "kms:Update",
           "kms:Revoke",
           "kms:Disable",
           "kms:Get",
           "kms:Delete",
           "kms:TagResource",
           "kms:UntagResource",
           "kms:ScheduleKeyDeletion",
           "kms:CancelKeyDeletion",
           "kms:RotateKeyOnDemand"
         ],
         "Resource": "arn:aws:kms:${AWS_REGION}:${ACCOUNT_B}:key/*"
       },
       {
         "Sid": "Allow access through AWS Secrets Manager",
         "Effect": "Allow",
         "Principal": {
           "AWS": "*"
         },
         "Action": [
           "kms:Encrypt",
           "kms:Decrypt",
           "kms:ReEncrypt",
           "kms:CreateGrant",
           "kms:DescribeKey"
         ],
         "Resource": "arn:aws:kms:${AWS_REGION}:${ACCOUNT_B}:key/*",
         "Condition": {
           "StringEquals": {
             "kms:ViaService": "secretsmanager.${AWS_REGION}.amazonaws.com",
             "kms:CallerAccount": [
               "$ACCOUNT_A",
               "$ACCOUNT_B"
             ]
           }
         }
       },
       {
         "Sid": "Allow access through AWS Secrets Manager for Generate Data Key",
         "Effect": "Allow",
         "Principal": {
           "AWS": "*"
         },
         "Action": "kms:GenerateDataKey",
         "Resource": "arn:aws:kms:${AWS_REGION}:${ACCOUNT_B}:key/*",
         "Condition": {
           "StringEquals": {
             "kms:CallerAccount": [
               "$ACCOUNT_A",
               "$ACCOUNT_B"
             ]
           },
           "StringLike": {
             "kms:ViaService": "secretsmanager.${AWS_REGION}.amazonaws.com"
           }
         }
       },
       {
         "Sid": "Allow direct access to key metadata to the account",
         "Effect": "Allow",
         "Principal": {
           "AWS": [
             "arn:aws:iam::${ACCOUNT_A}:root",
             "arn:aws:iam::${ACCOUNT_B}:root"
           ]
         },
         "Action": [
           "kms:Describe",
           "kms:Get",
           "kms:List",
           "kms:RevokeGrant"
         ],
         "Resource": "arn:aws:kms:${AWS_REGION}:${ACCOUNT_B}:key/*"
       }
     ]
   }
   EOF

   Then create the KMS key:

   $ aws kms create key --region $AWS_REGION --policy file://key-policy.json

   You should see this expected output:

   {
       "KeyMetadata": {
           "AWSAccountId": "$ACCOUNT_B",
           "KeyId": "abcdefgh-1234-5678-90ij-klmnopqrstuv",
           "Arn": "arn:aws:kms:${AWS_REGION}:${ACCOUNT_B}:key/abcdefgh-1234-5678-90ij-klmnopqrstuv",
           "CreationDate": "2025-01-23T18:00:00.756000-05:00",
           "Enabled": true,
           "Description": "",
           "KeyUsage": "ENCRYPT_DECRYPT",
           "KeyState": "Enabled",
           "Origin": "AWS_KMS",
           "KeyManager": "CUSTOMER",
           "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
           "KeySpec": "SYMMETRIC_DEFAULT",
           "EncryptionAlgorithms": [
               "SYMMETRIC_DEFAULT"
           ],
           "MultiRegion": false
       }
   }

   Take note of the KeyID and the ARN of the key. You can use it to create an alias for the newly created key:

   KEY_ID=<NOTED_KEY_ID_FROM_PREVIOUS_STEP>
   aws kms create-alias --region $AWS_REGION —-alias-name alias/secretsmanager-cross —-target-key-id $KEY_ID

2. Create a new secret in Secrets Manager, using the new KMS key to encrypt it:

   aws secretsmanager create-secret --region $AWS_REGION \
   --name $SECRET_NAME_CROSS \
   --tags Key=kubernetes-namespace,Value=$NAMESPACE Key=eks-cluster-name,Value=$CLUSTER_NAME \
   --secret-string "This is a Cross Account Secret" \
   --kms-key-id $KEY_ID

   You should see this expected output:

   {
   "ARN": "arn:aws:secretsmanager:$AWS_REGION:$ACCOUNT_B:secret:secret-b-12345",
   "Name": "secret-b",
   "VersionId": "abcdefgh-1234-5678-ijklmnopqrst"
   }

   Note the ARN of the new secret. You will use it in the next step.

3. Add a resource policy that allows the IAM role on Account A to access the newly created secret:

   SECRET_CROSS_ARN=< NOTED_SECRET_CROSS_ARN_FROM_PREVIOUS_STEP>
   cat << EOF > resource-policy.json
   
   {
     "Version" : "2012-10-17",
     "Statement" : [ {
       "Effect" : "Allow",
       "Principal" : {
         "AWS" : "arn:aws:iam::${ACCOUNT_A}:root"
       },
       "Action" : [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ],
       "Resource" : "$SECRET_CROSS_ARN"
       } 
     ]
   }
   
   EOF
   aws secretsmanager put-resource-policy --secret-id $SECRET_NAME_CROSS --resource-policy file://resource-policy.json

Now let’s go back to Account A.

On Account A

4. Create a new IAM role and trust policy with the required permissions to retrieve the newly created secret within Account B:

   KEY_ARN=< NOTED_KEY_ARN_FROM_PREVIOUS_STEP>
   cat << EOF > cross-policy.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
               "Resource": ["$SECRET_CROSS_ARN"],
               "Condition": {
                   "StringEquals": {
                       "secretsmanager:ResourceTag/kubernetes-namespace": "\${aws:PrincipalTag/kubernetes-namespace}",
                       "secretsmanager:ResourceTag/eks-cluster-name": "\${aws:PrincipalTag/eks-cluster-name}"
                   }
               }
           },
           {
               "Effect": "Allow",
               "Action": [
                   "kms:Decrypt",
                   "kms:DescribeKey"
                   ],
               "Resource": "$KEY_ARN"
           }
       ]
   }
   EOF

   In the preceding example IAM role permission policy, note that the resource ARN is pointing to the secret created on Account B.

   cat << EOF > trust.json
   {
       "Version": "2012-10-17",
       "Statement": [
           {  
               "Effect": "Allow",
               "Principal": {
                   "Service": "pods.eks.amazonaws.com" 
               },
               "Action": [
                   "sts:AssumeRole",
                   "sts:TagSession"
               ]
           }
       ]
   }
   EOF

   The IAM role trust policy needs to have pods.eks.amazonaws.com as a trusted entity to perform the sts:AssumeRole and sts:TagSession actions.

   Next, create the role and apply the policy:

   aws iam create-role --role-name ascp-podidentity-cross --assume-role-policy-document file://trust.json
   aws iam put-role-policy --role-name ascp-podidentity-cross --policy-name ascp-podidentity-cross --policy-document file://cross-policy.json

   Note the ARN of the new role. You will use it in the next step.

5. Create the second Kubernetes ServiceAccount and add the cross-account Pod Identity association:

   kubectl create serviceaccount serviceaccount-b
   
   ROLE_CROSS_ARN=<NOTED_ROLE_CROSS_ARN_FROM_PREVIOUS_STEP>
   aws eks create-pod-identity-association \
   --cluster-name "$CLUSTER_NAME" \
   --namespace "default" \
   --service-account serviceaccount-b \
   --role-arn $ROLE_CROSS_ARN

6. Create a SecretProviderClass to use the cross-account secret created in Account B in your Amazon EKS cluster:

   cat << EOF | kubectl apply -f -
   
   apiVersion: secrets-store.csi.x-k8s.io/v1
   kind: SecretProviderClass
   metadata:
     name: aws-secrets-manager-cross
   spec:
     provider: aws
     parameters:
       objects: |
         - objectName: "$SECRET_CROSS_ARN" # Full ARN of the Secret in the ACCOUNT B to be mounted on Pod
           objectType: "secretsmanager"
       usePodIdentity: "true" # Indicator to use Pod Identity instead of IRSA
       
   EOF

7. Create a new deployment to consume your newly created secret using Pod Identity as a mounted volume:

   cat << EOF | kubectl apply -f -
   
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: application-b
     name: application-b
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: application-b
     template:
       metadata:
         labels:
           app: application-b
       spec:
         containers:
         - image: public.ecr.aws/amazonlinux/amazonlinux:2023-minimal
           name: amazonlinux
           args:
           - infinity
           command:
           - sleep
    volumeMounts:
    - name: secrets-store-cross
    mountPath: "/mnt/secret" # Directory where secret will be mounted to readOnly: true
    volumes:
    - name: secrets-store-cross
    csi:
    driver: secrets-store.csi.k8s.io
    readOnly: true
    volumeAttributes:
    secretProviderClass: "aws-secrets-manager-cross" # Match SecretProviderClass name created for cross-account access serviceAccountName: serviceaccount-b # Specify service account created for cross-account access
   
   EOF

8. Validate that the deployment was created successfully and confirm the mounted secret:

   kubectl get pods -l app=application-b
   kubectl exec -it $(kubectl get pods -l app=application-b -o name) -- cat /mnt/secret/$SECRET_CROSS_ARN

   Note that the volume name is the secret ARN, because it’s a cross-account secret.

   You should see this expected output:

   NAME READY STATUS RESTARTS AGE
   application-b-67b755444f-ngrhv 1/1 Running 0 8s
   
   "This is a Cross Account Secret"%

Conclusion

The integration of ASCP with Pod Identity marks a significant step forward in secrets management for Amazon EKS. It offers enhanced security, simplified configuration, and improved operations. We encourage all EKS users to explore this new integration and take advantage of its benefits.

The integration of ASCP with Pod Identity offers these benefits over IRSA:

1. Simplified setup: With Pod Identity, you don’t need to create and manage service accounts for each application.
2. Enhanced security: Pod Identity provides more granular control over permissions at the Pod level.
3. Improved scalability: Pod Identity is easier to implement in large-scale environments.
4. Consistent AWS experience: Pod Identity aligns more closely with AWS best practices for IAM management.

For more information, see our documentation for: AWS Secrets Manager, AWS Secrets CSI Store Provider (ASCP), and Amazon EKS Pod Identity.

 
If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.