Tag Archives: AWS Identity and Access Management (IAM)

PACIFIC enables multi-tenant, sovereign product carbon footprint exchange on the Catena-X data space using AWS

Post Syndicated from Kevin S. Ridolfi original https://aws.amazon.com/blogs/architecture/pacific-enables-multi-tenant-sovereign-product-carbon-footprint-exchange-on-the-catena-x-data-space-using-aws/

This post is cowritten by Anil Akarsu and Dr. Renè Holschuh from BASF.

BASF is a global chemical industry leader and active member of the Catena-X Automotive Network. It pioneers sustainable solutions that enable automotive organizations to track carbon emissions across complex supply chains. CircularTree transforms sustainability reporting through innovative digital solutions that systematically identify and control Scope 3 greenhouse gas (GHG) emissions across global supply networks. They establish standardized data exchange protocols through participation in forward-thinking associations including PACT, Catena-X, and ESTAINIUM. BASF and CircularTree created PACIFIC, a product powered by AWS that streamlines transparent product carbon footprint (PCF) reporting across the value chain by automating PCF data exchange, reducing manual effort, and ensuring trustworthy data sharing. Through this unique relationship, AWS helps customers integrate software, services, and processes to accelerate business transformation. This post explores how PACIFIC enables multi-tenant, sovereign PCF exchange on the Catena-X data space using Amazon Elastic Container Service (Amazon ECS) on AWS Fargate, Amazon Cognito, and AWS Identity and Access Management (IAM) to deliver measurable environmental impact and competitive advantage in a carbon-conscious marketplace.

Carbon data at scale, across company borders

Sustainability is now an operational requirement, driven by growing regulatory pressure in the European Union and increasing customer expectations for credible and auditable emissions data. For manufacturers in the automotive supply chain, this is especially challenging because emissions data does not live in one place. It is distributed across fragmented tiers of suppliers, different internal systems, and partner-to-partner handoffs that still happen through spreadsheets, emails, and one-off integrations.

At the same time, the industry is converging on shared ways to exchange data, with Catena-X setting expectations for interoperability and trust in cross-company collaboration. PACIFIC was built for this reality as a multi-tenant SaaS product that enables companies to manage and exchange PCFs while maintaining data sovereignty. Its Catena-X certification signals alignment with industry standards, and the partnership with BASF grounds the platform in real supply chain requirements.

Data security, sovereignty, and interoperability

To make PCF exchange work in the real world, PACIFIC needed to solve two problems at the same time: enable frictionless collaboration across companies, while guaranteeing that each company stays in full control of their data and credentials. The platform had to operate as a multi-tenant software as a service (SaaS) for the supply chain, serving organizations on shared infrastructure without introducing any possibility of cross-tenant access.

They had to build an interoperable solution that could communicate with other solution providers on the Catena-X data space, using Eclipse Dataspace Components (EDC) connectors as a standard mechanism for cross-company data exchange. That meant enforcing strict data sovereignty, not only for PCF records but also for sensitive Catena-X integration configuration such as EDC and Digital Twin Registry (DTR) credentials. At the exchange layer, PACIFIC needed end-to-end authorization aligned with Catena-X expectations, where PCF data is shared only after explicit agreement and policy negotiation through EDC. Finally, the solution had to be practical to run and scale, so the following had to happen:

  • Onboard new companies without spinning up separate AWS accounts per tenant
  • Integrate suppliers’ PCF systems like BASF without tight coupling to the exchange workflow
  • Keep the platform secure, auditable, and operable as usage grows

Solution overview

Figure 1 gives a high-level view of how PACIFIC is built and deployed to enable secure, multi-tenant PCF exchange on the Catena-X data space. It shows the main building blocks of the product, how user traffic reaches the application, how tenant-aware identity and authorization are enforced, and how PACIFIC separates core platform features from integrations and exchange endpoints. The diagram also highlights the external connections enabling interoperability, including supplier PCF data sources like BASF services, and EDC and DTR “enablement service providers” for Catena-X based data sharing.

AWS Cloud architecture diagram for the PACIFIC platform showing a multi-layered system. At the top, a PACIFIC Web Client connects to an Identity & Authorization layer containing Amazon Cognito, AWS IAM, and AWS Secrets Manager. Traffic flows through AWS WAF to an Application Load Balancer within a VPC, which distributes requests to Amazon ECS (AWS Fargate) hosting four containerized microservices: core-modules, integration-module, pcf-exchange-module, and edc-dtr-module. These modules connect to Amazon RDS for relational database storage and Amazon S3 for object storage. External integrations at the bottom include BASF Product Carbon Footprint Services, an EDC/DTR Service Provider, and the Catena-X Automotive Network. The diagram illustrates a secure, microservices-based architecture for automotive industry carbon footprint data exchange.

Figure 1: PACIFIC high-level service architecture

Data protection through IAM-based tenant isolation

A core requirement for PACIFIC is maintaining data protection and security. Each company must have exclusive control over their PCF data, EDC connector, and DTR management credentials, without any possibility of cross-tenant access. Rather than provisioning separate AWS accounts per tenant PACIFIC implements a fine-grained IAM-based isolation model built on Amazon Cognito and AWS Secrets Manager. When a company joins the platform, PACIFIC automatically provisions a dedicated IAM role with a scoped policy that permits access only to that company’s secrets in Secrets Manager. Users are assigned to an Amazon Cognito user pool group linked to their company’s IAM role. When a user authenticates, the Amazon Cognito identity pool maps their group membership to the corresponding IAM role, and AWS Security Token Service (AWS STS) issues temporary credentials for that role. This means, a user’s credentials can only retrieve their own company’s EDC secrets and access to other tenants’ configuration is denied at the IAM policy level. This architecture delivers true multi-tenant isolation using native AWS identity services, without the overhead of managing dedicated accounts or Amazon Virtual Private Cloud (Amazon VPC) per customer.

Securing PCF exchange with EDC-issued authorization tokens

Beyond tenant isolation within PACIFIC, CircularTree enforces authorization at the data exchange layer through the pcf-exchange-module, a per-tenant endpoint that serves PCF data to authorized trading partners. When a consumer’s EDC connector requests PCF data from a supplier’s EDC, the two connectors negotiate and agree on usage policies governing how the exchanged information can be used. After this agreement is established, the supplier’s EDC issues a special authorization token to the consumer’s EDC. The token derives from the supplier company’s Cognito app client credentials stored within their EDC and grants access specifically to that supplier’s pcf-exchange-module endpoint in PACIFIC. The consumer’s EDC then uses this token to call the supplier’s dedicated endpoint and retrieve the authorized PCF data. Because each tenant’s pcf-exchange-module is published as an individual endpoint, which only accepts tokens issued through the EDC handshake process, unauthorized access is prevented at multiple levels. One level is through EDC policy negotiation, and the other is through company-scoped OAuth2 token validation. This ensures that PCF data is transmitted only after explicit consent and only to the specific trading partner authorized in the data exchange agreement. Figure 2 gives an overview of the communication flow.

Data flow diagram showing a six-step secure token exchange process between Consumer EDC and Supplier EDC systems via the PACIFIC platform on AWS. The flow proceeds as follows: (1) Consumer EDC sends a Request PCF to Supplier EDC, (2) Policy Negotiation occurs between the two EDCs (shown as a dashed line), (3) a token is issued (marked with a key icon), (4) Supplier EDC provides a Data Space URL to the pcf-exchange-module, (5) Token Validation occurs between the pcf-exchange-module and the supplier-oauth2-client component (marked with a key icon), and (6) PCF Data flows back to the Consumer EDC. The PACIFIC layer at the bottom contains two AWS-hosted components: the pcf-exchange-module (orange icon) and the supplier-oauth2-client (red icon with checkmark), demonstrating OAuth2-based secure authentication for Product Carbon Footprint data exchange.

Figure 2: EDC-to-EDC communication with Cognito Oauth2 Tokens

Integrating supplier PCF systems through the integration module

While the pcf-exchange-module handles secure data exchange between trading partners using the Catena-X data space, PACIFIC also needs to ingest PCF data from suppliers’ internal systems. Running on AWS Fargate, the integration-module provides a flexible, scalable integration layer that connects to proprietary supplier PCF systems, such as BASF’s internal PCF services. Each supplier integration requires handling unique authentication flows. This ranges from OAuth2 client credentials to certificate-based authentication or API key mechanisms, all of which are securely managed through AWS Secrets Manager. The integration-module expects incoming PCF data to already conform to the standardized Catena-X PCF JSON format, ensuring consistency at the point of ingestion. After received, PCF data is stored in Amazon Simple Storage Service (Amazon S3) under company-specific prefixes. This is where IAM policies make sure that only the PCF owner company can access their respective data. By decoupling supplier system integration from the data exchange layer, PACIFIC can onboard new supplier PCF data sources without impacting the downstream Catena-X data sharing workflows. This can be done while the S3-based storage model helps maintain strict data sovereignty, and each company’s PCF data remains isolated and accessible only to its rightful owner.

Conclusion

PACIFIC turns Catena-X PCF exchange from a specification into an interoperable, scalable workflow running on Amazon ECS and AWS Fargate without requiring companies to give up control of their data and credentials. The impact is measurable in both speed and operational scalability. From a business perspective, BASF highlights the most tangible improvement: when a requested PCF dataset is already available, a manual exchange can take up to around seven days, whereas PACIFIC responds in seconds, and can deliver automated updates when PCFs change. This capability delivers up to 75% time savings for both customers and BASF. It also reduces the time-to-data from days to seconds and making emissions information more visible and usable in day-to-day supply chain operations. PACIFIC’s multi-tenant architecture scales onboarding and operations without managing individual AWS accounts per company, while still enforcing strong tenant isolation through IAM-scoped access control and per-tenant exchange endpoints. This scalability translates into faster onboarding, enabling BASF to integrate significantly more partners into the data space. The results are measurable: an 80% increase in newly onboarded companies between 2024 and 2025, and 55% growth in requested products and shared PCFs over the same period. This provides a scalable baseline for expanding the number of onboarded organizations and increasing the volume of PCF data exchanged as adoption grows—without weakening data sovereignty or interoperability. These results underline PACIFIC’s role as a catalyst for accelerating decarbonization across supply chains.

We encourage you to join BASF, CircularTree, and AWS in industry data sharing through emerging data spaces and transparent, trusted PCF exchange across global supply networks. To explore more sustainability solutions and AWS architecture patterns, visit the AWS Architecture Blog and get started with PACIFIC through the Cofinity-X App Marketplace.


About the authors

Securely connecting on-premises data systems to Amazon Redshift with IAM Roles Anywhere

Post Syndicated from Zainab Syeda original https://aws.amazon.com/blogs/big-data/securely-connecting-on-premises-data-systems-to-amazon-redshift-with-iam-roles-anywhere/

Securely connecting on-premises data systems to Amazon Redshift requires removing static credentials while preserving seamless access for your data teams. This solution extends connectivity from your on-premises data centers to Amazon Redshift by using short-lived, auditable credentials. All traffic remains within trusted, private channels.

Developers and data engineers need a process to run ingestion pipelines, Extract, Transform, Load (ETL) jobs, and analytics queries without managing static credentials or complex authentication flows. You can use AWS Identity and Access Management (IAM) Roles Anywhere to obtain temporary security credentials in IAM. This service extends the short-term credential model of AWS beyond the cloud and allows on-premises workloads to authenticate with IAM using X.509 certificates from an existing certificate authority. This approach removes static IAM access keys and applies least-privilege access through IAM policies. Every request is recorded in AWS CloudTrail. Paired with private Domain Name System (DNS) and Amazon Virtual Private Cloud (Amazon VPC) endpoints for Amazon Redshift, it keeps authentication and data flows inside private networks without traversing the public internet.

In this post, you will learn how to use AWS IAM Roles Anywhere with Amazon Redshift for secure, private connections. This removes the need to expose traffic to the public internet or manage long-lived access keys.

The challenge

Organizations connecting on-premises data systems to Amazon Redshift typically choose from several established security patterns, each with tradeoffs in risk, complexity, and operational overhead. Static IAM access keys are straightforward to adopt but require ongoing rotation, secure distribution, and storage across systems. Their long-lived nature increases the impact of accidental exposure in code, configuration files, or logs. Shared database or service credentials can streamline setup but often reduce auditability, weaken least-privilege controls, and create accountability challenges across teams. VPN or private network connections improve network isolation, yet they still require strong application-layer authentication and add infrastructure management burdens. Custom secret-management or credential-brokering solutions can reduce reliance on long-lived credentials, but they introduce additional components that must be built, integrated, and maintained. As organizations scale, these patterns often force tradeoffs between strong security controls and the developer productivity needed to build and operate data pipelines efficiently.

Solution overview

The solution integrates on-premises workloads with Amazon Redshift using IAM Roles Anywhere and the built-in IAM authentication of Amazon Redshift. The core idea is that on-premises workloads use X.509 certificates to obtain short-term IAM credentials, then exchange them for temporary Amazon Redshift database credentials. Both provisioned clusters and serverless workgroups are supported. The architecture consists of these main components:

  • Amazon Redshift Service Endpoint – Handles secure API calls such as GetClusterCredentials, GetCredentials, and GetClusterCredentialsWithIAM. The on-premises workload uses these API endpoints to request temporary database credentials.
  • Amazon Redshift Cluster Endpoint – Provides the connection point for database operations on provisioned Amazon Redshift clusters. After obtaining temporary credentials, applications and tools like JDBC/ODBC drivers or psql connect to the cluster endpoint. They use this connection to execute SQL queries, load data, and perform analytics tasks.
  • Amazon Redshift Serverless Workgroup Endpoint – Serves the same function as the cluster endpoint but for serverless deployments. After temporary credentials are retrieved through the GetCredentials API, applications connect to this endpoint using standard database drivers (JDBC/ODBC) or command line tools like psql to run queries and load data.
  • Certificate authority – For this post, we use AWS Private Certificate Authority (AWS Private CA) as the certificate authority (CA) source. Alternatively, you can integrate with an external CA. For more details, see IAM Roles Anywhere with an external certificate authority.
  • X.509 Certificate – We use a sample private certificate stored in AWS Certificate Manager (ACM) and issued by AWS Private CA.
  • IAM Roles Anywhere – Issues short-term AWS credentials to on-premises processes based on X.509 certificates from an organization’s certificate authority. These temporary credentials allow the workload to assume an IAM role that grants access to Amazon Redshift APIs.

To retrieve temporary credentials using IAM Role Anywhere, we use the credential_process parameter in AWS Command Line Interface (AWS CLI) profile configurations to trigger an external process that generates or retrieves credentials. This post uses X.509 certificates to authenticate and return temporary IAM credentials through IAM Roles Anywhere. The AWS IAM Roles Anywhere Credential Helper is executed to handle the signing process for the CreateSession API, returning credentials in a JSON format that applications and tools can consume.

Amazon Redshift provides several APIs that work together to support temporary, IAM-based authentication for different deployment scenarios. When connecting to a provisioned Amazon Redshift cluster, applications typically use the GetClusterCredentials API, which returns short-term database credentials tied to an IAM role’s permissions. For organizations with fully IAM-managed identities, GetClusterCredentialsWithIAM streamlines this process by automatically mapping the IAM identity to a database user, removing the need to specify usernames manually. In serverless deployments, the GetCredentials API performs the same function, issuing temporary credentials for Amazon Redshift Serverless workgroups based on IAM permissions. Collectively, these APIs keep static credentials from being stored or distributed while offering flexible integration paths for both provisioned and serverless Amazon Redshift architectures.

Flow overview

An on-premises ETL job begins by initiating a request and authenticates with AWS using IAM Roles Anywhere to assume an IAM role securely. After obtaining temporary security credentials, the workload calls the Amazon Redshift service endpoint to execute the GetClusterCredentials API, which returns short-term database credentials. These credentials allow the workload to connect to the Amazon Redshift cluster endpoint through a VPC endpoint. This enables running SQL queries or loading data into the cluster as part of the ETL process.

Prerequisites

You must have the following prerequisites to follow along with this post.

AWS account requirements

  • An AWS account with permissions to deploy AWS CloudFormation templates.
  • Access to AWS CloudShell for exporting a sample private certificate that we create using AWS CloudFormation in a later step.

Remote environment

Network Connectivity requirements

Deploy AWS resources with AWS CloudFormation

  1. Navigate to the AWS CloudFormation console.
  2. Choose Create Stack.
  3. Download the redshift-iamra-template template.
  4. For Specify template, choose Upload a template file and upload redshift-iamra-template.
  5. Choose Next.
  6. Enter a unique name for Stack name. The default value is redshift-test.
  7. Configure the stack parameters. The following table provides default values.
Parameter name Default value Description
VPCCIDR 10.0.0.0/16 CIDR block for the VPC
PrivateSubnet1CIDR 10.0.1.0/24 CIDR block for the first private subnet
PrivateSubnet2CIDR 10.0.2.0/24 CIDR block for the second private subnet
CACommonName redshift-ca.example.com Common Name for the Certificate
CAOrganization Example Corp Organization for the Certificate Authority
CACountry US Country for the Certificate Authority
CAValidityInDays 1826 Validity period in days for the CA Certificate (5 years)
RedshiftClusterIdentifier my-redshift-cluster Identifier for the Amazon Redshift cluster
RedshiftDatabaseName dev Name of the initial database in the Amazon Redshift cluster
RedshiftMasterUsername admin Main username for the Amazon Redshift cluster
RedshiftNodeType ra3.xlplus Node type for the Amazon Redshift cluster
ServerlessNamespace my-serverless-namespace Namespace identifier for Amazon Redshift Serverless
ServerlessWorkgroup my-serverless-workgroup Workgroup identifier for Amazon Redshift Serverless
  1. Select the acknowledgement checkbox and choose Create Stack. Stack deployment takes about 10 minutes to complete.
  1. When stack creation is complete, navigate to the Outputs tab on the AWS CloudFormation console and note down the values for the resources that the stack created.

The following table shows a summarized view of the output values.

Output Description Example value
CertificateAuthorityArn Amazon Resource Name (ARN) of the Private Certificate Authority arn:aws:acm-pca:aa-example-1:111122223333:certificate-authority/a1b2c3d4-5678-90ab-cdef-EXAMPLE22222
ClientCertificateArn ARN of the sample client certificate arn:aws:acm:aa-example-1:111122223333:certificate/a1b2c3d4-5678-90ab-cdef-EXAMPLE11111
ProfileArn ARN of the IAM Roles Anywhere profile arn:aws:rolesanywhere:aa-example-1:111122223333:profile/a1b2c3d4-5678-90ab-cdef-EXAMPLE44444
RedshiftAccessRoleArn ARN of the Amazon Redshift Access role arn:aws:iam::1222345677:role/Redshift-test-RedshiftAccessRole
TrustAnchorArn ARN of the IAM Roles Anywhere profile. You will use this value for configuring credential_process for IAM Roles Anywhere in a later step. arn:aws:rolesanywhere:aa-example-1:111122223333:trust-anchor/a1b2c3d4-5678-90ab-cdef-EXAMPLE33333
RedshiftClusterEndpoint Private endpoint of the Amazon Redshift Cluster my-redshift-cluster-123456789012.aa-example-1.redshift.amazonaws.com
RedshiftClusterPort Port of the Amazon Redshift Cluster 5439
ServerlessWorkgroupEndpoint Private endpoint of Amazon Redshift Serverless Workgroup my-serverless-workgroup-123456789012.aa-example-1.redshift.serverless.amazonaws.com

Export a sample private certificate using CloudShell

To export a sample private certificate using CloudShell, complete the following steps.

  1. Open CloudShell. For more details, see Navigating the AWS CloudShell interface.
  2. Export the certificate ARN from the CloudFormation outputs. If you changed the stack name in the previous step, use that value for <stack-name>. Otherwise, use the default value redshift-public-iam-roles-anywhere.
export CERT_ARN=$(aws cloudformation describe-stacks \
    --stack-name <stack-name> \
    --query 'Stacks[0].Outputs[?OutputKey==`ClientCertificateArn`].OutputValue' \
    --output text)
  1. Extract the certificate and private key files:
# Generate and save the passphrase
export PASSPHRASE=$(openssl rand -base64 32)
# Export certificate using environment variables
aws acm export-certificate \
    --certificate-arn $CERT_ARN \
    --passphrase $(echo -n "$PASSPHRASE" | base64) \
    > cert_export.json
# Extract components to separate files
jq -r '.Certificate' cert_export.json > certificate.pem
jq -r '.PrivateKey' cert_export.json > encrypted_private_key.pem
# Decrypt the private key
openssl rsa -in encrypted_private_key.pem -out private_key.pem -passin pass:"$PASSPHRASE"
# Clear environment variables
unset PASSPHRASE CERT_ARN
  1. Download the extracted certificate and private key files from CloudShell:
/home/cloudshell-user/certificate.pem
/home/cloudshell-user/private_key.pem
  1. Secure the private key on your local workstation.

After downloading the files, restrict file permissions to prevent unauthorized access:

chmod 400 private_key.pem chmod 400 certificate.pem

For production workloads, consider storing private keys in your operating system’s keychain (macOS Keychain, Windows Certificate Store), a hardware security module (HSM), or a secrets management tool rather than as files on disk.

Configure an AWS CLI profile

These are the steps to configure an AWS CLI profile on your system:

  1. Store the downloaded certificate and private key to your environment. For an automated approach to generate and rotate certificates, see Set up AWS Private Certificate Authority to issue certificates for use with IAM Roles Anywhere.
  2. Create a new profile named onprem-redshift. This invokes the credential process. Replace the placeholders with your specific values. Find the values for trusted-anchor-arn, profile-arn, and role-arn in your CloudFormation stack outputs.
aws configure set profile.onprem-redshift.credential_process "</path/to/aws_signing_helper> credential-process \
      --certificate </path/to/certificate.pem> \
      --private-key </path/to/private_key.pem> \
      --trust-anchor-arn <trusted-anchor-arn> \
      --profile-arn <profile-arn> \
      --role-arn < role-arn>"
  1. Verify your configuration. Open the ~/.aws/config file and confirm that it contains a profile.
[profile onprem-redshift]
credential_process = </path/to/aws_signing_helper> credential-process       
--certificate </path/to/certificate.pem>       
--private-key </path/to/private_key.pem>       
--trust-anchor-arn <trusted-anchor-arn>       
--profile-arn <profile-arn>       
--role-arn <role-arn>

Test the solution

Follow these steps to validate your setup for provisioned clusters to confirm end-to-end connectivity:

  1. Verify network connectivity

Before testing authentication, confirm that your on-premises environment can reach the Amazon Redshift cluster endpoint:

telnet my-redshift-cluster.abc123.us-east-1.redshift.amazonaws.com 5439

If the connection succeeds, you should see a response indicating the port is open. If it fails, verify your VPN/Direct Connect configuration and security group rules.

  1. Create database user

If you haven’t already created a user, connect to your Amazon Redshift as the main user and create a dedicated user for testing:

CREATE USER analytics_user PASSWORD '[PASSWORD]';

  1. Retrieve Amazon Redshift database credentials

With the configuration in place, request temporary database credentials from Amazon Redshift:

aws redshift get-cluster-credentials \
  --db-user analytics_user \
  --cluster-identifier my-redshift-cluster \
  --region us-east-1 \
  --profile onprem-redshift

This call returns a short-lived username and password that’s valid for connecting to the cluster. By default, the temporary credentials expire in 900 seconds. You can optionally specify a duration between 900–3600 seconds (15–60 minutes).

  1. Connect using JDBC/ODBC or psql

Use the issued credentials in your connection string. For JDBC:

jdbc:redshift://my-redshift-cluster.abc123.redshift.amazonaws.com:5439/dev?ssl=true&UID=analytics_user&PWD=<temporary_password>

For psql:

PGPASSWORD=<temporary_password> psql \
  -h my-redshift-cluster.abc123.redshift.amazonaws.com \
  -p 5439 \
  -U analytics_user \
  -d dev \
  --set=sslmode=verify-full

Validate and monitor

  • Test authentication flows end-to-end using your ETL jobs.
  • Review AWS CloudTrail logs to validate. It records role assumptions and Amazon Redshift API calls.
  • Monitor session expiration to help workloads handle credential refresh seamlessly.

Testing end-to-end connectivity for Amazon Redshift Serverless

The testing process for Amazon Redshift Serverless follows a similar pattern to provisioned clusters, with minor differences in the API calls and connection parameters. These steps validate connectivity to your serverless workgroup.

  1. Verify network connectivity

telnet my-serverless-workgroup.abc123.us-east-1.redshift.amazonaws.com 5439

  1. Retrieve Amazon Redshift Serverless database credentials
aws redshift-serverless get-credentials \
  --workgroup-name my-serverless-workgroup \
  --db-name dev \
  --region us-east-1 \
  --profile onprem-redshift
  1. Connect using JDBC/ODBC or psql
PGPASSWORD="<password_from_get_credentials>" psql \
  -h my-serverless-workgroup.abc12.us-east-1.redshift-serverless.amazonaws.com \
  -p 5439 \
  -U "IAMR:Redshift-IAMRA-RedshiftAccessRole" \
  -d dev \
  --set=sslmode=verify-full

Clean up

To avoid future charges, remove the deployed resources:

  1. Delete the CloudFormation stack.
  2. Remove the generated files from CloudShell:

rm cert_export.json encrypted_private_key.pem certificate.pem private_key.pem

Conclusion

In this post, we showed how to implement IAM Roles Anywhere with Amazon Redshift so that enterprises can securely connect on-premises data systems to their cloud data warehouse without relying on static credentials or public internet access. This architecture provides short-lived, auditable credentials, integrates with existing certificate authorities, and helps ensure authentication and data flows remain private and trusted.

With this approach, data engineers and developers can run ingestion pipelines, ETL jobs, and analytics queries, while security teams maintain full control through IAM governance and CloudTrail auditing. You can remove manual credential rotation tasks, allow your data engineers to connect to Amazon Redshift without managing static keys, and achieve complete audit trails through CloudTrail integration for your hybrid analytics environments.

To get started, deploy the solution using the CloudFormation template and follow the steps in this post. To learn more about the services used, see the following resources:


About the authors

Kanwar Bajwa is a Principal Enterprise Account Engineer at AWS who works with customers to optimize their use of AWS services and achieve their business objectives.

Xiaoxue Xu is a Solutions Architect for AWS based in Toronto. She primarily works with Financial Services customers to help secure their workload and design scalable solutions on the AWS Cloud.

Zainab Syeda is a Technical Account Manager at Amazon Web Services in Toronto. She works with customers in the Financial Services segment, helping them leverage cloud-native solutions at scale.

Secure AI agent access patterns to AWS resources using Model Context Protocol

Post Syndicated from Riggs Goodman III original https://aws.amazon.com/blogs/security/secure-ai-agent-access-patterns-to-aws-resources-using-model-context-protocol/

AI agents and coding assistants interact with AWS resources through the Model Context Protocol (MCP). Unlike traditional applications with deterministic code paths, agents reason dynamically, choosing different tools or accessing different data depending on context. You must assume an agent can do anything within its granted entitlements, whether OAuth scopes, API keys, or AWS Identity and Access Management (IAM) permissions, and design your controls accordingly. Agents operate at machine speed, so the impact of misconfigured permissions scales quickly.

This blog post focuses on IAM as the authorization layer for AWS resource access and presents three security principles for building deterministic IAM controls for these non-deterministic AI systems. The principles apply whether you’re using AI coding assistants like Kiro and Claude Code, or deploying agents on hosting environments like Amazon Bedrock AgentCore. We cover deployment patterns, then explore each principle with concrete IAM policy examples and implementation guidance.

This post specifically addresses securing the MCP access path, where agents interact with AWS resources through MCP servers. AI coding assistants and agents can also access AWS service APIs directly through general-purpose tools like bash or shell execution, bypassing MCP servers entirely. For this reason, we recommend architecting agents to use MCP servers rather than direct service access where possible. MCP servers provide a layer of abstraction that enables the differentiation controls in principle 3 and creates additional monitoring capabilities through AWS CloudTrail. When agents bypass MCP, the differentiation mechanisms in principle 3 don’t apply, and principles 1 and 2 become your primary controls. We discuss this scope boundary in principle 3.

MCP deployment patterns

Your deployment pattern determines which security principles and implementation approaches apply. Three dimensions define this pattern, including where the agent runs, what type of MCP server offers the tools, and your level of control over the agent code. No matter how you connect to it, the MCP server needs AWS credentials to interact with AWS resources.

Where agents run

Agents access AWS resources from three locations: developer machines (where you control the infrastructure), hosting environments (where you control the infrastructure or significant aspects of it), and third-party agent platforms (where you do not control the infrastructure). This post focuses on the first two patterns. Each has a different credential model and different organizational control options.

AI coding assistants and local agents

AI coding assistants (Kiro, Claude Code) or local agent applications represent the first deployment pattern. These assistants run locally on developer machines and connect to MCP servers or use AWS Command Line Interface (AWS CLI) commands to access AWS resources. In this pattern, credentials come from the developer’s local environment. When a developer configures an MCP server in their mcp.json file, they specify which AWS credentials to use. Options include a named profile, which can use credential helpers and the credential provider chain for short-lived credentials, environment variables, or explicit credential configuration. This means the developer controls which IAM principal the agent uses to access AWS. This creates a governance challenge. Without additional controls, developers often use their developer admin credentials, shared development roles, or even production roles for agent access. Developer credentials often carry broad permissions designed for interactive use, where human judgment serves as a safeguard. When an agent inherits these permissions, it operates without that judgment at machine speed. Principle 1 explores this risk in detail.

Agents on hosting environments

Agents deployed on hosting environments represent the second deployment pattern. These agents run on infrastructure you manage, not on developer machines. This changes the credential management model. Using Amazon Bedrock AgentCore as an example, when an agent runs on AgentCore Runtime, it uses an execution IAM role that you configure when creating the runtime. The execution role’s permissions apply to all operations the agent performs and cannot be scoped down per-invocation at the runtime configuration level. For more granular control, agents can call AWS Security Token Service (AWS STS) AssumeRole or AssumeRoleWithWebIdentity (collectively referred to as AssumeRole in this post). This obtains temporary credentials with session policies that further restrict permissions beyond the role’s base permissions. Agents built with frameworks like Strands can also initialize individual MCP clients with different credential sets by calling AssumeRole and passing the resulting credentials to each client connection. This enables per-tool credential isolation within a single agent process. The same pattern applies to agents deployed on Amazon Elastic Compute Cloud (Amazon EC2) or Amazon Elastic Kubernetes Service (Amazon EKS).

With this centralized execution model, you can implement organizational controls. You define the available IAM roles through infrastructure configuration instead of relying on developer choice. However, you must design these roles carefully to prevent overly permissive access and implement session policies for tool-specific restrictions.

What type of MCP server

MCP servers come in two types, provider-managed and self-managed. AWS-managed servers are operated by AWS on your behalf. Self-managed servers are servers that you install and run yourself. The server type affects your operational overhead, available features, and how you implement security controls.

AWS offers fully managed MCP servers, including the AWS MCP Server, Amazon EKS MCP Server, and Amazon ECS MCP Server. These AWS-managed servers run on AWS infrastructure and require no installation or maintenance on your part. AWS-managed MCP servers automatically add IAM context keys (aws:ViaAWSMCPService and aws:CalledViaAWSMCP) to every downstream AWS service call. You can write IAM policies that check these keys to distinguish between AI-driven actions and human-initiated actions without any additional configuration.

Self-managed MCP servers include AWS-provided servers from the AWS MCP GitHub repository that you install and run yourself. They also include custom MCP servers that you build from scratch. With self-managed servers, you control the deployment location (local machine, Amazon EC2, Amazon EKS), the configuration, and the maintenance. These servers can be used with either AI coding assistants running locally or agents deployed on hosting environments. The key difference for security controls is that self-managed servers don’t automatically add IAM context keys for differentiation. You must configure the MCP server to add session tags when assuming IAM roles if you require differentiation between AI-driven and human-initiated actions. This requires modifying your MCP server code to call AWS STS AssumeRole with tags attached. You then write IAM policies that check for these tags using the aws:PrincipalTag condition key. Self-managed servers can also be extended to implement dynamic authorization flows, such as mapping inbound OAuth tokens to outbound IAM role assumptions, giving you control over the full authorization chain. Additionally, with AWS-managed MCP servers, AWS injects context keys at the service layer, so callers cannot spoof them. With self-managed servers, the entity calling AssumeRole sets the session tags, so you must trust that your MCP server code hasn’t been modified.

The responsibility model differs between server types. With AWS-managed MCP servers, AWS is responsible for server infrastructure, patching, and context key injection. You’re responsible for IAM policy design and credential configuration. With self-managed MCP servers, you’re additionally responsible for server patching, dependency and library supply chain security, session tag implementation, and verifying server integrity. This connects to the supply chain risk described in principle 1. While self-managed servers require more operational overhead to implement and maintain, they give you flexibility and control.

Level of client control

A third dimension shapes your security implementation, whether you control the agent and MCP client code (code-controlled) or are limited to configuring pre-built tools without modifying their runtime behavior (configuration-bound). This determines which security mechanisms are available to you at runtime.

In configuration-bound scenarios, you use an AI coding assistant such as Kiro or Claude Code and configure credentials in your mcp.json file. You select which IAM role or profile the agent uses, but you cannot modify the agent’s runtime behavior. The agent calls AWS APIs using whatever credentials you configured ahead of time, and you cannot inject session policies or tags into those calls programmatically. Your security controls must be in place before the agent runs. You select narrowly scoped roles at configuration time, and your organization enforces guardrails through permission boundaries and service control policies (SCPs). These mechanisms restrict what the agent can do regardless of which role the developer selects.

In code-controlled scenarios, you build or deploy a custom agent on Amazon Bedrock AgentCore, Amazon EC2, Amazon EKS, or your local machine, or you build and run a custom MCP server. Because you control the runtime code, you can implement credential management programmatically. For custom agents, this means calling AssumeRole with session policies scoped to each tool invocation, attaching session tags for differentiation, and obtaining temporary credentials with the minimum permissions each operation requires. For custom MCP servers, you can inject session policies into every AWS API call the server makes, applying a consistent set of restrictions across all operations. Both approaches give you runtime IAM controls that are not available in config-bound scenarios.

Deployment pattern summary

The following table summarizes how these dimensions combine.

Source type MCP server type Client control Credential source Differentiation mechanism Example use case
AI coding assistant AWS-managed MCP Config-bound Local (AWS CLI, env vars, ) Automatic context keys Kiro calling AWS-managed MCP server
AI coding assistant Self-managed MCP (local or remote) Config-bound Local (AWS CLI, env vars, ) Manual session tags or session policies Kiro calling local AWS MCP server
Agent on hosting environment AWS-managed MCP Code-controlled Execution role or AssumeRole Automatic context keys Amazon Bedrock AgentCore agent calling AWS-managed MCP server
Agent on hosting environment Self-managed MCP (remote) Code-controlled Execution role or AssumeRole Manual session tags or session policies Agent calling AWS MCP server deployed on Amazon Bedrock AgentCore

Your deployment pattern and level of client control determine which of the following security principles apply and how you implement them.

Three security principles for agent access

With this understanding of deployment patterns, let’s explore the three security principles that apply across all patterns.

  • Principle 1 – Assume all granted permissions could be used: Design permissions based on the acceptable scope of impact, not intended functionality alone.
  • Principle 2 – Provide organizational guidance on role usage: Enforce permission design through role governance, session policies, permission boundaries, and organizational policies.
  • Principle 3 – Differentiate AI-driven from human-initiated actions: Apply different IAM rules based on whether the action comes from an agent or a human.

Security principle 1: Assume all granted permissions could be used

The first security principle is fundamental. Any permission you grant to an agent can be exercised, regardless of your intended use case. If you give an agent s3:DeleteObject permission with a tool that can call the API, you must assume it can delete any Amazon Simple Storage Service (Amazon S3) object it has access to. This can happen in ways you cannot predict or fully prevent through code review alone. This non-deterministic behavior requires a shift in your approach to IAM permissions.

Traditional applications follow deterministic code paths. You can review the source code, identify every API call, and grant the permissions needed. AI agents operate differently. They make decisions at runtime based on reasoning, context, and learned patterns. You cannot predict which AWS APIs or tools an agent will call or which resources it will access. Static analysis of agent code tells you what tools are available, but not which tools will be invoked or how they’ll be used.

This creates a challenge when developers configure agents to use AWS credentials. Developers commonly use existing IAM roles, such as the role their traditional application uses or their local admin role for the AWS CLI. These roles were designed assuming predictable behavior and human judgment. Your local admin role has s3:* permissions because you exercise judgment on what to delete and when. You understand the context, recognize production resources, and can assess the impact of your actions.

An agent with that same role operates at machine speed without human judgment. It can delete production data through hallucination or be directed through prompt injection to perform unintended actions. It can also make a logical error in its reasoning that leads to unintended operations. The speed and scale at which agents operate increases the potential scope of these issues. An agent can make thousands of API calls in seconds, so the impact of misconfigured permissions scales quickly.

Consider the following scenarios with overly permissive access.

  • Hallucination: The agent misinterprets a user request and performs the wrong action. An agent designed to clean up temporary files might hallucinate that production data is temporary and delete it.
  • Prompt injection: An outside party crafts unexpected input that influences the agent’s reasoning. An agent designed to query Amazon DynamoDB tables could be directed to call dynamodb:PutItem or dynamodb:DeleteItem on resources outside its intended scope.
  • Logic errors: The agent’s reasoning leads to an incorrect conclusion. An agent analyzing S3 storage costs might conclude that frequently accessed production data is unused and delete it to save costs.
  • Tool poisoning: A compromised MCP server or dependency performs unintended operations using the agent’s credentials. An agent with broad S3 and DynamoDB permissions connects to an MCP server whose dependency has been modified to exfiltrate data. The compromised tool reads sensitive objects and writes them to an attacker-controlled location, all within the agent’s granted permissions.

This security principle reframes how you approach IAM permissions for agents. Instead of asking what does the agent need to do?, ask what is the scope of impact if the agent acts outside its intended use case? Design permissions based on the acceptable scope of access, not only on intended functionality. If an agent needs to read S3 objects, grant s3:GetObject, not s3:*. If it needs to write to specific paths, use resource-level conditions to restrict access to those paths. Consider what tools the agent has access to and what API calls those tools can make. Design permissions that limit what the agent is allowed to perform based on organizational policy. This doesn’t mean agents can’t have write or delete permissions. It means you and your organization must consider what resources those permissions apply to and what safeguards are in place.

Beyond IAM policies, consider implementing data perimeters as an additional layer of defense. Data perimeters use VPC endpoint policies, resource control policies (RCPs), resource policies, and service control policies (SCPs) to restrict access based on identity, resource, and network boundaries. For agents, data perimeters help verify that even if IAM permissions are broader than intended, access is limited to trusted resources from expected networks. For more information, see Building a data perimeter on AWS.

Practical implementation guidance:

  • Apply least privilege rigorously: If an agent needs read access, grant read permissions. If it needs write access, grant write to specific resources, not all resources of that type.
  • Use resource-level restrictions: Employ IAM policy conditions to limit permissions to specific buckets, paths, tables, or other resources. Don’t grant blanket permissions across all resources.
  • Consider read-only alternatives: Evaluate whether the agent’s task can be accomplished with read-only access. Many analysis and reporting tasks don’t require write or delete permissions.
  • Implement comprehensive monitoring: Set up Amazon CloudWatch alarms for unexpected agent actions, unusual access patterns, or operations on sensitive resources. Monitor for sensitive operations like deletions or modifications to production resources.
  • Conduct regular permission audits: As agents gain new tools and capabilities, developers often add permissions incrementally without removing unused ones. An agent that started with read-only access can gradually accumulate write and delete permissions across multiple services. Review agent IAM roles and policies regularly to identify and remove permissions that are no longer needed.
  • Verify MCP server integrity: Verify the provenance and integrity of MCP servers before granting them access to AWS credentials. Maintain an organizational registry of approved MCP servers and their expected behavior, and monitor for unauthorized server deployments that might have assumed execution roles. For more on agentic application risks, see the OWASP Top 10 for Agentic Applications.

Security principle 1 establishes the foundation. Understand the scope of every permission you grant. The next two security principles build on this foundation.

Security principle 2: Provide organizational guidance on role usage

The second security principle addresses organizational governance. Principle 1 requires that you design permissions based on acceptable scope of impact. Principle 2 addresses how your organization enforces that design through role governance, session policies, permission boundaries, and organizational policies.

When developers adopt AI coding assistants and configure MCP servers, they choose which credentials to use. Without organizational controls, developers often use existing roles (such as personal admin roles, shared development roles, or production roles) that were designed for human use with far more permissions than agents need. For agents deployed on hosting environments, you configure execution roles, but the same question applies. What permissions should those roles have, and how do you enforce consistency across deployments? The answer depends on your level of client control.

When you control the agent code

When you build or deploy custom agents on Amazon Bedrock AgentCore, Amazon EC2, Amazon EKS, or locally, you control the runtime code and can implement dynamic credential management. This is the strongest enforcement model because you can scope permissions per tool invocation at runtime. The same applies if you build or modify a custom MCP server. Because you control the server code, you can inject session policies into every AWS API call the server makes.

The IAM role defines the permission ceiling for the agent across all its tools. Instead of creating a separate role for every tool or MCP server, you use session policies to scope down the role’s permissions per operation. When the agent invokes a specific tool, it calls AssumeRole with a session policy that restricts permissions to just what that tool requires. The effective permissions are the intersection of the role’s policies and the session policy. Session policies restrict permissions but never expand them. If a role grants broad permissions but you attach the ReadOnlyAccess managed policy as a session policy, the agent can only perform read operations. You can also use inline session policies for resource-specific restrictions, such as limiting access to specific S3 buckets or DynamoDB tables.

The following example shows how to implement session policies in agent code.

import boto3

# Uses the execution IAM role as part of AgentCore Runtime
sts = boto3.client('sts')

# Assume role with ReadOnlyAccess managed policy as session policy
response = sts.assume_role(
    RoleArn='arn:aws:iam::111122223333:role/AgentDataRole',
    RoleSessionName='agent-data-reader',
    PolicyArns=[
        {'arn': 'arn:aws:iam::aws:policy/ReadOnlyAccess'}
    ],
    DurationSeconds=3600
)

# Use the temporary credentials
credentials = response['Credentials']
s3 = boto3.client(
    's3',
    aws_access_key_id=credentials['AccessKeyId'],
    aws_secret_access_key=credentials['SecretAccessKey'],
    aws_session_token=credentials['SessionToken']
)

For agents on hosting environments like Amazon Bedrock AgentCore, the execution role serves two purposes. It’s the trust anchor that lets the agent call AssumeRole for tool-specific credentials, and it can supply baseline permissions that all operations need, such as writing logs to CloudWatch. For tool-specific operations that access customer resources, use AssumeRole with session policies to obtain scoped temporary credentials rather than using the execution role’s permissions directly. This centralized execution model simplifies enforcing consistent session policies across all agent deployments. Agents can also attach tags when assuming roles for differentiation purposes (covered in Security principle 3).

When you’re configuration bound

When you use an AI coding assistant like Kiro or Claude Code with off-the-shelf MCP servers, you configure credentials in your mcp.json file but cannot modify the agent’s runtime behavior. Your security controls must be established before the agent runs.

Your first control is role selection. As described in the preceding deployment patterns section, AI coding assistants use credentials from the developer’s local environment. Create agent-specific IAM roles with narrower permissions than equivalent human roles, and direct developers to use them. For self-managed MCP servers running locally, the developer specifies the role through environment variables in the mcp.json configuration.

{
  "mcpServers": {
    "awslabs.aws-pricing-mcp-server": {
      "command": "uvx",
      "args": ["awslabs.aws-pricing-mcp-server@latest"],
      "env": {
        "AWS_PROFILE": "agent-dev-role",
        "AWS_REGION": "us-east-1"
      }
    }
  }
}

For AWS-managed MCP servers, the developer connects through the mcp-proxy-for-aws proxy and specifies the role through the profile parameter.

{
  "mcpServers": {
    "aws-mcp": {
      "command": "uvx",
      "args": [
        "mcp-proxy-for-aws@latest",
        "https://aws-mcp.us-east-1.api.aws/mcp",
        "--profile", "agent-dev-role",
        "--metadata", "AWS_REGION=us-east-1"
      ]
    }
  }
}

Only role selection depends on developer compliance. IAM permission boundaries provide organizational enforcement without requiring code changes or developer cooperation. A permission boundary is a managed policy that your security team attaches to an IAM role to set the maximum permissions that role can grant. The effective permissions are the intersection of the role’s identity-based policies and the permission boundary. Permission boundaries are most effective on agent-specific roles that your organization creates for agent use. They ensure those roles cannot exceed their intended permissions even if misconfigured. If a developer configures their existing role in mcp.json instead, a permission boundary on that role restricts all use of the role, not just agent use. For AWS-managed MCP servers, principle 3’s context keys address this gap. They let you write IAM policies that restrict actions only when they come through an MCP server, leaving the developer’s direct use of the same role unaffected. For self-managed MCP servers, modifying the server code to AssumeRole into an organization-defined role provides a similar override, and session tags can be attached during that AssumeRole for differentiation (see principle 3). For multi-account environments, SCPs in AWS Organizations provide guardrails at the account or organizational unit level. SCPs set the maximum permissions for all principals in an account, giving your central governance team control over agent permissions across your organization.

Organizational governance at scale

Whether your agents are config-bound or code-controlled, you need organizational mechanisms to enforce consistent governance across teams and accounts.

Tag IAM roles intended for agent use with a consistent identifier, such as a tag key of Usage with a value of Agent. This lets your governance team inventory all agent roles across accounts, identify roles that don’t have permission boundaries, and distinguish agent roles from human roles in audit reports. You can also use tag-based conditions in SCPs to enforce that only properly tagged roles are used for agent operations. For AWS-managed MCP servers, the automatic context keys (principle 3) provide this identification without requiring role tags, but tagging remains useful for role inventory and audit purposes.

Use CloudTrail to monitor all API calls made by agent sessions and set up CloudWatch alarms for sensitive operations like resource deletion or permission changes. Principle 3 covers how to filter and analyze agent activity using context keys (AWS-managed MCP) and session tags (self-managed MCP).

For multi-account environments, combine SCPs with permission boundaries and resource control policies (RCPs) for layered enforcement. SCPs set the maximum permissions for principals within your organization at the account or organizational unit level, while permission boundaries constrain individual roles. RCPs enforce controls at the resource level regardless of the caller’s organizational membership, protecting resources even from cross-account access. Verify that the AWS services you use support MCP context keys in RCP evaluation. This layered approach gives your central governance team control over agent permissions across your organization, even when individual teams manage their own accounts and roles. Conduct quarterly reviews of agent roles and session policies to identify permissions that are no longer needed as agent capabilities evolve.

Practical implementation guidance:

  • For code controlled agents: Implement session policies for every tool invocation. Use AssumeRole with the minimum permissions each operation requires rather than relying on the execution role’s base permissions.
  • For config-bound agents: Create agent-specific IAM roles with narrower permissions than human roles or configure self-managed MCP servers to AssumeRole into an organization-defined role. Have your security team attach permission boundaries to agent-specific roles to enforce maximum permissions regardless of developer role selection.
  • At the organization level: Tag agent roles consistently, enforce guardrails through SCPs, and monitor agent activity through CloudTrail. Conduct quarterly reviews to remove unused permissions.

Security principle 2 gives you organizational control over agent permissions through mechanisms matched to your level of client control. Session policies and dynamic credential scoping enforce permissions at runtime for code-controlled agents. Permission boundaries and SCPs enforce permissions at the organizational level for config-bound agents. The next principle adds a complementary layer of governance at the resource level based on whether a human or agent is performing the action.

Security principle 3: Differentiate AI-driven from human-initiated actions

The third security principle adds an additional level of control on top of principle 2. Where principle 2 governs what permissions an agent has, this principle governs what the agent can do with those permissions based on whether the action is AI-driven or human-initiated.

This principle is essential for two reasons. For AWS-managed MCP servers, you cannot modify the server code to inject session policies or call AssumeRole with scoped credentials. The developer’s credentials flow through as-is. Context keys are your primary mechanism to restrict agent actions differently from human-initiated actions on the same role. For self-managed MCP servers where principle 2’s session policies are already in place, differentiation adds a second layer of defense at the resource level. Even if the session policy is broader than intended, differentiation policies can deny specific dangerous operations when performed through an agent.

For example, you can allow both humans and agents to read Amazon S3 objects, but deny delete operations when accessed through agents. Without a differentiation mechanism, IAM policies can’t distinguish between AI-driven actions and human-initiated actions. If a developer has s3:DeleteObject permission and uses an agent with their credentials, the agent also has s3:DeleteObject permission with no way to restrict it.

Differentiation gives you granular governance. Allow human-initiated actions with broad permissions while restricting agent actions to narrower permissions. Apply different rules based on context and implement progressive restrictions. Allow read operations for everyone, require approval for AI-driven write operations, and deny delete operations for agent actions entirely. Maintain audit trails showing which actions were AI-driven versus human-initiated, essential for compliance and security investigations.

When agents bypass MCP servers

Differentiation through condition keys and session tags applies when the agent accesses AWS through an MCP server. AI coding assistants like Kiro and Claude Code have access to general-purpose tools, including bash, shell, and code execution. When an agent uses a bash tool to run an AWS CLI command like aws s3 rm s3://my-bucket/my-object or executes a Python script that calls boto3 directly, the request goes straight to AWS using the developer’s existing credentials. The request bypasses MCP servers entirely. The aws:ViaAWSMCPService condition key isn’t set, session tags from MCP server AssumeRole calls aren’t applied, and IAM policies conditioned on these values don’t evaluate.

This means a deny policy like “Condition": {"Bool": {"aws:ViaAWSMCPService": “true"}} blocks the agent when it calls Amazon S3 through a managed MCP server, but doesn’t block the same agent when it runs the equivalent AWS CLI command through a bash tool. The agent has two paths to the same AWS API, and differentiation controls govern one path.

The condition keys work as designed, differentiating MCP-mediated access from direct access. This is a scope boundary. Differentiation controls secure the MCP access path. For the direct access path, principles 1 and 2 are your controls. Least privilege on the underlying IAM role (principle 1) and organizational guardrails like permission boundaries and SCPs (principle 2) apply regardless of how the agent reaches AWS. If the role doesn’t have s3:DeleteObject permission, the agent can’t delete objects through a bash tool or through an MCP server.

Restricting which tools an agent can access is a complementary control outside the scope of IAM. You can use agent frameworks and hosting environments such as Amazon Bedrock AgentCore to limit the set of available tools, removing general-purpose execution capabilities for agents that interact with AWS exclusively through MCP servers. When you combine tool restriction with the IAM controls in this post, you close the gap between the MCP access path and the direct access path.

AWS-managed MCP servers: Automatic context keys

AWS-managed MCP servers, including the AWS MCP Server, Amazon EKS MCP Server, and Amazon ECS MCP Server, offer differentiation by default. They automatically add IAM context keys to every downstream AWS service call. These context keys are aws:ViaAWSMCPService, a boolean set to true when the request comes through any AWS-managed MCP server. The second key is aws:CalledViaAWSMCP, a string containing the MCP server name like aws-mcp.amazonaws.com, eks-mcp.amazonaws.com, or ecs-mcp.amazonaws.com. No configuration is required on your part. You only need to write IAM policies that check for these keys to apply different rules for agent actions.

The following IAM policy denies delete operations when accessed through any AWS-managed MCP server.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowS3ReadOperations",
    "Effect": "Allow",
    "Action": [
      "s3:GetObject",
      "s3:ListBucket"
    ],
    "Resource": "*"
  }, {
    "Sid": "DenyDeleteWhenAccessedViaMCP",
    "Effect": "Deny",
    "Action": [
      "s3:DeleteObject",
      "s3:DeleteBucket"
    ],
    "Resource": "*",
    "Condition": {
      "Bool": {
        "aws:ViaAWSMCPService": "true"
      }
    }
  }]
}

When a request doesn’t come through an AWS-managed MCP server, the aws:ViaAWSMCPService condition key isn’t present in the request context. The Deny statement only applies when the key is explicitly set to true, so human-initiated actions are unaffected by this policy.

You can also restrict operations to specific MCP servers. With this policy, you can run EKS operations only when accessed through the EKS MCP server, not through the AWS API MCP server.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowEKSOperationsViaEKSMCP",
    "Effect": "Allow",
    "Action": "eks:*",
    "Resource": "*",
    "Condition": {
      "StringEquals": {
        "aws:CalledViaAWSMCP": "eks-mcp.amazonaws.com"
      }
    }
  }, {
    "Sid": "DenyEKSOperationsViaOtherMCP",
    "Effect": "Deny",
    "Action": "eks:*",
    "Resource": "*",
    "Condition": {
      "Bool": {
        "aws:ViaAWSMCPService": "true"
      },
      "StringNotEquals": {
        "aws:CalledViaAWSMCP": "eks-mcp.amazonaws.com"
      }
    }
  }]
}

Self-managed MCP servers: Manual session tags

Self-managed MCP servers, whether AWS-provided servers from the AWS MCP GitHub repository or custom servers you build yourself, don’t automatically add IAM context keys. To implement differentiation with self-managed servers, you must configure the MCP server to add session tags when assuming IAM roles. This requires modifying your MCP server to call AWS STS AssumeRole with tags attached. The tags remain active for the duration of the assumed role session and can be referenced in IAM policies using the aws:PrincipalTag condition key. This approach gives you flexibility and control over the session tag configuration. To maintain consistency, verify that all MCP server instances add the appropriate tags.

The following example shows how to configure your MCP server to add session tags.

import boto3

sts = boto3.client('sts')

response = sts.assume_role(
    RoleArn='arn:aws:iam::111122223333:role/MCPServerRole',
    RoleSessionName='mcp-server-session',
    Tags=[
        {'Key': 'AccessType', 'Value': 'AI'},
        {'Key': 'Source', 'Value': 'AgentRuntime'},
        {'Key': 'MCPServer', 'Value': 'org-data-server'}
    ]
)

# Use the temporary credentials from response['Credentials']
credentials = response['Credentials']

After your MCP server has added session tags, you can write IAM policies that check for these tags to differentiate agent actions.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "AllowS3ReadOperations",
    "Effect": "Allow",
    "Action": [
      "s3:GetObject",
      "s3:ListBucket"
    ],
    "Resource": "*"
  }, {
    "Sid": "DenyDeleteWhenAccessedViaAI",
    "Effect": "Deny",
    "Action": [
      "s3:DeleteObject",
      "s3:DeleteBucket"
    ],
    "Resource": "*",
    "Condition": {
      "StringEquals": {
        "aws:PrincipalTag/AccessType": "AI"
      }
    }
  }]
}

Session tags and session policies are both passed to AssumeRole, but serve different purposes. Session policies (covered in security principle 2) constrain what permissions the agent has. Session tags (covered here in security principle 3) mark the session as AI-driven, enabling IAM policies to differentiate between agent and human actions. You can use both in the same AssumeRole call for defense-in-depth. The session policy constrains what the agent can do. The session tags let IAM policies apply different rules based on the actor type.

The following example uses both session policies and session tags together.

import boto3

sts = boto3.client('sts')

# Assume role with both managed session policy and tags
response = sts.assume_role(
    RoleArn='arn:aws:iam::111122223333:role/AgentDataRole',
    RoleSessionName='agent-data-reader',
    PolicyArns=[                              # Principle 2: Constrains permissions
        {'arn': 'arn:aws:iam::aws:policy/ReadOnlyAccess'}
    ],
    Tags=[                                    # Principle 3: Enables differentiation
        {'Key': 'AccessType', 'Value': 'AI'},
        {'Key': 'Source', 'Value': 'AgentRuntime'},
        {'Key': 'MCPServer', 'Value': 'org-data-server'}
    ],
    DurationSeconds=3600
)

CloudTrail logging and audit trails

Both differentiation mechanisms generate CloudTrail logs for audit trails. For AWS-managed MCP servers, downstream AWS API calls include the MCP service identifier in the invokedBy, sourceIPAddress, and userAgent fields. You can filter on these fields to isolate agent activity. MCP-originated downstream calls are classified as data events, so you must enable data event logging on your CloudTrail trail to capture them.

{
  "eventVersion": "1.11",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROAEXAMPLE:developer-session",
    "arn": "arn:aws:sts::111122223333:assumed-role/DeveloperRole/developer-session",
    "accountId": "111122223333",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "AROAEXAMPLE",
        "arn": "arn:aws:iam::111122223333:role/DeveloperRole",
        "accountId": "111122223333",
        "userName": "DeveloperRole"
      }
    },
    "invokedBy": "aws-mcp.amazonaws.com"
  },
  "eventSource": "s3.amazonaws.com",
  "eventName": "GetObject",
  "sourceIPAddress": "aws-mcp.amazonaws.com",
  "userAgent": "aws-mcp.amazonaws.com",
  "eventType": "AwsApiCall",
  "managementEvent": false,
  "eventCategory": "Data"
}

For self-managed MCP servers with session tags, the tags appear in the requestParameters.principalTags field of the AssumeRole CloudTrail event. You can correlate the session name from the AssumeRole event to downstream API calls to trace agent activity.

{
  "eventSource": "sts.amazonaws.com",
  "eventName": "AssumeRole",
  "requestParameters": {
    "roleArn": "arn:aws:iam::111122223333:role/MCPServerRole",
    "roleSessionName": "mcp-server-session",
    "principalTags": {
      "AccessType": "AI",
      "Source": "AgentRuntime",
      "MCPServer": "org-data-server"
    }
  }
}

With these logs, you can query CloudTrail to find all AI-driven actions and analyze patterns of agent behavior. You can also identify unexpected or unauthorized operations and maintain compliance audit trails. Set up CloudWatch alarms to detect agent actions on sensitive resources or unusual patterns that indicate unintended access or misconfiguration.

Things to consider

When deciding between AWS-managed and self-managed MCP servers, consider the trade-offs. AWS-managed MCP servers offer the most straightforward path. Context keys are added automatically with no configuration on your part. Self-managed MCP servers require modifying code to add session tags. However, they give you complete control over the tags and let you implement custom functionality not available in AWS-managed servers. Organizations can use both approaches, AWS-managed servers for standard AWS operations and self-managed servers for specialized use cases.

Practical implementation guidance:

  • Assess direct access paths: Evaluate whether your agents have access to general-purpose tools (bash, shell, code execution) that can bypass MCP servers. If they do, rely on principles 1 and 2 for those paths and consider restricting tool availability where possible.
  • Choose a differentiation mechanism: Select based on your MCP server type (for managed, use context keys, for self-managed, use session tags).
  • For AWS-managed MCP: Write IAM policies that check aws:ViaAWSMCPService and aws:CalledViaAWSMCP condition keys. No MCP server configuration needed.
  • For self managed MCP: Modify MCP server code to add session tags when assuming roles. Verify consistent tag application across all instances.
  • Update IAM policies: Add differentiation conditions to existing policies. Test in non-production first to verify behavior.
  • Monitor CloudTrail logs: Verify differentiation is working by checking for context keys or session tags in CloudTrail events.
  • Set up alerts: Configure CloudWatch alarms for AI-driven sensitive operations or policy violations.
  • Perform regular audits: Review IAM policies quarterly to verify differentiation conditions remain correct as agent capabilities evolve.

Conclusion

Securing AI agent access to AWS resources requires building deterministic IAM controls for non-deterministic AI systems. The three security principles give you a defense-in-depth framework that adapts to your deployment pattern and level of client control.

Your implementation path depends on your situation. Start with principle 1. Audit current agent permissions and default to read-only access where possible. Next, implement principle 2. For config-bound scenarios, establish permission boundaries and select agent-specific roles. For code-controlled scenarios, implement dynamic session policies scoped to each tool invocation. Finally, add principle 3 differentiation based on your MCP server type. Use automatic context keys with AWS-managed MCP servers, or configure session tags with self-managed servers.

By applying these three security principles, you can use AI agents while maintaining the governance and compliance controls your organization requires.

Riggs Goodman III

Riggs Goodman III

Riggs is a Principal Solution Architect at AWS. His current focus is on AI security, providing technical guidance, architecture patterns, and leadership for customers and partners to build AI workloads on AWS. Internally, Riggs focuses on driving overall technical strategy and innovation across AWS service teams to address customer and partner challenges.

Scale fine-grained permissions across warehouses with Amazon Redshift and AWS IAM Identity Center

Post Syndicated from Raghu Kuppala original https://aws.amazon.com/blogs/big-data/scale-fine-grained-permissions-across-warehouses-with-amazon-redshift-and-aws-iam-identity-center/

Amazon Redshift is a fully managed, petabyte-scale cloud-based data warehouse that you can use to scale analytics workloads effortlessly. As organizations expand their analytics capabilities across multiple business units, they need streamlined approaches for defining and managing fine-grained permissions for each warehouse. Many organizations use external identity providers (IdPs) like Microsoft Entra ID, Okta, or Ping to manage workforce identities centrally and need streamlined data warehouse integration with consistent access controls. We address these challenges by introducing Amazon Redshift federated permissions with AWS IAM Identity Center integration so that you can define security policies once and automatically enforce them across the warehouses in your account.

Amazon Redshift federated permissions are now supported with IAM Identity Center across multiple AWS Regions, where you can use identities from supported identity provider (IdP) such as Microsoft Entra ID, Okta, Ping Identity, or OneLogin across supported AWS Regions with IAM Identity Center. This enables you to align with business requirements including resiliency and proximity to users. You can now extend IAM Identity Center from your primary AWS Region to additional Regions of your choice based on your data residency requirements. In that region, you can get horizontal multi-warehouse scalability by adding new warehouses using Amazon Redshift federated permissions across multiple warehouses. With Redshift federated permissions, you define data permissions once from any Redshift warehouse in that region and automatically enforce them across all warehouses in the account in that region.

This post provides a comprehensive technical walkthrough for implementing Amazon Redshift federated permissions with AWS IAM Identity Center to help achieve scalable data governance across multiple data warehouses. It demonstrates a practical architecture where an Enterprise Data Warehouse (EDW) serves as the producer data warehouse with centralized policy definitions, helping automatically enforce security policies to consuming Sales and Marketing data warehouses without manual reconfiguration. You will learn how to do the following:

  • Configure IAM Identity Center connections for both data sharing producers and consumers
  • Register Amazon Redshift serverless namespaces with AWS Glue Data Catalog
  • Set up trusted identity propagation (TIP)
  • Create and attach Dynamic data masking policies to help protect personally identifiable information (PII) like customer dates of birth
  • Implement row-level security policies to control data visibility based on user roles
  • Map IdP groups to Amazon Redshift database roles for seamless access management

Prerequisites

Before you begin, verify that you have the following:

  • An AWS account with admin role privileges
  • Assign data lake admin permissions to above admin role. For instructions, see Create a data lake administrator
  • Enable IAM Identity Center integration using the Lake Formation
  • Review the blog post to understand the setup process of AWS IAM Identity Center integration with Amazon Redshift Query Editor v2
  • IAM Identity Center enabled in your AWS account, with users and groups created as listed under Solution overview section of User access (figure 2)
  • As an Amazon Redshift superuser, grant CONNECT, CREATE TABLE, INSERT, SELECT, and sys:secadmin permissions to AWSIDC:awssso-admin database role
  • An IAM role for IAM Identity Center access:
    • Step 1:Create an IAM policy for Amazon Redshift access. To integrate Amazon Redshift with IAM Identity Center, create an IAM policy (for example, aws-idc-policy) in the account where your Amazon Redshift data warehouse exists:
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
              "redshift:DescribeQev2IdcApplications",
              "redshift-serverless:ListNamespaces",
              "redshift-serverless:ListWorkgroups",
              "redshift-serverless:GetWorkgroup"
            ],
            "Resource": [
              "arn:aws:redshift-serverless:<AWS Region>:<AWS Account ID>:workgroup/*",
              "arn:aws:redshift-serverless:<AWS Region>:<AWS Account ID>:namespace/*"
            ]
          },
          {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
              "sso:DescribeApplication",
              "sso:DescribeInstance"
            ],
            "Resource": [
              "arn:aws:sso:::instance/<IAM Identity Center Instance ID>",
              "arn:aws:sso::<AWS Account ID>:application/<IAM Identity Center Instance ID>/*"
            ]
          }
        ]
      }

    • Step 2: Create the IAM role. Create an IAM role (Amazon Redshift – Customizable) in the account where your Amazon Redshift data warehouse exists (for example, IAMIDCRedshiftRole).
    • Step 3: Attach IAM policies to the role. Attach the following two IAM policies to the previously mentioned role:
    • Step 4: Update the trust relationships. Update the trust relationships for this role with the following:
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": {
              "Service": "redshift.amazonaws.com"
            },
            "Action": [
              "sts:AssumeRole",
              "sts:SetContext"
            ]
          }
        ]
      }

      Note: AmazonRedshiftFederatedAuthorization is a managed policy that provides the necessary permissions for running queries with Amazon Redshift federated authorization.

  • Attach above IAMIDCRedshiftRole IAM role to all Redshift serverless endpoints

Solution overview

The following architecture diagram demonstrates federated permissions in a multi-warehouse environment, enabling scalable data governance across Amazon Redshift warehouses by automatically enforcing security policies.

Figure 1 : Sample architecture diagram

Figure 1: Sample architecture diagram

User access

Users can access data warehouses through Amazon Redshift Query Editor v2, third-party SQL editors (such as DBeaver and SQL Workbench), or custom client applications. The access methods help provide consistent security enforcement.

Figure 2: Solution overview flow

Figure 2: Solution overview flow

AWS IAM Identity Center integration

IAM Identity Center provides centralized authentication with single sign-on capabilities and automatically assigns role-based permissions based on organizational roles. This identity federation links corporate identities directly to AWS resources, making sure that authentication occurs at the identity layer before warehouse access.

Multi-warehouse architecture

This architecture uses three distinct data warehouses that serve different business functions while sharing centralized security policies.

Enterprise Data Warehouse (EDW)

The EDW serves as the central repository for enterprise data. In this architecture, customer and product data are stored in the Customer Profile Database (CPD), where administrators define two critical security policies:

  • Dynamic data masking (DDM) – Masks sensitive customer Date of Birth (DOB) fields for both Sales Analyst and Marketing Analyst roles, helping protect personally identifiable information (PII) while allowing analytical work
  • Row-level security (RLS) – Controls product visibility based on user roles. Sales Analysts view only launched products, while Marketing Analysts view both launched and planned products

The EDW registers with the AWS Glue Data Catalog, creating a unified metadata repository that makes data discoverable across the warehouses in the account. This registration establishes the foundation for federated permissions, enabling automatic policy propagation.

Sales data warehouse

When Sales Analysts query customer and product tables, the system automatically enforces policies defined in the EDW through federated permissions. The registered namespace from the EDW automatically mounts as an external database, alleviating the need to recreate or reattach policies. Customer DOB fields appear masked, and only launched products are visible without additional configurations.

Marketing data warehouse

The Marketing Data Warehouse automatically inherits and enforces EDW security policies. Customer DOB fields remain masked to help protect PII, but with RLS policies, Marketing Analysts can view both launched and planned products. This provides the broader visibility needed for marketing planning. This differentiated access control is automatically enforced based on user roles.

Walkthrough

In this walkthrough, you create two Amazon Redshift IAM Identity Center (IDC) connections:

  1. Data sharing producer identity center connection – Assigned to the edw-wg Amazon Redshift serverless workgroup
  2. Data sharing consumer identity center connection – Assigned to the cpd-sales-wg and cpd-marketing-wg Amazon Redshift serverless workgroups

Set up IDC connections for Amazon Redshift federated permissions

In this section, you configure the IAM Identity Center connections that enable federated authentication across your warehouses. You will create separate connections for the producer (policy-defining) warehouse and consumer warehouses.

Configure Amazon Redshift data sharing producer IDC connection

To create the producer IDC connection:

  1. Open the Amazon Redshift Serverless console.
  2. Choose IAM Identity Center connections by expanding the hamburger menu.
  3. Choose Create application.
  4. Verify that you see “Amazon Redshift connected to IAM Identity Center”, and then choose Next.
  5. Configure the connection properties:
    • For IAM Identity Center display name, enter a name.
    • For Managed application name, enter rs-multicluster-producer.
    • For Identity provider namespace, choose AWSIDC.
    • For IAM role for IAM Identity Center access, choose the TIP IAM role that you created.
    • For Query editor v2 application, choose Enable the query editor v2 application.
    • For IAM Identity Center application type, choose Configure Amazon Redshift federated permissions using AWS IAM Identity Center (Recommended).
    • Choose Next.
  6. For Configure client connections that use third-party IdPs, choose No.
  7. Choose Next.
  8. Verify that the configuration details match your inputs and then choose Create Application.
Figure 3: Data sharing producer IDC connection

Figure 3: Data sharing producer IDC connection

Configure data sharing consumer IDC connection

To create the consumer IDC connection:

  1. Open the Amazon Redshift Serverless console.
  2. Choose IAM Identity Center connections by expanding the hamburger menu.
  3. Choose Create application.
  4. Verify that you see “Amazon Redshift connected to IAM Identity Center”, and then choose Next.
  5. Configure the connection properties:
    • For IAM Identity Center display name, enter a name.
    • For Managed application name, enter rs-multicluster-consumer.
    • For Identity provider namespace, choose AWSIDC.
    • For IAM role for IAM Identity Center access, choose the TIP IAM role that you created.
    • For Query editor v2 application, you will see the notification “You already have a query editor v2 application.”
    • For IAM Identity Center application type, deselect Configure Amazon Redshift federated permissions using AWS IAM Identity Center (Recommended).
    • For Trusted identity propagation, choose AWS Lake Formation access grants and Amazon Redshift Connect.
    • Choose Next.
  6. For Configure client connections that use third-party IdPs, choose No.
  7. Choose Next.
  8. Verify that the configuration details match your inputs, and then choose Create Application.
  9. Add your required users or groups to the IDC application for Amazon Redshift data sharing consumers.
Figure 4: Data sharing consumer IDC connection

Figure 4: Data sharing consumer IDC connection

Configure Amazon Redshift data sharing producer IDC connection for Amazon Redshift serverless namespace

To register the edw-ns namespace with federated permissions:

  1. Open the Amazon Redshift Serverless Namespace console.
  2. Choose your Amazon Redshift Serverless namespace.
  3. Choose Actions, and then select Register with AWS Glue Data Catalog.
  4. Choose Register with Amazon Redshift federated permissions.
  5. Choose Amazon Redshift federated permissions using AWS IAM Identity Center.
  6. Choose Register.
Figure 5: Amazon Redshift data warehouse registration with Glue Data Catalog

Figure 5: Amazon Redshift data warehouse registration with Glue Data Catalog

Figure 6: Amazon Redshift data warehouse registration with Glue Data Catalog

Figure 6: Amazon Redshift data warehouse registration with Glue Data Catalog

Note: IAM Identity Center managed application ARN Data sharing producer IDC connection created would be used.

Configure Amazon Redshift data sharing consumer IDC connection for existing serverless namespace

For cpd-sales-wg and cpd-marketing-wg serverless workgroups, gather the following information from your registered IAM Identity Center connection:

  • IAM Identity Center display name
  • Identity provider namespace
  • IAM Identity Center managed application ARN
  • IAM role for IAM Identity Center access

Run the following SQL command as a database administrator to enable the integration:

CREATE IDENTITY PROVIDER "<IAM Identity Center display name>" TYPE AWSIDC
NAMESPACE '<Identity provider namespace>'
APPLICATION_ARN '<IAM Identity Center managed application ARN>'
IAM_ROLE '<IAM role for IAM Identity Center access>';

To modify an existing identity provider, use the ALTER IDENTITY PROVIDER command:

ALTER IDENTITY PROVIDER "<IAM Identity Center display name>"
NAMESPACE '<Identity provider namespace>';
ALTER IDENTITY PROVIDER "<IAM Identity Center display name>"
IAM_ROLE default | '<IAM role for IAM Identity Center access>';

Data preparation and access setup from producer

In this section, you create the customer and product tables, load sample data, create DDM and RLS policies, attach the policies to database roles and grant SELECT permissions to the roles.

Prepare data on EDW

Connect to the EDW data warehouse as an IDC Admin user and run the following SQL commands.

Create the product table:

CREATE TABLE product (
  product_id VARCHAR(16) NOT NULL,
  product_desc VARCHAR(200),
  current_price NUMERIC(7,2),
  wholesale_cost NUMERIC(7,2),
  category_desc VARCHAR(50),
  launch_status VARCHAR(50)
);

Insert sample product data:

INSERT INTO product 
VALUES 
  ('AAAAAAAAAFNPEAAA','At least concerned authors adopt just brown, federal',7.12,4.12,'Jewelry','launched'),
  ('AAAAAAAAOAAGDAAA','Complex services may not find totally changing accountants. Tiny, available ministers could not know always systems. Hot, male speakers discer',8.08,5.49,'Shoes','planned'),
  ('AAAAAAAAMJJMCAAA','Rows could prevent political, old duties. Just international stairs would regret police. Conditions discard always interesting, warm years. Present jobs shall take nearby relatively dreadful',8.18,5.31,'Jewelry','launched'),
  ('AAAAAAAAKLBLBAAA','Suddenly external sentences believe then by the assets. Simultaneously young feet could not probe separately shortly new men. Forms work again individuals. Images',17.96,7.9,'Shoes','launched'),
  ('AAAAAAAAMBKMCAAA','Clubs see finally materials. Significant objectives sell fairly left, civil power',3.18,3.84,'Books','launched'),
  ('AAAAAAAACPCAAAAA','Perhaps past preferences tell rather to a accounts. Very common feet can command never available final years; minutes expect recent, due employers. Altogether english shoes',9.84,0.19,'Electronics','planned'),
  ('AAAAAAAAFOIABAAA','More responsible characters go left factors. Championships shall stand twice new, important shows. Books could receive too able, national pounds. Central',3.55,2.2,'Books','launched'),
  ('AAAAAAAAKGBIAAAA','High, political changes shall not',9.55,5.25,'Electronics','launched');

Create the customer table:

CREATE TABLE customer (
  customer_id VARCHAR(16),
  first_name VARCHAR(20),
  last_name VARCHAR(30),
  date_of_birth VARCHAR(32),
  birth_country VARCHAR(20),
  email_address VARCHAR(50)
);

Insert sample customer data:

INSERT INTO customer
VALUES
  ('AAAAAAAALAMKHGBA','Regina','Coleman','1926-12-17','GAMBIA','[email protected]'),
  ('AAAAAAAAMCMKHGBA','John','Bell','1980-01-07','PAPUA NEW GUINEA','[email protected]'),
  ('AAAAAAAANNMKHGBA','Jacqueline','Pierre','1951-12-18','SAMOA','[email protected]'),
  ('AAAAAAAANFNKHGBA','Frank','Mackay','1992-03-19','HONG KONG','[email protected]'),
  ('AAAAAAAAOGNKHGBA','Anthony','Miller','1948-02-26','ALGERIA','[email protected]'),
  ('AAAAAAAACPOKHGBA','Bradley','Sawyer','1956-12-25','ZAMBIA','[email protected]'),
  ('AAAAAAAAOIPKHGBA','Robert','Carter','1951-01-01','UNITED STATES','[email protected]'),
  ('AAAAAAAALJPKHGBA','Ola','High','1980-11-19','SUDAN','[email protected]');

Create DDM and RLS policies

Create the masking policy for customer date of birth:

CREATE MASKING POLICY mask_cust_dob  
WITH (date_of_birth VARCHAR(32))  
USING (sha2(date_of_birth, 256)::TEXT);

Create RLS policies for product launch status:

CREATE RLS POLICY product_launch_status  
WITH (launch_status VARCHAR(50))   
USING (launch_status = 'launched');
  
CREATE RLS POLICY product_launch_status_all
WITH (launch_status VARCHAR(50))   
USING (launch_status IN ('launched','planned'));

Create Amazon Redshift DB roles for Sales and Marketing groups

Create the database roles:

CREATE ROLE "AWSIDC:awssso-sales";
CREATE ROLE "AWSIDC:awssso-marketing";

Attach masking policies

Attach the masking policy to both roles:

ATTACH MASKING POLICY mask_cust_dob  
ON dev.public.customer (date_of_birth)  
TO ROLE "AWSIDC:awssso-marketing";
ATTACH MASKING POLICY mask_cust_dob  
ON dev.public.customer (date_of_birth)  
TO ROLE "AWSIDC:awssso-sales";

Attach RLS policies and enable RLS on product table

Attach the RLS policies and enable row-level security:

ATTACH RLS POLICY product_launch_status  
ON dev.public.product  
TO ROLE "AWSIDC:awssso-sales"; 
ATTACH RLS POLICY product_launch_status_all  
ON dev.public.product  
TO ROLE "AWSIDC:awssso-marketing";
ALTER TABLE dev.public.product ROW LEVEL SECURITY ON;

Grant access to tables to roles

Grant SELECT permissions to both roles:

GRANT SELECT ON dev.public.customer TO ROLE "AWSIDC:awssso-sales";
GRANT SELECT ON dev.public.customer TO ROLE "AWSIDC:awssso-marketing";
GRANT SELECT ON dev.public.product TO ROLE "AWSIDC:awssso-sales"; 
GRANT SELECT ON dev.public.product TO ROLE "AWSIDC:awssso-marketing";

Connect to SALES data warehouse using IAM Identity Center

To connect as a Sales Analyst:

  1. Connect to cpd-sales-wg using the IAM Identity Center connection type as user sales-analyst, and then choose Continue.
  2. Choose sales-analyst, and then choose Next.
  3. Enter your password, and then choose Sign in.
  4. Enter your MFA code, and then choose Sign in.

You are now connected to Amazon Redshift Query Editor V2 with a successful connection to cpd-sales-wg as sales-analyst.

Figure 7: Connect to Sales data warehouse as IDC user

Figure 7: Connect to Sales data warehouse as IDC user

Query shared data as Sales Analyst

Query the customer table with dynamic data masking applied:

SELECT * FROM "dev@edw-ns"."public"."customer";

You can successfully access the customer table, but the sensitive information in the date_of_birth column is encrypted.

Figure 8: Result set of customer table

Figure 8: Result set of customer table

Query the product table with row-level security enabled:

SELECT * FROM "dev@edw-ns"."public"."product";

You can successfully access the product table, but only view data for products with a launch_status value of launched.

Figure 9: Result set of product table

Figure 9: Result set of product table

Note: To connect to the data sharing producer onboarded to Amazon Redshift federated permissions as an IDC user, a superuser is required to provide a CONNECT privilege to the IDC user trying to connect. For more information about how to grant the CONNECT privileges to the user, see Connect privileges in the Amazon Redshift Database Developer Guide.

Connect to Marketing data warehouse using IAM Identity Center

To connect as a Marketing Analyst:

  1. Connect to cpd-marketing-wg using the IAM Identity Center connection type as user marketing-analyst, and then choose Continue.
  2. Choose marketing-analyst, and then choose Next.
  3. Enter your password, and then choose Sign in.
  4. Enter your MFA code, and then choose Sign in.

You are now connected to Amazon Redshift Query Editor V2 with a successful connection to cpd-marketing-wg as marketing-analyst.

Figure 10: Connect to Marketing data warehouse as IDC user

Figure 10: Connect to Marketing data warehouse as IDC user

Query shared data as Marketing Analyst

Query the customer table with dynamic data masking applied:

SELECT * FROM "dev@edw-ns"."public"."customer";

You can successfully access the customer table, but the sensitive information in the date_of_birth column is encrypted.

Figure 11: Result set of customer table

Figure 11: Result set of customer table

Query the product table with row-level security enabled:

SELECT * FROM "dev@edw-ns"."public"."product";

You can successfully access the product table and view data for products with launch_status values of both launched and planned.

Figure 12: Result set of product table

Figure 12: Result set of product table

Additional resources

For more information about implementing federated permissions in your environment, see the following resources:

AWS Documentation

AWS Blogs

AWS Demo

Key benefits

  • Reduced administrative overhead – Centralized policy management removes manual replication
  • Consistent security enforcement – Policies apply uniformly across the warehouses and access methods
  • Seamless identity integration – Single sign-on with existing identity providers through trusted identity propagation and role-based access control

Conclusion

This post showed you how Amazon Redshift federated permissions with AWS IAM Identity Center integration helps streamline multi-warehouse data governance by centralizing security policy management. You define dynamic data masking and row-level security policies once in a central Enterprise Data Warehouse, and they automatically enforce across the connected data warehouses in the same account and Region.


About the authors

Raghu Kuppala

Raghu Kuppala

Raghu is an Analytics Specialist Solutions Architect experienced working in the databases, data warehousing, and analytics space. Outside of work, he enjoys trying different cuisines and spending time with his family and friends.

Satesh Sonti

Satesh Sonti

Satesh is a Principal Specialist Solutions Architect based out of Atlanta, specializing in building enterprise data platforms, data warehousing, and analytics solutions. He has over 20 years of experience in building data assets and leading complex data platform programs for banking and insurance clients across the globe.

Sandeep Adwankar

Sandeep Adwankar

Sandeep is a Senior Product Manager with Amazon SageMaker Lakehouse. Based in the California Bay Area, he works with customers around the globe to translate business and technical requirements into products that help customers improve how they manage, secure, and access data.

Sumukh Bapat

Sumukh Bapat

Sumukh is a Software Engineer at AWS. He works on improving customer experience for Amazon Redshift by solving complex problems in authentication, connectivity, and security. His work focuses on identity management, secure access, and distributed database systems.

Praveen Kumar Ramakrishnan

Praveen Kumar Ramakrishnan

Praveen is a Senior Software Engineer at AWS. He has nearly 20 years of experience spanning various domains including filesystems, storage virtualization and network security. At AWS, he focuses on enhancing the Redshift data security.

Ashish Ghodke

Ashish Ghodke

Ashish is a Software Engineer at Amazon Web Services, where he works on identity and access management systems for large-scale cloud services like Amazon Redshift. His work focuses on building secure authentication and single sign-on solutions for distributed systems. He is passionate about distributed systems, cloud security, and building reliable infrastructure at scale.

AWS Weekly Roundup: Amazon Connect Health, Bedrock AgentCore Policy, GameDay Europe, and more (March 9, 2026)

Post Syndicated from Sébastien Stormacq original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-amazon-connect-health-bedrock-agentcore-policy-gameday-europe-and-more-march-9-2026/

Fiti AWS Student Community Kenya!

Last week was an incredible whirlwind: a round of meetups, hands-on workshops, and career discussions across Kenya that culminated with the AWS Student Community Day at Meru University of Science and Technology, with keynotes from my colleagues Veliswa and Tiffany, and sessions on everything from GitOps to cloud-native engineering, and a whole lot of AI agent building.

JAWS Days 2026 is the largest AWS Community Day in the world, with over 1,500 attendees on March 7th. This event started with a keynote speech on building an AI-driven development team by Jeff Barr, and included over 100 technical and community experience sessions, lightning talks, and workshops such as Game Days, Builders Card Challenges, and networking parties.

Now, let’s get into this week’s AWS news…

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

  • Introducing Amazon Connect Health, Agentic AI Built for Healthcare — Amazon Connect Health is now generally available with five purpose-built AI agents for healthcare: patient verification, appointment management, patient insights, ambient documentation, and medical coding. All features are HIPAA-eligible and deployable within existing clinical workflows in days.
  • Policy in Amazon Bedrock AgentCore is now generally available — You can now use centralized, fine-grained controls for agent-tool interactions that operate outside your agent code. Security and compliance teams can define tool access and input validation rules using natural language that automatically converts to Cedar, the AWS open-source policy language.
  • Introducing OpenClaw on Amazon Lightsail to run your autonomous private AI agents — You can deploy a private AI assistant on your own cloud infrastructure with built-in security controls, sandboxed agent sessions, one-click HTTPS, and device pairing authentication. Amazon Bedrock serves as the default model provider, and you can connect to Slack, Telegram, WhatsApp, and Discord.
  • AWS announces pricing for VPC Encryption Controls — Starting March 1, 2026, VPC Encryption Controls transitions from free preview to a paid feature. You can audit and enforce encryption-in-transit of all traffic flows within and across VPCs in a region, with monitor mode to detect unencrypted traffic and enforce mode to prevent it.
  • Database Savings Plans now supports Amazon OpenSearch Service and Amazon Neptune Analytics — You can save up to 35% on eligible serverless and provisioned instance usage with a one-year commitment. Savings Plans automatically apply regardless of engine, instance family, size, or AWS Region.
  • AWS Elastic Beanstalk now offers AI-powered environment analysis — When your environment health is degraded, Elastic Beanstalk can now collect recent events, instance health, and logs and send them to Amazon Bedrock for analysis, providing step-by-step troubleshooting recommendations tailored to your environment’s current state.
  • AWS simplifies IAM role creation and setup in service workflows — You can now create and configure IAM roles directly within service workflows through a new in-console panel, without switching to the IAM console. The feature supports Amazon EC2, Lambda, EKS, ECS, Glue, CloudFormation, and more.
  • Accelerate Lambda durable functions development with new Kiro power — You can now build resilient, long-running multi-step applications and AI workflows faster with AI agent-assisted development in Kiro. The power dynamically loads guidance on replay models, step and wait operations, concurrent execution patterns, error handling, and deployment best practices.
  • Amazon GameLift Servers launches DDoS Protection — You can now protect session-based multiplayer games against DDoS attacks with a co-located relay network that authenticates client traffic using access tokens and enforces per-player traffic limits, at no additional cost to GameLift Servers customers.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

From AWS community
Here are my personal favorite posts from AWS community and my colleagues:

  • I Built a Portable AI Memory Layer with MCP, AWS Bedrock, and a Chrome Extension — Learn how to build a persistent memory layer for AI agents using MCP and Amazon Bedrock, packaged as a Chrome extension that carries context across sessions and applications.
  • When the Model Is the Machine — Mike Chambers built an experimental app where an AI agent generates a complete, interactive web application at runtime from a single prompt — no codebase, no framework, no persistent state. A thought-provoking exploration of what happens when the model becomes the runtime.

Upcoming AWS events
Check your calendar and sign up for upcoming AWS events:

  • AWS Community GameDay Europe — Think you know AWS? Prove it at the AWS Community GameDay Europe on March 17, a gamified learning event where teams compete to solve real-world technical challenges using AWS services.
  • AWS at NVIDIA GTC 2026 — Join us at our AWS sessions, booths, demos, and ancillary events in NVIDIA GTC 2026 on March 16 – 19, 2026 in San Jose. You can receive 20% off event passes through AWS and request a 1:1 meeting at GTC.
  • AWS Summits — Join AWS Summits in 2026: free in-person events where you can explore emerging cloud and AI technologies, learn best practices, and network with industry peers and experts. Upcoming Summits include Paris (April 1), London (April 22), and Bengaluru (April 23–24).
  • AWS Community Days — Community-led conferences where content is planned, sourced, and delivered by community leaders. Upcoming events include Slovakia (March 11), Pune (March 21), and the AWSome Women Summit LATAM in Mexico City (March 28)

Browse here for upcoming AWS led in-person and virtual events, startup events, and developer-focused events.

That’s all for this week. Check back next Monday for another Weekly Roundup!

— seb

Enhanced access denied error messages with policy ARNs

Post Syndicated from Stella Hie original https://aws.amazon.com/blogs/security/enhanced-access-denied-error-messages-with-policy-arns/

To help you troubleshoot access denied errors, we recently added the Amazon Resource Name (ARN) of the denying policy to access denied error messages. This builds on our 2021 enhancement that added the type of the policy denying the access to access denied error messages. The ARN of the denying policy is only provided in same-account and same-organization scenarios. This change is gradually rolling out across all AWS services in all AWS Regions.

What changed?

We added the policy ARN to access denied error messages for AWS Identity and Access Management (IAM) and AWS Organizations policies. Because of this change, you can now pinpoint the exact policy causing the denial. You don’t have to evaluate all the policies of the same type in your AWS environment to identify the culprit. The policy types covered in this update are service control policies (SCPs), resource control policies (RCPs), permissions boundaries policies, session policies, and identity-based policies.

For example, when a developer attempts to perform the ListRoles action in IAM and is denied because of an SCP:

Before:
An error occurred (AccessDenied) when calling the ListRoles operation: User: arn:aws:iam::123456789012:user/Matt is not authorized to perform: iam:ListRoles on resource: arn:aws:iam::123456789012:role/* with an explicit deny in a service control policy

Enhanced:
An error occurred (AccessDenied) when calling the ListRoles operation: User: arn:aws:iam::123456789012:user/Matt is not authorized to perform: iam:ListRoles on resource: arn:aws:iam::123456789012:role/* with an explicit deny in a service control policy: arn:aws:organizations::987654321098:policy/o-qv5af4abcd/service_control_policy/p-2kgnabcd

How this enhancement works

This enhancement is designed with three principles:

  • Limited scope – Same account and same organization only: Policy ARNs are only included when the request originates from either the same AWS account or the same organization as the policy. This limits the scope of the flow of information.
  • Additional context in the form of ARN only and not policy content: The additional context covers only the policy ARN, which is a resource identifier, not the policy document itself. It does not reveal the policy’s permissions or conditions that you would have to update to grant access. Users would still need appropriate permissions to read the policy content or take actions.
  • No change to authorization logic: This enhancement only affects the error message displayed, not the authorization decision-making process. The same policies deny or allow access as before, and we are not changing how the decision is made.

How this benefits you

This accelerates troubleshooting across your organization. Previously, when you received an access denied error from a policy, for example an SCP, you had to review all SCPs in your organization, determine which applied to the account, and evaluate each one—a process that could take time. Now, with the specific SCP ARN included in the error message, whoever has the necessary permission can review the identified SCP and more quickly resolve the issue. This precision reduces the investigative burden. Clear error messages with policy ARNs also improve communication between teams who need access and teams who troubleshoot issues by providing a common reference point, eliminating ambiguity and reducing back-and-forth communication. Lastly, when validating security controls, the policy ARN in access denied errors provides immediate confirmation of which policy is enforcing the restriction, enabling customers to quickly verify their policies are correctly denying access.

How you can use the new information

Let’s say you’re trying to describe your Amazon Relational Database Service (Amazon RDS) snapshots in the us-east-2 Region by calling this API:
aws rds describe-db-snapshots --region us-east-2

Unfortunately you get an access denied error. The error message shows:
An error occurred (AccessDenied) when calling the DescribeDBSnapshots operation: User: arn:aws:sts::123456789012:assumed-role/ReadOnly/ReadOnlySession is not authorized to perform: rds:DescribeDBSnapshots on resource: arn:aws:rds:us-east-2:123456789012:snapshot:* with an explicit deny in a service control policy: arn:aws:organizations::987654321098:policy/o-qv5af4abcd/service_control_policy/p-lvi9abcd

You can see the context to understand what happens:

  • It’s an explicit deny. This means there’s a policy that denies this action for a specific context
  • The deny comes from the SCP with this ARN: arn:aws:organizations::987654321098:policy/o-qv5af4abcd/service_control_policy/p-lvi9abcd

Here’s how you can troubleshoot this error:

  1. Ensure you have necessary permission to view the SCP. If you don’t, contact your administrator and provide the message that includes the policy ARN.
  2. If you have the necessary permission, go to the AWS Management Console for AWS Organizations to access the SCP.
  3. Check for a Deny statement for the action. In the preceding example, the action is rds:DescribeDBSnapshots.
  4. You can alter the statement to remove the Deny if it’s no longer applicable. For more information, see Update a service control policy (SCP).
  5. Re-try your operation. Repeat the troubleshooting process if you get other access denied errors due to different reasons or policies.

When will this change become available?

This update is gradually rolling out across all AWS services in all AWS Regions, beginning early 2026.

Need more assistance?

If you have any questions or issues, contact AWS Support or your Technical Account Manager (TAM).

Stella Hie

Stella Hie

Stella is a Senior Technical Product Manager for AWS Identity and Access Management (IAM). She specializes in improving developer experience and tooling while maintaining strong security standards. Her work focuses on making IAM straightforward to use and improving the troubleshooting experience for AWS customers. In her free time, she enjoys playing piano and bouldering.

Digital Transformation at Santander: How Platform Engineering is Revolutionizing Cloud Infrastructure

Post Syndicated from Julio Bando original https://aws.amazon.com/blogs/architecture/digital-transformation-at-santander-how-platform-engineering-is-revolutionizing-cloud-infrastructure/

This post is cowritten by Julio Bando from Santander.

Santander faced a significant technical challenge in managing an infrastructure that processes billions of daily transactions across more than 200 critical systems. The expansion into diverse financial services, including investment banking, wealth management, insurance, and payment solutions, had created unprecedented technological complexity, requiring a robust, agile, and scalable infrastructure solution. This raised two main issues. Santander needed to ensure that provisioned services followed established architecture definitions, and they needed to reduce infrastructure provisioning time, which took up to 90 days. This situation demanded intensive operational effort. The solution emerged through an innovative platform engineering initiative called Catalyst, which transformed the bank’s cloud infrastructure and development management. This post analyzes the main cases, benefits, and results obtained with this initiative.

The Catalyst solution

Santander is a global financial services company present in more than 10 countries, with over 160 million customers worldwide. They conceived Catalyst in conjunction with the Platform Strategy Program (PSP), an Amazon Web Services (AWS) program specialized in infrastructure platform design. Implemented through a partnership between AWS Professional Services and Santander, the platform was designed to abstract infrastructure provisioning complexity, standardize architectural compliance, and create a framework that enables new technologies in the bank.

The platform’s in-house frontend was developed as an intuitive developer portal, offering a unified interface for all provisioning and resource management needs. At the platform’s core is the control plane cluster, based on Amazon Elastic Kubernetes Service (Amazon EKS). This cluster is the brain of the operation, orchestrating all components and workflows. Within the cluster, Crossplane plays a fundamental role, acting as a universal resource provisioner that Santander uses to manage resources across multiple cloud providers consistently and declaratively.

The control plane cluster has three components:

  • Data plane claims – Managed by ArgoCD, a continuous delivery tool, the component is responsible for continuous synchronization and deployment of application stacks (integrated sets of cloud resources) and configurations, exploring the GitOps concept.
  • Policies catalog – A central repository of policies ensuring compliance and security across all operations using Open Policy Agent (OPA).
  • Stacks catalog – A library of composite resource definitions and Compositions enabling quick and standardized creation of complex environments.

Santander used this innovative architecture to significantly reduce provisioning time from 90 days to only a few hours and in some cases only minutes. Catalyst brought significant benefits in terms of standardization, security, and governance. The provisioning cycle decreased from 30 days to 2 days, and proof of concept preparation time jumped from 90 days to only 1 hour. The consolidation of over 100 pipelines into a single control plane will further simplify infrastructure management. The following diagram shows the Santander catalyst architecture.

This diagram shows the AWS architecture of Santander's Catalyst platform that provides AI capabilities to teams across the company.

Key platform capabilities

Catalyst’s implementation enabled the creation of strategic workloads demonstrating the platform’s versatility and robustness:

  • Generative AI agents stack – The first success case was implementing a complete stack for AI agents integrating:
  • Modern data platform – One of the most complex workloads implemented through Catalyst was the new data platform, including:
    • Built-in integration with Databricks
    • Data lakes
    • Automated extract, transform, and load (ETL) workflows
    • Integration with centralized data catalog
    • Segregated environments for experimentation. With this implementation, the bank significantly reduces approximately 3,000 monthly tickets related to data experimentation environment provisioning.
  • Cloud process orchestration – creation of a modern process orchestration environment with significant results:
    • Migration of legacy workflows to AWS Step Functions
    • Implementation of retry patterns and error handling
    • Centralized process monitoring

Overall result

This stack reduced AI agent implementation time from 105 days to only 24 hours, eliminating dozens of provisioning tickets per environment. The success of these workloads demonstrates Catalyst’s technical capability and the solution’s versatility in meeting different business needs. Each implementation brought valuable learnings that were incorporated into the platform, creating a virtuous cycle of continuous improvement. The variety of implemented workloads also shows how Catalyst has the potential to be a universal platform, capable of supporting everything from traditional use cases to the most innovative ones involving AI and legacy system modernization. Catalyst’s success wasn’t limited to operational efficiency. The platform also catalyzed a cultural change within Santander, promoting an automation and self-service mindset among development teams. This resulted in faster overall development velocity, more agile teams, and enhanced capability to respond quickly to market changes.

Conclusion

Catalyst represents more than merely a technological tool—it’s a digital transformation enabler that’s redefining cloud development standards at the bank. With the platform, Santander addressed the challenges of a scaled environment and established a solid foundation for continuous innovation and future growth.

With these practical cases, Santander proves that investment in platform engineering solves technical problems and enables new business possibilities, keeping the bank at the forefront of digital transformation in the financial sector.


About the authors

Using Amazon SageMaker Unified Studio Identity center (IDC) and IAM-based domains together

Post Syndicated from Praveen Kumar original https://aws.amazon.com/blogs/big-data/using-amazon-sagemaker-unified-studio-identity-center-idc-and-iam-based-domains-together/

Amazon SageMaker Unified Studio now offers two domain configurations: Amazon SageMaker Unified Studio Identity Center(IDC)-based domains with comprehensive governance features, and Amazon SageMaker Unified Studio IAM-based domains with enhanced developer productivity tools.

In this post, we demonstrate how you can use both of these domain configurations of Amazon SageMaker Unified Studio using AWS Identity and Access Management (IAM) role reuse and attribute-based access control.

How authentication works in each configuration

Amazon SageMaker Unified Studio IDC-based domains authenticate users through AWS Identity and Access Management (IAM) Identity Center with Single Sign-On, preserving individual user identities throughout their sessions. These domains excel in governance with identity-based authorization, fine-grained access controls between users, and comprehensive catalog management featuring formal Publisher/Subscriber (Pub/Sub) data sharing workflows with approval processes—ideal for enterprise environments requiring strong identity management, compliance tracking, and identity-based audit trails.

Amazon SageMaker Unified Studio IAM-based domains authenticate through federated AWS Identity and Access Management (IAM) roles where all users accessing a project share the same role permissions. These domains prioritize developer productivity with modern tools including new serverless Notebooks, Athena Spark integration, the improved interface with vertical navigation, and built-in AI assistance, designed for development teams that need streamlined access and advanced analytics capabilities.

This solution facilitates organizations that are already using IDC-based domains to preserve their existing governance frameworks established in IDC-based domains while unlocking modern development capabilities for their teams through IAM-based domains. If you prefer to use the newly launched IAM-based domains, you can continue to do as well. The choice depends on your company’s needs.

Please note that at the time of writing this blog, IAM-based domains do not support Trusted identity propagation. This solution uses the project execution role to configure data access.

The challenge

Imagine a data steward (Sam) uses the IDC-based domain to define data access policies, manage the data catalog, and approve subscription requests to verify compliance and proper data governance.

On the other hand, a data engineer (Sarah), wants to use IDC-based domain for governance features such as SageMaker catalog and IAM-based domain for the new serverless Notebook to build data pipelines, perform advanced analytics, and accelerate development cycles. Sarah will request access to the data through IDC-based domain, and once access is approved by Sam, Sarah can access this data in serverless notebook available in IAM-based domain.

Solution overview

The integration leverages IAM role reuse, AWS Lake Formation Attribute-Based Access Control (ABAC) and Amazon SageMaker Catalog pub-sub model to automatically carry permissions from the IDC-based domain to the new IAM-based domain. When properly configured, data subscriptions managed through the IDC-based domain’s Pub/Sub model become immediately accessible in IAM-based domain projects, providing a unified data access experience.

The solution we will implement in the post involves creating an IAM-based domain project that is similar to your IDC consumer project (eg same team members, use case) , configuring execution roles, and enabling role reuse. This approach maintains the familiar subscription workflow while extending benefits to the IAM-based domain.The following diagram shows the high-level architecture of how this approach works.

AWS SageMaker data governance workflow diagram showing data engineer Sarah performing data discovery and exploration through SageMaker IDC and IAM domains, with data steward and owner Sam managing approvals via Business Data Catalog, connecting to Polyglot AI Notebook and SQL tools.

The solution architecture consists of:

  • Existing IDC-based domain: Contains producer and consumer projects with established data sharing via Pub/Sub model
  • IAM-based domain: New projects with federated and execution roles configured for modern development tools
  • IAM Identity Center: Manages federated access and permission sets
  • Attribute-Based Access Control: Tags on execution roles enable automatic permission inheritance

The solution provides 2 options: Option 1: IDC-Based Domain project role reuse provides the simplest integration path by directly reusing the existing consumer project IAM role from your IDC-based domain as the execution role in the IAM-based domain. The primary benefits include simplified setup requiring only policy changes (covered later in the blog), reduced administrative overhead with one less role to manage and lower risk of misconfiguration since you’re leveraging proven, existing roles. Choose Option 1 when you want the fastest implementation path, your organization prefers minimal role proliferation, you have well-established IDC-based domain roles that already have data access permissions, or your team has limited IAM expertise and wants to avoid complex tagging configurations.

Option 2: Creating a new execution role for the IAM-based domain project and use attribute-based access control (ABAC) through tagging with the IDC-based domain project ID. The key benefits include enhanced auditability with two distinct roles (one for IDC-based domain, one for IAM-based domain), clear separation showing which domain generated each request in CloudTrail logs, greater flexibility to customize permissions specific to IAM-based domain needs without affecting IDC-based domain operations, and better security isolation between the two domain types. The `AmazonDatazoneProject` tag enables attribute based access control, while maintaining distinct role identities. Choose Option 2 when: your organization requires detailed audit trails distinguishing between domain types, compliance policies mandate separation of concerns between governance and development environments, you want to track and attribute costs separately for each domain, or you need to provide evidence showing which domain (governance vs. development) accessed specific data resources for compliance reporting.

Here is the high-level view of how the identity and domain entities map to each other for both options:

AWS IAM Identity Center integration with Amazon SageMaker diagram showing access flow from IdC Groups through Permission Sets to AWS SSO IAM Roles, connecting to SageMaker domains with two implementation options: Option 1 using identical IAM roles, or Option 2 using project-tagged execution roles

Prerequisites

To follow along with this post, you should have:

For this demonstration, we use a simplified setup with a sales producer project and a marketing consumer project that subscribes to these tables.

Understanding the current IDC-based domain setup

Our starting point includes a well-established Amazon SageMaker Unified Studio IDC-based domain structure:

Sales Producer Project

  • Contains a database with pipeline and sales tables
  • Managed by Sam, the data steward who creates and publishes data assets
  • Has its own project IAM role

Marketing Consumer Project

  • Managed by Sarah, the data engineer who subscribes to published data via IDC domain project
  • Has its own project IAM role
  • Successfully queries subscribed data through the IDC-based domain interface

Each project has an associated IAM role that governs access to data assets, and the Pub/Sub model manages subscription workflows and permissions.

Setting up federated role through permission sets

Federated roles through permission sets are used to authenticate and provide users with console access to IAM-based domains through AWS IAM Identity Center, where all users within a project share the same role permissions. When you assign a permission set, IAM Identity Center creates corresponding IAM Identity Center-controlled IAM role in AWS account, and attaches the policies specified in the permission set to that role.

IAM-based SMUS domains enable streamlined access to modern development tools (serverless Notebooks, Athena Spark, AI assistance) while maintaining governance, automatically propagating permissions across domains without requiring duplicate access approvals, and simplifying team member onboarding.You can use any IAM role to access IAM-based domain. For this post, we will use federated role option using AWS IAM Identity Center (IDC).

Grant access to Data engineer group for IAM-based domains in Identity Center

1) Set up federated role in AWS IAM Identity Center

Navigate to IAM Identity Center (IDC) in the AWS Management Console, then complete the following steps:

  1. Go to permission set section in IDC. Create a new permission set called Marketing-federated-role and select Attach Policy.

AWS IAM Identity Center console screenshot displaying the marketing-federated-role permission set configuration page with provisioned status, 1-hour session duration, and empty AWS managed and customer managed policy sections with attach policy options.

  1. Search for SageMakerStudioUserIAMConsolePolicy in the existing policy name from list and select SageMakerStudioUserIAMConsolePolicy from the list. Note that the managed policy SageMakerStudioUserIAMConsolePolicy must be attached or have the same permissions added via another policy to be able to access projects in a SageMaker IAM domain.

AWS IAM Identity Center console screenshot showing AWS managed policies section with one attached SageMakerStudioUserIAMConsolePolicy and empty customer managed policies section with detach and attach policy options available.

  1. Go to the AWS account section of IDC.
  2. Assign the created permission set to your AWS account.

AWS IAM Identity Center console screenshot showing AWS accounts page in hierarchy view with organization o-9svtz1aavh, displaying Root organizational unit containing AWS account n.com with marketing-federated-role permission set assigned and assign users or groups option.

  1. For this post we assigned the permission set to marketing group, As a best practice, you should setup and grant access to groups rather than individual users.

AWS IAM Identity Center console screenshot showing marketing group details page with AWS accounts tab selected, displaying one AWS account access (management account amazon.com) with marketing-federated-role permission set applied.

  1. Add Sarah to marketing group.

AWS IAM Identity Center console screenshot showing marketing group's Users tab with one enabled member (user sarah, Display name: Sarah M) who inherits permissions to AWS accounts and Identity Center enabled applications.

This creates a federated role that Sarah can use to access the IAM-based domain. The federated role appears as an IAM role within your account and serves as the entry point for console access.

Setting up IAM-based domain execution role

There are 2 options to setup execution role for IAM-based domain project. The execution role has a one-to-one mapping with the federated role.

Option 1 – IDC-based domain Project Role reuse

Instead of creating a new execution role and tagging it, you can configure the IAM-based domain project to directly reuse the consumer project IAM role from the IDC-based domain as the execution role. This option only needs policy changes to the consumer project IAM role. To find the IDC-based domain consumer project IAM role:

  1. Navigate to the Amazon SageMaker Unified Studio IDC-based domain portal.
  2. Open the Marketing Consumer Project.
  3. Copy the project role ARN from the project overview page.

Amazon DataZone project overview page displaying marketing-project details with active status, project ID 4tcycvm4c684rt, domain ID dzd-47supbt0i3jysp, All capabilities profile, Corp domain unit, Amazon S3 location in us-east-2, and project role ARN with up-to-date status.

  1. You will need to modify this execution role’s policy with detailed instructions provided later in the blog.

Setting up IAM-based domain project for option 1

To create an IAM-based domain project that will integrate with your existing IDC-based domain permissions, complete the following steps:

  1. Log in to the AWS Console using IAM-based domain administrator.
  2. Navigate to Amazon SageMaker page within console.
  3. Choose Open.

Amazon SageMaker landing page displaying "The center for data, analytics, and AI" with tagline about next-generation integrated analytics experience, serverless notebooks with built-in AI Agent, Amazon DataZone integration note, and call-to-action panel featuring "Get started with Amazon SageMaker Unified Studio" with Open button and View existing domains

  1. Once logged in to IAM-based domain as admin, choose Manage projects.

Amazon SageMaker admin-project dashboard displaying left navigation menu with data analytics and AI/ML sections, quick-start cards for exploring data, building in notebooks, and discovering ML models, plus four sample data project templates: Customer usage analysis (3 mins), Customer segmentation (8 mins), Customer churn prediction (5 mins), and Retail sales forecasting (20 mins).

  1. Next, click on Create Project.

Amazon DataZone Domain Administration Projects page showing "Projects (3)" with description about enabling IAM role-based access to AWS Analytics and AI/ML tools, search functionality to find projects, last refreshed timestamp, and green Create project button.

  1. Enter project name as “Marketing Consumer Project”.

Amazon DataZone Create project dialog showing Step 1 "Enter Details" with required Project name field containing "Marketing Consumer Project" (1-64 characters, a-z, A-Z, 0-9, spaces, dashes, underscores allowed) and optional Description field with 0/2048 character count, followed by Step 2 "Assign roles".

  1. During project creation, select the following crucial roles and then choose Create Project:
  • Project IAM Role: The marketing federated role created in IAM Identity Center above. This is the role in the member account that has a role name with suffix AWSReservedSSO.
  • Project Role: – Choose project role for data engineer, copied from option 1.

Amazon SageMaker Unified Studio Create project dialog showing IAM role configuration with AWSReservedSSO_marketing-federated-role selected, blue alert requiring SageMakerStudioUserIAMConsolePolicy attachment, Execution role section with "Use an existing role" option selected, and datazone_usr_role_4tcycvm4c684rt_ajtckkwo2fnhyh IAM role specified with note that role is not editable after project creation

  1. Make policy changes to this project role as per the instruction on the SMUS UI page.

Amazon SageMaker Unified Studio role selection interface showing "Use an existing role" option selected with IAM role datazone_usr_role_4tcycvm4c684rt_ajtckkwo2fnhyh, blue information box displaying required permissions including SageMakerStudioUserIAMDefaultExecutionPolicy managed policy, trust policy enabling Amazon SageMaker Unified Studio service assumption, and inline policy for role pass-through, with note that role is not editable after project creation.

Option 2 – Bring your own execution role. 

To create an IAM-based domain project that will integrate with your existing IDC-based domain permissions., you must tag the execution role for permission propagation. Amazon SageMaker Catalog and AWS Lake Formation use attribute-based access control, which means permissions can be inherited based on resource tags. For this option, you will need consumer project ID.To find the IDC-based domain consumer project ID:

  1. Navigate to the Amazon SageMaker Unified Studio IDC-based domain portal.
  2. Open the Marketing Consumer Project.
  3. Copy the project ID from the project details.

Amazon SageMaker Unified Studio marketing-project overview page displaying navigation breadcrumb (Home > Projects > marketing-project > Project overview), left sidebar menu with Project overview, Data, Compute, Members, and Project catalog sections, Project files section listing 3 JupyterLab files (.libs.json, README.md, getting_started.ipynb) last modified November 18, 2025, Readme section with Welcome heading describing SageMaker Unified Studio, and Project details tab showing project name, ID, last modified date November 21, 2025, and Amazon S3 location.” width=”2196″ height=”1164″></p>
<h3>Setting up IAM-based domain project for option 2</h3>
<p>Complete the following steps:</p>
<ol>
<li>Create another project with name “Marketing Consumer Project 2” in the IAM-based domain while logged in as admin.</li>
<li>During project creation, select the following roles:
<ol type=

  • Federated Role: The marketing federated role created in IAM Identity Center above.
  • Execution Role: – Choose execution role from option 2.
  • Make policy changes to this execution role as per the instruction.
  • Amazon SageMaker Unified Studio role selection interface showing "Use an existing role" option selected with IAM role field containing "sagemaker-marketing-execution-role", blue information box displaying required permissions including SageMakerStudioUserIAMDefaultExecutionPolicy managed policy, trust policy enabling Amazon SageMaker Unified Studio and related services to assume the role, and inline policy allowing role pass-through to other services, with note that role is not editable after project creation

    1. Next, navigate to the IAM console and locate the execution role created for your IAM-based domain consumer project.
    2. Add the following tag, this step relies on ABAC policies with projectId for subscriptions.
    • Key: AmazonDatazoneProject
    • Value: The project ID from your Amazon SageMaker Unified Studio IDC-based domain consumer project

    AWS IAM console displaying sagemaker-marketing-execution-role details page with Summary section showing creation date November 18, 2025, last activity 3 days ago, ARN arn:aws:iam::role/sagemaker-marketing-execution-role, 1-hour maximum session duration, five tabs (Permissions, Trust relationships, Tags (1), Last Accessed, Revoke sessions), and Tags section displaying one tag with Key "AmazonDataZoneProject" and Value "4tcycvm4c684rt" with Delete, Edit, and Manage tags buttons available.

    This tag configuration results in data access grant from IDC-based domain consumer project to the IAM-based domain project execution role.

    Verify data access in the IAM-based domain

    After tagging the execution role, verify that permissions are set up correctly.Complete the following steps:

    1. Use the SSO URL to log into the SSO Identity Center as Sarah.

    AWS IAM Identity Center Dashboard displaying left navigation menu with Dashboard, Users, Groups, Settings, Multi-account permissions (AWS accounts, Permission sets), and Application assignments sections; central management panel showing service control policies guidance with yellow warning banner about member account instances and CloudTrail monitoring section; IAM Identity Center setup area with three action cards for confirming identity source, managing multi-account permissions, and setting up application assignments; right panel Settings summary showing Identity Center directory as identity source, us-east-2 region, organization ID o-9svtz1aavh, AWS access portal URL, and issuer URL; What's new section highlighting customer-managed KMS keys support and Amazon SageMaker Studio user background sessions; Related consoles links to CloudTrail, AWS Organizations, and IAM.

    1. Open the AWS console using federated role created earlier in setting federated role section.
    2. Navigate to Amazon SageMaker.
    3. Choose Amazon SageMaker Unified Studio IAM-based domain option (this will show up if project is already created with federated role).

    Amazon SageMaker Unified Studio marketing-project dashboard displaying left navigation menu with Overview, Files, Data, Connections, Code (Notebooks, JupyterLab), Data analytics (Query Editor, Visual ETL, Data processing jobs), and AI/ML sections (Models, MLflow, Training jobs, Inference endpoints); main content area showing "Jump into your data and models" with three quick-start cards (Explore your data, Build in the notebook, Discover ML models) and four sample data projects: Retail sales forecasting (20 mins), Customer churn prediction (5 mins), Customer segmentation (8 mins), and Customer usage analysis (3 mins); top-right panel displaying account details with us-east-2 region, federated user aws-reserved/sarah, and execution role sagemaker-marketing-execution-role.

    1. In the Amazon SageMaker Unified Studio IAM-based domain project, navigate to the Data tab. If you created 2 projects with both option 1 and option 2 execution role, then 2 projects will show up and you can login to either to validate data access.

    Amazon SageMaker Unified Studio data explorer interface displaying SQL query "SELECT * FROM glue_db_6doxdp1wuy165l.sales_table LIMIT 100" executed via Athena in 6 seconds, showing six columns (ord_num, sales_qty_sld, wholesale_cost, lst_pr, sell_pr, disnt) with green distribution histograms above data preview table containing six sample sales records with order numbers ranging from 46776931 to 146776932, left navigation showing AwsDataCatalog database structure with glue_db_6doxdp1wuy165l containing pipeline_table and sales_table, last saved 2 minutes ago.

    1. Verify that the consumer database and subscribed tables appear.

    Create and use the new serverless notebooks

    With permissions properly configured, you can now use IAM-based domain capabilities like serverless Notebooks. Complete the following steps:

    1. In the Amazon SageMaker Unified Studio IAM-based domain project, select a table from the Data tab.
    2. Choose Create notebook.
    3. The Notebook opens with Athena SQL as the default cell type.
    4. Write and run queries against your subscribed data.

    Amazon SageMaker Unified Studio marketing-project notebook displaying sales_table data from 2025-11-18 21:42:01, left Data explorer showing AwsDataCatalog with glue_db_6doxdp1wuyi65l database containing pipeline_table and sales_table, main data table showing 11 rows with columns (ord_num, sales_qty_sld, wholesale_cost, lst_pr, sell_pr, disnt) displaying rows 4-9 on page 1 of 2, Python PySpark SQL query "SELECT * FROM 'glue_db_6doxdp1wuyi65l'.'pipeline_table' LIMIT 100" executed in 27 seconds, and Filters section displaying distribution histograms for all numerical columns.

    The notebook runs with the execution role’s permissions, which now include access to all data subscribed through the IDC-based domain.

    Key benefits of this integration

    This integration approach delivers several important advantages:

    Preserve existing investments

    • Continue using IDC-based domain governance and catalogs.
    • Maintain established Pub/Sub workflows.
    • No migration required for existing data assets.

    Get modern capabilities

    • Provide developers with the new serverless Notebooks.
    • Access Athena Spark for advanced analytics.
    • Provides improved user experience and navigation.

    Simplified permission management

    • Single subscription workflow manages access across both domains.
    • Consistent data access via role reuse and attribute-based access control.
    • No duplicate access requests or approvals needed.

    Unified data experience

    • Developers access all subscribed data from one interface.
    • Consistent data catalog across domains.
    • Simplified onboarding for new team members.

    Cleanup

    Complete the following steps to delete the resources you created:

    1. Delete the serverless Notebooks created in the IAM-based domain projects.
    2. Delete the IAM-based domain projects (Marketing Consumer Project and Marketing Consumer Project 2).
    3. Remove the permission set assignment from marketing group in IAM Identity Center.
    4. Delete the Marketing-federated-role permission set in IAM Identity Center.
    5. Remove the tags (AmazonDatazoneProject) from the execution role (if using Option 2).
    6. Delete the execution role created for the IAM-based domain (if using Option 2 and not reusing the IDC-based domain project role).
    7. Revert any policy changes made to the IDC-based domain consumer project IAM role (if using Option 1).
    8. If you do not need the IAM-based domain anymore, delete it.
    9. If you created any test data subscriptions in the IDC-based domain, remove them.

    Conclusion

    In this post, we demonstrated how to access Amazon SageMaker Unified Studio IDC-based domain with the new IAM-based domain using role reuse and attribute-based access control. This setup offers data engineers the best of both worlds: access to specialized modern development tools—including the new serverless Notebooks, Athena Spark integration, and built-in AI assistance , while maintaining proper governance that includes comprehensive catalog management and robust security controls established in the IDC-based domain.You can now confidently adopt Amazon SageMaker Unified Studio IAM-based domain capabilities knowing their established data governance, subscription workflows, and access controls remain intact and continue to function as expected.

    Ready to get started with Amazon SageMaker Unified Studio and unlock the power of integrated governance and modern development tools for your organization? Visit the Amazon SageMaker Unified Studio documentation to learn more and begin your implementation today.


    About the authors

    Praveen Kumar

    Praveen Kumar

    Praveen is a Principal Analytics Solutions Architect at AWS with expertise in designing, building, and implementing modern data and analytics platforms using cloud-based services. His areas of interest are serverless technology, data governance, and data-driven AI applications.

    Durga Mishra

    Durga Mishra

    Durga is a Principal Data and AI solutions architecture strategist at AWS . Outside of work, Durga enjoys building new things and spending time with family. He loves to hike on Appalachian trails and spend time in nature.

    Joel

    Joel Farvault

    Joel is a Principal Specialist SA Analytics for AWS with 25 years’ experience working on enterprise architecture, data governance and analytics. He uses his experience to advise customers on their data strategy and technology foundations.

    author name

    Satish Sarapuri

    Satish is a Sr. Data Architect for Data Mesh/Data Lake/Gen AI at AWS. He helps enterprise-level customers build generative AI, data mesh, data lake, and analytics platform solutions on AWS to help them make data-driven decisions and gain impactful outcomes for their business. In his spare time, he enjoys trail running and spending quality time with his family.

    author name

    Leonardo Gomez

    Leonardo is a Principal Analytics Specialist Solutions Architect at AWS. He has over a decade of experience in data management, helping customers around the globe address their business and technical needs.

    Sovereign failover – Design for digital sovereignty using the AWS European Sovereign Cloud

    Post Syndicated from Ivo Kammerath original https://aws.amazon.com/blogs/architecture/sovereign-failover-design-for-digital-sovereignty-using-the-aws-european-sovereign-cloud/

    Organizations operating across multiple jurisdictions need to consider the impact of regulatory changes or geopolitical events on their access to cloud infrastructure. This post explains how to design failover architectures that span AWS partitions—including the AWS European Sovereign Cloud, AWS GovCloud (US) and other AWS Regions in the global infrastructure — so workloads can continue operating when sovereignty requirements shift.

    Although the AWS European Sovereign Cloud is designed to help customers with operational autonomy and data residency requirements, it can also be used to address broader geopolitical and sovereignty risks. This post explores the architectural patterns, challenges, and best practices for building cross-partition failover, covering network connectivity, authentication, and governance. By understanding these constraints, you can design resilient cloud-native applications that balance regulatory compliance with operational continuity.

    Understanding sovereignty risks

    Digital sovereignty entails managing digital dependencies — deciding how data, technologies, and infrastructure are used, and reducing the risk of loss of access, control, or connectivity. As with any disaster recovery strategy, there are several means to provide continuity for the systems to be designed. Most of them involve some form of failover architecture, i.e. providing a second set of infrastructure to be used when the disaster incapacitates the original infrastructure. What differs for sovereign disaster recovery are the control mechanics and structures of the target to fail over to. Incorporating the AWS European Sovereign Cloud into your workload design adds failover capabilities that help you to reestablish or maintain enhanced sovereignty if the primary environment becomes unavailable.

    As regulatory requirements evolve, modern failover architectures must account for sovereign environments such as the AWS European Sovereign Cloud, AWS GovCloud (US), and multi-vendor deployments. This post focuses on three core areas for incorporating sovereignty requirements into failover design: failover strategy, network connectivity across isolated partitions, and authentication and authorization in cross-partition architectures. These patterns apply to both short regional outages and long-term partition failures.

    Understanding AWS partitions

    As a global cloud provider, AWS operates multiple infrastructure partitions tailored to meet specific operational and regulatory requirements. In addition to its AWS global infrastructure, AWS offers specialized partitions such as AWS GovCloud for US government agencies, the AWS China Regions, and the AWS European Sovereign Cloud for customers that require stringent data residency and control within the EU.

    Each partition is a logically isolated group of AWS Regions with its own set of resources, including AWS Identity and Access Management (IAM). Because of this separation, partitions act as hard boundaries. Credentials don’t carry over, and services such as Amazon S3 and features like S3 Cross-Region Replication or AWS Transit Gateway inter-region peering cannot function across partitions. These limitations are intentional, providing operational isolation. AWS GovCloud (US), launched in 2011, supports US public sector customers with compliance needs such as FedRAMP and ITAR. The AWS China regions are operated through local partnerships to meet Chinese data sovereignty laws. Similarly, the AWS European Sovereign Cloud is a partition built entirely within the EU, launched in 2026.

    These partitions provide enhanced data control and physical infrastructure isolation, making them essential if you operate in regulated sensitive sectors and need to satisfy strict compliance requirements.

    Key benefits of AWS partitions

    AWS introduced partitions for several reasons. They are key to helping customers meet country-specific compliance and regulatory requirements, whether in AWS GovCloud (US), AWS China, or the AWS European Sovereign Cloud. This is underpinned by multiple safeguards and controls, including physical, logical, and operational separation of the cloud infrastructure between partitions. This directly corresponds to the security aspects of partitions. Partitions allow AWS to provide a complete isolation of resources, which helps manage security, especially for architectures running sensitive workloads.

    Another important point to keep in mind when talking about partitions is service availability. Not all AWS services are available in every partition. To learn more about the AWS services available by Region, refer to AWS Capabilities by Region.

    Cross-partition architectures

    A cross-partition architecture enables partition failover by deploying resources and infrastructure across multiple isolated AWS partitions. Because partitions are fully separated by identity, networking, and service boundaries, failover can’t simply switch between them as within a single partition or region. Instead, environments must be pre-provisioned and kept in sync through internal or external tooling. Without such an architecture, failover between partitions is impractical. Cross-partition architectures make failover possible but require duplicate infrastructure, separate identity systems, and custom data synchronization.

    Figure 1: Different reasons for failover and their possible locations

    When designing cross-Region or cross-partition failover strategies, the choice of Regions depends on the type of disaster you want to mitigate:

    • Natural disasters – select Regions in different geographic zones or with distinct geographic features.
    • Technical disasters – separate workloads across independent parts of the global technical infrastructure, such as power grids, networks, and other shared resources.
    • Human-driven disasters – consider political, socioeconomic, and legal factors that might affect operations.

    Figure 2: Active-active failover scenario including a sovereign failover option

    Partition failover

    Cross-partition workloads arise from industry needs to maintain continuity across sovereign domains while meeting regional regulations. Examples include military and defense connecting specialized clouds (such as AWS GovCloud (US)) with commercial environments, and emergency response systems requiring secure partition isolation combined with unified management (a single pane of glass approach). Control planes managing workloads across partitions are critical for handling multi-tenant structures, enabling centralized metrics, log aggregation, onboarding, security management and more.

    However, cross-partition connections increase operational complexity, security and compliance overhead, costs, and governance challenges. These factors make it important to implement such architectures only when they are truly required. Standard cloud resilience models range from simple backups to multi-site setups, and can be implemented across multiple Availability Zones as well as multiple Regions. The same concept equally applies across multiple partitions. We can move backups into a second partition to be able to recover into that partition. Equally we can run an application pilot light in another partition. This greatly reduces the cost of the infrastructure required in the second partition because it will only be built up when needed. Finally, warm standby or multi-site active-active setups mainly differ in the need for more complex network synchronization across partitions.

    Figure 3: Different types of disaster recovery scenarios

    You might also consider vendor independence as an additional sovereignty requirement when planning failover. One way to achieve vendor independence is to use another cloud provider. However, failing over to another AWS partition is simpler than switching cloud providers because you can reuse your infrastructure as code templates across partitions.

    Reasons to connect partitions

    Although partitions are designed for isolation, some workloads within a partition might need to communicate with workloads in less regulated partitions or with external systems accessible over the public internet. For such instances several architectural strategies and the corresponding architectural decisions should be considered. There might be use cases where you need AWS Services to communicate across partitions and orchestrate actions spanning multiple partitions, such as:

    • Cross-domain applications
    • Feature parity and service availability
    • Cost-optimization while meeting security demands
    • Infrastructure consolidations
    • Control plane patterns

    Implementing these use cases requires a deeper look into the technical aspects of connecting partitions from both a network standpoint and a security standpoint.

    Regional connections vs. connected partitions

    Regional connections let you link AWS Regions within the same partition using features like S3 Cross-Region Replication and Transit Gateway peering, facilitating relatively seamless workload distribution and failover within the partition’s global infrastructure. Understanding the distinction between regional connections and connected partitions is crucial for designing resilient, compliant architectures that meet both operational and regulatory demands.

    Connecting partition networks

    You can connect AWS partitions in three ways: internet connectivity secured by TLS, IPsec Site-to-Site VPN over the internet, or through an AWS Direct Connect gateway to on-premises routers or using Direct Connect point of presence (PoP) partner connections to another Direct Connect PoP. Each approach offers different trade-offs in terms of security complexity and recovery. For more information about connectivity patterns between AWS GovCloud (US) and the global AWS infrastructure, see Connectivity patterns between AWS GovCloud (US) and AWS commercial partition. In addition to the customer gateway solution shown previously, partners located in Direct Connect PoPs can provide cross-partition connectivity services. These services can move traffic from one Direct Connect PoP to another. Such a setup enables dedicated lines between the AWS European Sovereign Cloud Direct Connect PoPs and the Direct Connect locations in other partitions.

    Because IAM credentials don’t work across partitions, you need to create separate roles or use external identity providers. Common approaches include using IAM roles with trust relationships and external IDs, AWS Security Token Service (AWS STS) regional endpoints, resource-based policies, or cross-account roles managed through AWS Organizations. A modern best practice is to federate identities from a single, centralized identity provider to multiple partitions, avoiding the need for IAM users wherever possible. If IAM users are still used, credentials can be stored in AWS Secrets Manager, rotated using Lambda, and a backup user can improve availability. These patterns are often combined with standard access controls, such as Amazon API Gateway with authorizers, to secure cross-partition interactions. For a deeper dive into cross-partition authentication and authorization with AWS IAM, see IAM Identity Center for AWS environments spanning AWS GovCloud (US) and standard Regions.

    When securing communication between AWS partitions, certificate-based approaches present both opportunities and challenges. Because AWS Certificate Manager (ACM) certificates and AWS Private Certificate Authority (AWS Private CA) are bound to individual partitions, you must typically deploy and manage separate public key infrastructure (PKI) infrastructures in each environment, including dedicated root CAs and manual handling of private key transfers. To establish secure cross-partition communication, a more advanced solution involves using double-signed certificates, where root CAs in each partition cross-sign each other’s certificates, creating a bidirectional chain of trust. Implementing this requires setting up root CAs with AWS Certificate Manager Private CA, establishing cross-signing agreements, managing trust stores across partitions, and handling complex certificate validation and revocation checks. You must also comply with differing regulatory requirements and maintain detailed audit trails. Although this approach adds operational complexity, it is essential for enabling authenticated, encrypted communication across isolated partitions, particularly in regulated environments where security and compliance are paramount.

    Managing AWS Organizations across partitions

    Setting up AWS European Sovereign Cloud accounts within your AWS Organization must be done in a completely separate organization. In the AWS GovCloud (US) partition, accounts can be paired into a commercial organization, as described in Inviting Accounts into an Organization for AWS GovCloud. With sovereignty as the main goal, failing over to an AWS European Sovereign Cloud-only state is simpler if the AWS Organizations setup is separate from the start. This doesn’t require starting from scratch. Instead, you can manage the same organizational units (OUs) and policies for the AWS European Sovereign Cloud by reusing your existing deployment automation.

    Ideally, AWS Organizations account structures should be separated to make it straightforward to use the AWS landscape within the AWS European Sovereign Cloud without relying on the other partitions.

    Figure 4: connectivity and service distribution across AWS partitions like the AWS European Sovereign Cloud

    Security controls should be tailored per partition using distinct Service Control Policies (SCPs), with AWS Control Tower managing the commercial side. Networking requires isolated Transit Gateways, separate Amazon Route 53 DNS zones, and secure cross-partition communication using AWS PrivateLink. For monitoring, AWS Config aggregators and AWS Security Hub instances must be configured separately in each partition, while consolidated billing can be managed through Organizations. It’s important to consider limitations (for example, AWS Control Tower can’t directly manage AWS GovCloud (US) or AWS European Sovereign Cloud accounts), and the limited availability of some AWS Organizations features in these partitions. Overall, this approach supports governance, security, and operational clarity across partitions.

    Conclusion

    Navigating sovereignty-driven cloud architectures requires a strategy that addresses partition isolation, network connectivity, and secure cross-partition authentication. Prioritizing sovereignty in failover design adds complexity, but it might be worth the trade-off if your workloads need protection against geopolitical risks or regulatory changes. Start by identifying the disaster scenarios that matter most to your business, then select the simplest architecture that addresses those risks. By designing proactively for evolving regulations, you can maintain both compliance and resilience in the cloud.


    About the authors

    Simplify IAM policy creation with IAM Policy Autopilot, a new open source MCP server for builders

    Post Syndicated from Micah Walter original https://aws.amazon.com/blogs/aws/simplify-iam-policy-creation-with-iam-policy-autopilot-a-new-open-source-mcp-server-for-builders/

    Today, we’re announcing IAM Policy Autopilot, a new open source Model Context Protocol (MCP) server that analyzes your application code and helps your AI coding assistants generate AWS Identity and Access Management (IAM) identity-based policies. IAM Policy Autopilot accelerates initial development by providing builders with a starting point that they can review and further refine. It integrates with AI coding assistants such as Kiro, Claude Code, Cursor, and Cline, and it provides them with AWS Identity and Access Management (IAM) knowledge and understanding of the latest AWS services and features. IAM Policy Autopilot is available at no additional cost, runs locally, and you can get started by visiting our GitHub repository.

    Amazon Web Services (AWS) applications require IAM policies for their roles. Builders on AWS, from developers to business leaders, engage with IAM as part of their workflow. Developers typically start with broader permissions and refine them over time, balancing rapid development with security. They often use AI coding assistants in hopes of accelerating development and authoring IAM permissions. However, these AI tools don’t fully understand the nuances of IAM and can miss permissions or suggest invalid actions. Builders seek solutions that provide reliable IAM knowledge, integrate with AI assistants and get them started with policy creation, so that they can focus on building applications.

    Create valid policies with AWS knowledge
    IAM Policy Autopilot addresses these challenges by generating identity-based IAM policies directly from your application code. Using deterministic code analysis, it creates reliable and valid policies, so you spend less time authoring and debugging permissions. IAM Policy Autopilot incorporates AWS knowledge, including published AWS service reference implementation, to stay up to date. It uses this information to understand how code and SDK calls map to IAM actions and stays current with the latest AWS services and operations.

    The generated policies provide a starting point for you to review and scope down to implement least privilege permissions. As you modify your application code—whether adding new AWS service integrations or updating existing ones—you only need to run IAM Policy Autopilot again to get updated permissions.

    Getting started with IAM Policy Autopilot
    Developers can get started with IAM Policy Autopilot in minutes by downloading and integrating it with their workflow.

    As an MCP server, IAM Policy Autopilot operates in the background as builders converse with their AI coding assistants. When your application needs IAM policies, your coding assistants can call IAM Policy Autopilot to analyze AWS SDK calls within your application and generate required identity-based IAM policies, providing you with necessary permissions to start with. After permissions are created, if you still encounter Access Denied errors during testing, the AI coding assistant invokes IAM Policy Autopilot to analyze the denial and propose targeted IAM policy fixes. After you review and approve the suggested changes, IAM Policy Autopilot updates the permissions.

    You can also use IAM Policy Autopilot as a standalone command line interface (CLI) tool to generate policies directly or fix missing permissions. Both the CLI tool and the MCP server provide the same policy creation and troubleshooting capabilities, so you can choose the integration that best fits your workflow.

    When using IAM Policy Autopilot, you should also understand the best practices to maximize its benefits. IAM Policy Autopilot generates identity-based policies and doesn’t create resource-based policies, permission boundaries, service control policies (SCPs) or resource control policies (RCPs). IAM Policy Autopilot generates policies that prioritize functionality over minimal permissions. You should always review the generated policies and refine if necessary so they align with your security requirements before deploying them.

    Let’s try it out
    To set up IAM Policy Autopilot, I first need to install it on my system. To do so, I just need to run a one-liner script:

    curl https://github.com/awslabs/iam-policy-autopilot/raw/refs/heads/main/install.sh | bash

    Then I can follow the instructions to install any MCP server for my IDE of choice. Today, I’m using Kiro!

    In a new chat session in Kiro, I start with a straightforward prompt, where I ask Kiro to read the files in my file-to-queue folder and create a new AWS CloudFormation file so I can deploy the application. This folder contains an automated Amazon Simple Storage Service (Amazon S3) file router that scans a bucket and sends notifications to Amazon Simple Queue Service (Amazon SQS) queues or Amazon EventBridge based on configurable prefix-matching rules, enabling event-driven workflows triggered by file locations.

    The last part asks Kiro to make sure I’m including necessary IAM policies. This should be enough to get Kiro to use the IAM Policy Autopilot MCP server.

    Next, Kiro uses the IAM Policy Autopilot MCP server to generate a new policy document, as depicted in the following image. After it’s done, Kiro will move on to building out our CloudFormation template and some additional documentation and relevant code files.

    IAM Policy Autopilot

    Finally, we can see our generated CloudFormation template with a new policy document, all generated using the IAM Policy Autopilot MCP server!

    IAM Policy Autopilot

    Enhanced development workflow
    IAM Policy Autopilot integrates with AWS services across multiple areas. For core AWS services, IAM Policy Autopilot analyzes your application’s usage of services such as Amazon S3, AWS Lambda, Amazon DynamoDB, Amazon Elastic Compute Cloud (Amazon EC2), and Amazon CloudWatch Logs, then generates necessary permissions your code needs based on the SDK calls it discovers. After the policies are created, you can copy the policy directly into your CloudFormation template, AWS Cloud Development Kit (AWS CDK) stack, or Terraform configuration. You can also prompt your AI coding assistants to integrate it for you.

    IAM Policy Autopilot also complements existing IAM tools such as AWS IAM Access Analyzer by providing functional policies as a starting point, which you can then validate using IAM Access Analyzer policy validation or refine over time with unused access analysis.

    Now available
    IAM Policy Autopilot is available as an open source tool on GitHub at no additional cost. The tool currently supports Python, TypeScript, and Go applications.

    These capabilities represent a significant step forward in simplifying the AWS development experience so builders of different experience levels can develop and deploy applications more efficiently.

    Introducing attribute-based access control for Amazon S3 general purpose buckets

    Post Syndicated from Matheus Guimaraes original https://aws.amazon.com/blogs/aws/introducing-attribute-based-access-control-for-amazon-s3-general-purpose-buckets/

    As organizations scale, managing access permissions for storage resources becomes increasingly complex and time-consuming. As new team members join, existing staff changes roles, and new S3 buckets are created, organizations must constantly update multiple types of access policies to govern access across their S3 buckets. This challenge is especially pronounced in multi-tenant S3 environments where administrators must frequently update these policies to control access across shared datasets and numerous users.

    Today we’re introducing attribute-based access control (ABAC) for Amazon Simple Storage Service (S3) general purpose buckets, a new capability you can use to automatically manage permissions for users and roles by controlling data access through tags on S3 general purpose buckets. Instead of managing permissions individually, you can use tag-based IAM or bucket policies to automatically grant or deny access based on tags between users, roles, and S3 general purpose buckets. Tag-based authorization makes it easy to grant S3 access based on project, team, cost center, data classification, or other bucket attributes instead of bucket names, dramatically simplifying permissions management for large organizations.

    How ABAC works
    Here’s a common scenario: as an administrator, I want to give developers access to all S3 buckets meant to be used in development environments.

    With ABAC, I can tag my development environment S3 buckets with a key-value pair such as environment:development and then attach an ABAC policy to an AWS Identity and Access Management (IAM) principal that checks for the same environment:development tag. If the bucket tag matches the condition in the policy, the principal is granted access.

    Let’s see how this works.

    Getting started
    First, I need to explicitly enable ABAC on each S3 general purpose bucket where I want to use tag-based authorization.

    I navigate to the Amazon S3 console, select my general purpose bucket then navigate to Properties where I can find the option to enable ABAC for this bucket.

    I can also use the AWS Command Line Interface (AWS CLI) to enable it programmatically by using the new PutBucketAbac API. Here I am enabling ABAC on a bucket called my-demo-development-bucket located in the US East (Ohio) us-east-2 AWS Region.

    aws s3api put-bucket-abac --bucket my-demo-development-bucket abac-status Status=Enabled --region us-east-2

    Alternatively, if you use AWS CloudFormation, you can enable ABAC by setting the AbacStatus property to Enabled in your template.

    Next, let’s tag our S3 general purpose bucket. I add an environment:development tag which will become the criteria for my tag-based authorization.

    Now that my S3 bucket is tagged, I’ll create an ABAC policy that verifies matching environment:development tags and attach it to an IAM role called dev-env-role. By managing developer access to this role, I can control permissions to all development environment buckets in a single place.

    I navigate to the IAM console, choose Policies, and then Create policy. In the Policy editor, I switch to JSON view and create a policy that allows users to read, write and list S3 objects, but only when they have a tag with a key of “environment” attached and its value matches the one declared on the S3 bucket. I give this policy the name of s3-abac-policy and save it.

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket"
                ],
                "Resource": [
                    "*"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:ResourceTag/environment": "development"
                    }
                }
            }
        ]
    }

    I then attach this s3-abac-policy to the dev-env-role.

    That’s it! Now a user assuming the dev-role can access any ABAC-enabled bucket with the tag environment:development such as my-demo-development-bucket.

    Using your existing tags
    Keep in mind that although you can use your existing tags for ABAC, because these tags will now be used for access control, we recommend reviewing your current tag setup before enabling the feature. This includes reviewing your existing bucket tags and tag-based policies to prevent unintended access, and updating your tagging workflows to use the standard TagResource API (since enabling ABAC on your buckets will block the use of the PutBucketTagging API). You can use AWS Config to check which buckets have ABAC enabled and review your usage of PutBucketTagging API in your application using AWS Cloudtrail management events.

    Additionally, the same tags you use for ABAC can also serve as cost allocation tags for your S3 buckets. Activate them as cost allocation tags in the AWS Billing Console or through APIs, and your AWS Cost Explorer and Cost and Usage Reports will automatically organize spending data based on these tags.

    Enforcing tags on creation
    To help standardize access control across your organization, you can now enforce tagging requirements when buckets are created through service control policies (SCPs) or IAM policies using the aws:TagKeys and aws:RequestTag condition keys. Then you can enable ABAC on these buckets to provide consistent access control patterns across your organization. To tag a bucket during creation you can add the tags to your CloudFormation templates or provide them in the request body of your call to the existing S3 CreateBucket API. For example, I could enforce a policy for my developers to create buckets with the tag environment=development so all my buckets are tagged accurately for cost allocation. If I want to use the same tags for access control, I can then enable ABAC for these buckets.

    Things to know

    With ABAC for Amazon S3, you can now implement scalable, tag-based access control across your S3 buckets. This feature makes writing access control policies simpler, and reduces the need for policy updates as principals and resources come and go. This helps you reduce administrative overhead while maintaining strong security governance as you scale.

    Attribute-based access control for Amazon S3 general purpose buckets is available now through the AWS Management Console, API, AWS SDKs, AWS CLI, and AWS CloudFormation at no additional cost. Standard API request rates apply according to Amazon S3 pricing. There’s no additional charge for tag storage on S3 resources.

    You can use AWS CloudTrail to audit access requests and understand which policies granted or denied access to your resources.

    You can also use ABAC with other S3 resources such as S3 directory bucket, S3 access points and S3 tables buckets and tables. To learn more about ABAC on S3 buckets see the Amazon S3 User Guide.

    You can use the same tags you use for access control for cost allocation as well. You can activate them as cost allocation tags through the AWS Billing Console or APIs. Check out the documentation for more details on how to use cost allocation tags.

    Simplify access to external services using AWS IAM Outbound Identity Federation

    Post Syndicated from Donnie Prakoso original https://aws.amazon.com/blogs/aws/simplify-access-to-external-services-using-aws-iam-outbound-identity-federation/

    When building applications that span multiple cloud providers or integrate with external services, developers face a persistent challenge: managing credentials securely. Traditional approaches require storing long-term credentials like API keys and passwords, creating security risks and operational overhead.

    Today, we’re announcing a new capability called AWS Identity and Access Management (IAM) outbound identity federation that customers can use to securely federate their Amazon Web Services (AWS) identities to external services without storing long-term credentials. You can now use short-lived JSON Web Tokens (JWTs) to authenticate your AWS workloads with a wide range of third-party providers, software-as-a-service (SaaS) platforms and self-hosted applications.

    This feature enables IAM principals—such as IAM roles and users—to obtain cryptographically signed JWTs that assert their AWS identity. External services, such as third-party providers, SaaS platforms, and on-premises applications, can verify the token’s authenticity by validating its signature. Upon successful verification, you can securely access the external service.

    How it works
    With IAM outbound identity federation, you exchange your AWS IAM credentials for short-lived JWTs. This mitigates the security risks associated with long-term credentials while enabling consistent authentication patterns.

    Let’s walk through a scenario where your application running on AWS needs to interact with an external service. To access the external service’s APIs or resources, your application calls the AWS Security Token Service (AWS STS) `GetWebIdentityToken` API to obtain a JWT.

    The following diagram shows this flow:

    1. Your application running on AWS requests a token from AWS STS by calling the GetWebIdentityToken API. The application uses its existing AWS credentials obtained from the underlying platform (such as Amazon EC2 instance profiles, AWS Lambda execution roles, or other AWS compute services) to authenticate this API call.
    2. AWS STS returns a cryptographically signed JSON Web Token (JWT) that asserts the identity of your application.
    3. Your application sends the JWT to the external service for authentication.
    4. The external service fetches the verification keys from the JSON Web Key Set (JWKS) endpoint to verify the token’s authenticity.
    5. The external service validates the JWT’s signature using these verification keys and confirms the token is authentic and was issued by AWS.
    6. After successful verification, the external service exchanges the JWT for its own credentials. Your application can then use these credentials to perform its intended operations.

    Setting up AWS IAM outbound identity federation
    To begin using this feature, I need to enable outbound identity federation for my AWS account. I navigate to IAM and choose Account settings under Access management in the left-hand navigation pane.

    After I enable the feature, AWS generates a unique issuer URL for my AWS account that hosts the OpenID Connect (OIDC) discovery endpoints at /.well-known/openid-configuration and /.well-known/jwks.json. The OpenID Connect (OIDC) discovery endpoints contain the keys and metadata necessary for token verification.

    Next, I need to configure IAM permissions. My IAM principal (role or user) must have the sts:GetWebIdentityToken permission to request tokens.

    For example, the following identity policy specifies access to the STS GetWebIdentityToken API, enabling the IAM principal to generate tokens.

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Action": "sts:GetWebIdentityToken",
          "Resource": "*",
        }
      ]
    }
    

    At this stage, I need to configure the external service to trust and accept tokens issued by my AWS account. The specific steps vary by service, but generally involve:

    1. Registering my AWS account issuer URL as a trusted identity provider
    2. Configuring which claims to validate (audience, subject patterns)
    3. Mapping token claims to permissions in the external service

    Let’s get started
    Now, let me walk you through an example showing both the client-side token generation and server-side verification process.

    First, I call the STS GetWebIdentityToken API to obtain a JWT that asserts my AWS identity. When calling the API, I can specify the intended audience, signing algorithm, and token lifetime as request parameters.

    • Audience: Populates the `aud` claim in the JWT, identifying the intended recipient of the token (for example, “my-app”)
    • DurationSeconds: The token lifetime in seconds, ranging from 60 seconds (1 minute) to 3600 seconds (1 hour), with a default of 600 seconds (5 minutes)
    • SigningAlgorithm: Choose either ES384 (ECDSA using P-384 and SHA-384) or RS256 (RSA using SHA-256)
    • Tags (optional): An array of key-value pairs that appear as custom claims in the token, which you can use to include additional context that enables external services to implement fine-grained access control

    Here’s an example of getting an identity token using the AWS SDK for Python (Boto3). I can also do this using AWS Command Line Interface (AWS CLI).

    
    import boto3
    
    sts_client = boto3.client('sts')
    response = sts_client.get_web_identity_token(
        Audience=['my-app'],
        SigningAlgorithm='ES384',  # or 'RS256'
        DurationSeconds=300
    )
    jwt_token = response['IdentityToken']
    print(jwt_token)
    
    

    This returns a signed JWT that I can inspect using any JWT parser.

    {
    eyJraWQiOiJFQzM4NF8wIiwidHlwIjoiSldUIiwiYWxnIjoiRVMzODQifQ.hey<REDACTED FOR BREVITY>...
    

    I can decode the token using any JWT parser like this JWT Debugger. The token header shows it’s signed with ES384 (ECDSA).

    
    {
      "kid": "EC384_0",
      "typ": "JWT",
      "alg": "ES384"
    }
    

    Also, the payload contains standard OIDC claims plus AWS specific metadata. The standard OIDC claims include subject (“sub”), audience (“aud”), issuer (“iss”), and others.

    {
      "aud": "my-app",
      "sub": "arn:aws:iam::ACCOUNT_ID:role/MyAppRole",
      "https://sts.amazonaws.com/": {
        "aws_account": "ACCOUNT_ID",
        "source_region": "us-east-1",
        "principal_id": "arn:aws:iam::ACCOUNT_ID:role/MyAppRole"
      },
      "iss": "https://abc12345-def4-5678-90ab-cdef12345678.tokens.sts.global.api.aws",
      "exp": 1759786941,
      "iat": 1759786041,
      "jti": "5488e298-0a47-4c5b-80d7-6b4ab8a4cede"
    }
    

    AWS STS also enriches the token with identity-specific claims (such as account ID, organization ID, and principal tags) and session context. These claims provide information about the compute environment and session where the token request originated. AWS STS automatically includes these claims when applicable based on the requesting principal’s session context. You can also add custom claims to the token by passing request tags to the API call. To learn more about claims provided in the JWT, visit the documentation page.

    Note the iss (issuer) claim. This is your account-specific issuer URL that external services use to verify that the token originated from a trusted AWS account. External services can verify the JWT by validating its signature using AWS’s verification keys available at a public JSON Web Key Set (JWKS) endpoint hosted at the /.well-known/jwks.json endpoint of the issuer URL.

    Now, let’s look at how external services handle this identity token.

    Here’s a snippet of Python example that external services can use to verify AWS tokens:

    
    import json
    import jwt
    import requests
    from jwt import PyJWKClient
    
    # Trusted issuers list - obtained from EnableOutboundFederation API response
    TRUSTED_ISSUERS = [
        "https://EXAMPLE.tokens.sts.global.api.aws",
        # Add your trusted AWS account issuer URLs here
        # Obtained from EnableOutboundFederation API response
    ]
    
    def verify_aws_jwt(token, expected_audience=None):
        """Verify an AWS IAM outbound identity federation JWT"""
        try:
            # Get issuer from token
            unverified_payload = jwt.decode(token, options={"verify_signature": False})
            issuer = unverified_payload.get('iss')
    
     	# Verify issuer is trusted
            if not TRUSTED_ISSUERS or issuer not in TRUSTED_ISSUERS:
                raise ValueError(f"Untrusted issuer: {issuer}")
    
            # Fetch JWKS from AWS using PyJWKClient
            jwks_client = PyJWKClient(f"{issuer}/.well-known/jwks.json")
            signing_key = jwks_client.get_signing_key_from_jwt(token)
    
            # Verify token signature and claims
            decoded_token = jwt.decode(
                token,
                signing_key.key,
                algorithms=["ES384", "RS256"],
                audience=expected_audience,
                issuer=issuer
            )
            return decoded_token
        except Exception as e:
            print(f"Token verification failed: {e}")
            return None
    
    

    Using IAM policies to control access to token generation
    An IAM principal (such as a role or user) must have the sts:GetWebIdentityToken permission in their IAM policies to request tokens for authentication with external services. AWS account administrators can configure this permission in all relevant AWS policy types such as identity policies, service control policies (SCPs), resource control policies (RCPs), and virtual private cloud endpoint (VPCE) policies to control which IAM principals in their account can generate tokens.

    Additionally, administrators can use the new condition keys to specify signing algorithms (sts:SigningAlgorithm), permitted token audiences (sts:IdentityTokenAudience), and maximum token lifetimes (sts:DurationSeconds). To learn more about the condition keys, visit IAM and STS Condition keys documentation page.

    Additional things to know
    Here are key details about this launch:

    Get started with AWS IAM outbound identity federation by visiting AWS IAM console and enabling the feature in your AWS account. For more information, visit Federating AWS Identities to External Services documentation page.

    Happy building!
    — Donnie

    Simplified developer access to AWS with ‘aws login’

    Post Syndicated from Shreya Jain original https://aws.amazon.com/blogs/security/simplified-developer-access-to-aws-with-aws-login/

    Getting credentials for local development with AWS is now simpler and more secure. A new AWS Command Line Interface (AWS CLI) command, aws login, lets you start building immediately after signing up for AWS without creating and managing long-term access keys. You use the same sign-in method you already use for the AWS Management Console.

    In this blog, we’ll show you how to get temporary credentials to your workstation for use with the AWS CLI, AWS Software Development Kits (AWS SDKs), and tools or applications built using them with the new aws login command.

    Getting started with programmatic access to AWS

    You can use the aws login command with your AWS Management Console sign-in method, as described in the following sections.

    Scenario 1: Using IAM credentials (root or IAM user)

    To obtain programmatic credentials using your root or IAM user username and password:

    1. Install the latest AWS CLI (version 2.32.0 or later).
    2. Run the aws login command.
    3. If you have not set a default Region, the CLI prompts you to specify the AWS Region of your choice (e.g., us-east-2, eu-central-1). The CLI remembers which Region you set once you enter it into this prompt.
      Figure 1: CLI Region prompt

      Figure 1: CLI Region prompt

    4. The CLI opens your default browser.
    5. Follow the instructions in the browser window:
      1. If you have already signed into the AWS Management Console, you will see a screen that says, “Continue with an active session.”
        Figure 2: Sign in to AWS - active session selection

        Figure 2: Sign in to AWS – active session selection

      2. If you haven’t signed into the AWS Management Console, you will see the sign-in options page. Select “Continue with Root or IAM user” and log in to your AWS account.
        Figure 3: AWS Sign in to AWS - Sign-in options

        Figure 3: AWS Sign in to AWS – Sign-in options

    6. Success! You’re ready to run AWS CLI commands. Try the aws sts get-caller-identity command to verify the identity you’re currently using.
      Figure 4: Sign in to AWS - completion

      Figure 4: Sign in to AWS – completion

    Scenario 2: Using federated sign-in

    This scenario applies when you authenticate through your organization’s identity provider. To retrieve programmatic credentials for roles you assumed with federation:

    1. Complete steps 1–4 from Scenario 1, then continue with the following instructions.
    2. Follow the instructions in the browser window:
      1. If you have already signed into the AWS Management Console, the browser provides you with the option to select your active IAM role session from federated sign-in to the console. This enables you to switch between 5 active AWS sessions if you have multi-session support enabled on your AWS Management Console.
        Figure 5: Sign in to AWS - active IAM role session selection

        Figure 5: Sign in to AWS – active IAM role session selection

      2. If you have not signed into the AWS Management Console or want to get temporary credentials for a different IAM role, sign into your AWS account using your current authentication mechanism in another browser tab. Upon successful login, switch back to this tab and select the “Refresh” button. Your console session should now be available under the active sessions.
    3. Return to the AWS CLI once you have successfully completed the aws login process.

    Regardless of the console sign-in method you choose, the temporary credentials issued by the aws login command are automatically rotated by the AWS CLI, AWS Tools for PowerShell and AWS SDKs every 15 minutes. They are valid up to the set session duration of the IAM principal (maximum of 12 hours). After reaching the session duration limit, you will be prompted to log in again.

    Figure 6: AWS Sign in - session expiration

    Figure 6: AWS Sign in – session expiration

    Accessing AWS using local developer tools

    The aws login command supports switching between multiple AWS accounts and roles using profiles. You can configure a profile with aws login --profile <PROFILE_NAME> and run AWS commands with the profile using: aws sts get-caller-identity --profile <PROFILE_NAME>. The short-term credentials issued by aws login work with more than the AWS CLI. You can also use them with:

    • AWS SDKs: If you use AWS SDKs for development, the SDK clients can use these temporary credentials to authenticate with AWS.
    • AWS Tools for PowerShell: Use the Invoke-AWSLogin command.
    • Remote development servers: Use aws login --remote on a remote server without browser access, to deliver temporary credentials from your device with browser access to the AWS console.
    • Older versions of AWS SDKs that do not support the new console credentials provider: Any software written using these older SDKs can support credentials delivered by aws login by using the credential_process provider with the AWS CLI.

    Controlling access to aws login with IAM policies

    The aws login command is controlled by two IAM actions: signin:AuthorizeOAuth2Access and signin:CreateOAuth2Token. Use the SignInLocalDevelopmentAccess managed policy or add these actions to your IAM policies to allow IAM users and IAM roles with console access to use this feature.

    AWS Organizations customers looking to control the usage of this login feature on member accounts can deny the two actions above using Service Control Policies (SCPs). These IAM actions and their resources are usable in all relevant IAM policies.

    AWS recommends using centralized root access management in AWS Organizations to eliminate long-term root credentials from member accounts. This feature allows security teams to perform privileged tasks through short-term, task-scoped root sessions from a central management account. After you enable centralized root management and delete root credentials on member accounts, root login to member accounts is denied, which also prevents programmatic access with root credentials using aws login. For developers using root credentials or IAM users, aws login delivers short-lived credentials to development tools, providing a secure alternative to long-term static access keys.

    Logging and security of programmatic access using aws login

    AWS Sign-In logs API activity through AWS CloudTrail, which now includes two new events specific to aws login. The service logs two new event names called AuthorizeOAuth2Access and CreateOauth2Token in the AWS Region where the user logs in.

    Here’s a CloudTrail sample for an AuthorizeOAuth2Access event:

    {
        "eventVersion": "1.11",
        "userIdentity": {
            "type": "AssumedRole",
            "principalId": "AROATJHQDX737YZP72NTF:testuser”,
            "arn": "arn:aws:sts::225989345271:assumed-role/Admin/testuser,
            "accountId": “111111111111”,
            "sessionContext": {
                "sessionIssuer": {
                    "type": "Role",
                    "principalId": "AROATJHQDX737YZP72NTF",
                    "arn": "arn:aws:iam::111111111111:role/Admin",
                    "accountId": “11111111111”,
                    "userName": "Admin"
                },
                "attributes": {
                    "creationDate": "2025-11-17T22:50:14Z",
                    "mfaAuthenticated": "false"
                }
            }
        },
        "eventTime": "2025-11-17T22:51:32Z",
        "eventSource": "signin.amazonaws.com",
        "eventName": "AuthorizeOAuth2Access",
        "awsRegion": "us-east-1",
        "sourceIPAddress": “192.0.2.2”,
        "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36",
        "requestParameters": {
            "scope": "openid",
            "redirect_uri": "http://127.0.0.1:53037/oauth/callback",
            "code_challenge_method": "SHA-256",
            "client_id": "arn:aws:signin:::devtools/same-device"
        },
        "responseElements": null,
        "additionalEventData": {
            "success": "true",
            "x-amzn-vpce-id": ""
        },
        "requestID": "e2854c76-1cba-4360-9fd1-5037b591466b",
        "eventID": "59e1720d-3deb-44ff-933d-6828be2a860a",
        "readOnly": true,
        "eventType": "AwsApiCall",
        "managementEvent": true,
        "recipientAccountId": “111111111111”,
        "eventCategory": "Management",
        "tlsDetails": {
            "tlsVersion": "TLSv1.3",
            "cipherSuite": "TLS_AES_128_GCM_SHA256",
            "clientProvidedHostHeader": "us-east-1.signin.aws.amazon.com"
        }
    }
    

    Here’s a CloudTrail sample for a CreateOAuth2Token event:

    {
        "eventVersion": "1.11",
        "userIdentity": {
            "type": "AssumedRole",
            "principalId": "AROATJHQDX737YZP72NTF:testuser-Isengard",
            "arn": "arn:aws:sts::111111111111:assumed-role/Admin/testuser-Isengard",
            "accountId": "111111111111",
            "sessionContext": {
                "sessionIssuer": {
                    "type": "Role",
                    "principalId": "AROATJHQDX737YZP72NTF",
                    "arn": "arn:aws:iam::111111111111:role/Admin",
                    "accountId": "111111111111",
                    "userName": "Admin"
                },
                "attributes": {
                    "creationDate": "2025-11-18T20:38:10Z",
                    "mfaAuthenticated": "false"
                }
            }
        },
        "eventTime": "2025-11-18T20:38:44Z",
        "eventSource": "signin.amazonaws.com",
        "eventName": "CreateOAuth2Token",
        "awsRegion": "us-east-1",
        "sourceIPAddress": "192.0.2.2",
        "userAgent": "aws-cli/2.32.0 md/awscrt#0.28.4 ua/2.1 os/macos#24.6.0 md/arch#arm64 lang/python#3.13.9 md/pyimpl#CPython m/b,AA,Z,E cfg/retry-mode#standard md/installer#exe sid/35033f4ca1bd md/prompt#off md/command#login",
        "requestParameters": {
            "client_id": "arn:aws:signin:::devtools/same-device"
        },
        "responseElements": null,
        "additionalEventData": {
            "success": "true",
            "x-amzn-vpce-id": ""
        },
        "requestID": "94562943-c85b-4dc1-bf72-43b0fd42d6de",
        "eventID": "0b338fac-6a10-4740-b34d-1bb6923e799e",
        "readOnly": true,
        "eventType": "AwsApiCall",
        "managementEvent": true,
        "recipientAccountId": "111111111111",
        "eventCategory": "Management",
        "tlsDetails": {
            "tlsVersion": "TLSv1.3",
            "cipherSuite": "TLS_AES_128_GCM_SHA256",
            "clientProvidedHostHeader": "us-east-1.signin.aws.amazon.com"
        }
    }
    

    The aws login command uses the OAuth 2.0 authorization code flow with PKCE (Proof Key for Code Exchange) to protect against authorization code interception attacks. This provides a secure alternative to setting up IAM user access keys for getting started with development on AWS. For guidance on additional modern authentication approaches and alternatives to long-term IAM access keys, see the AWS Security Blog post “Beyond IAM access keys: Modern authentication approaches for AWS.”

    Conclusion

    The login for AWS local development feature is a secure-by-default enhancement that helps customers eliminate the use of long-term credentials for programmatic access with AWS. With aws login, you can start building immediately using the same credentials you use to sign in to the AWS Management Console. This feature is now available across all AWS commercial Regions (excluding China and GovCloud) at no additional cost to customers.

    For more information, visit the authentication and access section in the CLI user guide.

    If you have feedback about this post, submit comments in the Comments section below.

    Shreya Jain

    Shreya Jain

    Shreya is a Senior Technical Product Manager in AWS Identity. She is energized by bringing clarity and simplicity to complex ideas. When she’s not applying her creative energy at work, you’ll find her at Pilates, dancing, or discovering her next favorite coffee shop.

    Sowjanya Rajavaram

    Sowjanya Rajavaram

    Sowjanya is a Sr Solutions Architect who specializes in Identity and Security in AWS. She works on helping customers of all sizes solve their identity and access management problems. She enjoys traveling and exploring new cultures and food.

    Controlling AWS API Calls from Amazon Q Developer: Enterprise Governance with Built-in User Agent Markers

    Post Syndicated from Kirankumar Chandrashekar original https://aws.amazon.com/blogs/devops/controlling-aws-api-calls-from-amazon-q-developer-enterprise-governance-with-built-in-user-agent-markers/

    As organizations increasingly adopt AI-powered development tools, a critical challenge emerges: how do you maintain security governance when AI assistants execute AWS operations on behalf of users? Organizations want to leverage AI assistance for development and read operations while maintaining strict controls over write operations that impact production systems and auditing calls made via AI assistants. Consider this scenario: A developer asks Amazon Q Developer “List my S3 buckets”, Q Developer suggests aws s3 ls, the developer approves, and Q Developer executes the command via AWS CLI. From an AWS perspective, this looks identical to the developer manually running the aws s3 ls command on the terminal outside of Amazon Q Developer. But what if your organization needs to distinguish between AI-assisted operations and manual commands for governance or compliance?

    Amazon Q Developer, the most capable generative AI–powered assistant for software development, generates AWS CLI commands in response to user requests and executes them using its use_aws and execute_bash built-in tools. The challenge of distinguishing AI-assisted operations from manual commands is a key consideration for Amazon Q Developer adoption in enterprise environments. To address this governance challenge, Amazon Q Developer includes a built-in solution: user-agent markers that automatically identify AWS CLI calls made through Q Developer in CloudTrail logs, enabling precise IAM policy controls.

    This blog post explores how Amazon Q Developer’s built-in user agent markers set for AWS CLI calls enable precise IAM policy controls, allowing organizations to distinguish and govern AI-assisted AWS operations while maintaining the productivity benefits of AI-powered development. The following sections demonstrate how these user agent markers work, how to implement IAM policies that leverage them, and how to monitor their effectiveness in your environment.

    Understanding Amazon Q Developer User Agent Markers

    Prerequisites

    This section builds on your knowledge of these concepts and assumes you have the necessary setup in place. These foundational elements are essential for understanding how user agent markers work and for implementing the governance controls discussed later in this post. If you need guidance on any of these topics, please refer to the linked documentation:

    Amazon Q Developer automatically includes identifiable markers in the user agent string of all AWS API calls it makes via AWS CLI. These markers appear in two primary contexts: CLI tool operations and IDE integration operations.

    Q Developer CLI Tool

    When using Amazon Q Developer CLI (both use_aws and execute_bash tools), all AWS CLI calls include:

    exec-env/AmazonQ-For-CLI-Version-<QCLI-VersionNo>

    How It Works: Amazon Q Developer CLI automatically sets:

    AWS_EXECUTION_ENV=AmazonQ-For-CLI-Version-<QCLI-VersionNo>

    This means all AWS CLI commands executed through Q Developer CLI – whether via the use_aws tool or execute_bash commands – automatically include this marker.

    Q Developer IDE Integration

    When using Amazon Q Developer from IDE integrations, AWS CLI calls include:

    exec-env/AmazonQ-For-IDE-Version-<QIDE-Plugin-VersionNo>

    How It Works: Amazon Q Developer IDE plugin automatically sets:

    AWS_EXECUTION_ENV=AmazonQ-For-IDE-Version-<QIDE-Plugin-VersionNo>

    This applies when Q Developer makes AWS API calls through IDE integrations, such as when analyzing your codebase or suggesting AWS resource configurations. The IDE marker enables you to distinguish between CLI-based and IDE-based Q Developer operations.

    Complete User Agent Example

    Here’s how a complete user agent string appears in CloudTrail:

    From Q Developer CLI:

    "userAgent": "aws-cli/2.27.17 md/awscrt#0.26.1 ua/2.1 os/macos#24.6.0 md/arch#x86_64 lang/python#3.13.3 md/pyimpl#CPython exec-env/AmazonQ-For-CLI-Version-1.15.0 
    cfg/retry-mode#standard md/installer#exe md/prompt#off md/command#sts.get-caller-identity"
    

    From Q Developer IDE Integration:

    "user-agent": "aws-cli/2.27.17 md/awscrt#0.26.1 ua/2.1 os/macos#24.6.0 md/arch#x86_64 lang/python#3.13.3 md/pyimpl#CPython exec-env/AmazonQ-For-IDE-Version-1.93.0 
    cfgretry-mode#standard md/installer#exe md/prompt#off md/command#sts.get-caller-identity"
    

    The key identifiers are exec-env/AmazonQ-For-CLI-Version-* and exec-env/AmazonQ-For-IDE-Version-*, which clearly distinguish Amazon Q Developer operations from regular AWS CLI/SDK usage executed outside of Q Developer.

    Architecture Diagram

    ┌─────────────────────────────────────────────────────────────────────────────┐
    │                           Amazon Q Developer Flow                           │
    └─────────────────────────────────────────────────────────────────────────────┘
    
    ┌──────────────────┐    ┌──────────────────┐    ┌──────────────────┐
    │   Developer      │    │   Amazon Q       │    │   AWS APIs       │
    │                  │    │   Developer      │    │                  │
    │ ┌──────────────┐ │    │                  │    │                  │
    │ │ Q CLI        │ │    │ ┌──────────────┐ │    │ ┌──────────────┐ │
    │ │ use_aws tool │ │────┼─│ Adds marker: │ │────┼─│ CloudTrail   │ │
    │ └──────────────┘ │    │ │ exec-env/    │ │    │ │ Event with   │ │
    │                  │    │ │ AmazonQ-For- │ │    │ │ User Agent   │ │
    │ ┌──────────────┐ │    │ │ CLI-Version  │ │    │ │ Marker       │ │
    │ │ IDE          │ │    │ └──────────────┘ │    │ └──────────────┘ │
    │ │ Integration  │ │────┼─│ Adds marker: │ │    │                  │
    │ └──────────────┘ │    │ │ exec-env/    │ │    │                  │
    │                  │    │ │ AmazonQ-For- │ │    │                  │
    │ ┌──────────────┐ │    │ │ IDE-Version  │ │    │                  │
    │ │ execute_bash │ │────┼─└──────────────┘ │    │                  │
    │ │ commands     │ │    │                  │    │                  │
    │ └──────────────┘ │    │                  │    │                  │
    └──────────────────┘    └──────────────────┘    └──────────────────┘
             │                        │                        │
             │                        │                        │
             ▼                        ▼                        ▼
    ┌──────────────────────────────────────────────────────────────────────────────┐
    │                              IAM Policy Engine                               │
    │                                                                              │
    │  ┌─────────────────────────────────────────────────────────────────────────┐ │
    │  │ Condition: StringLike                                                   │ │
    │  │ "aws:userAgent": "*exec-env/AmazonQ-For-*"                              │ │
    │  │                                                                         │ │
    │  │ ┌─────────────────┐              ┌─────────────────┐                    │ │
    │  │ │ Q Developer     │              │ Regular AWS     │                    │ │
    │  │ │ Operations      │              │ CLI Operations  │                    │ │
    │  │ │                 │              │                 │                    │ │
    │  │ │ • Block writes  │              │ • Allow writes  │                    │ │
    │  │ │ • Allow reads   │              │ • Allow reads   │                    │ │
    │  │ └─────────────────┘              └─────────────────┘                    │ │
    │  └─────────────────────────────────────────────────────────────────────────┘ │
    └──────────────────────────────────────────────────────────────────────────────┘
    

    IAM Policy Implementation

    Use the aws:userAgent condition in IAM policies to control Amazon Q Developer operations through two approaches:

    IAM Policies: Deploy in each AWS account where developers have access for deploying workloads or performing AWS operations. Q Developer operates using the developer’s existing AWS credentials and permissions – it doesn’t have additional access beyond what the user already possesses. Attach these policies to the same IAM users, groups, or roles that developers use for their regular AWS work.

    Service Control Policies (SCPs): Deploy once at the AWS Organizations level for organization-wide governance. SCPs apply to all member accounts automatically and cannot be overridden by account-level policies.

    The following policy allows read operations from Q Developer, blocks write operations from Q Developer, and allows write operations from regular AWS CLI executed outside Q Developer:

    Note: This IAM policy example is for illustration purposes only. Follow least privilege principles in production environments. For more details refer prepare for least previlege permissions.

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "AllowReadOperationsFromQDeveloper",
          "Effect": "Allow",
          "Action": [
            "s3:GetObject*",
            "s3:ListBucket*",
            "ec2:Describe*"
          ],
          "Resource": "*",
          "Condition": {
            "StringLike": {
              "aws:userAgent": "*exec-env/AmazonQ-For-*"
            }
          }
        },
        {
          "Sid": "BlockWriteOperationsFromQDeveloper",
          "Effect": "Deny",
          "Action": [
            "s3:DeleteObject*",
            "ec2:TerminateInstances",
            "iam:DeleteUser"
          ],
          "Resource": "*",
          "Condition": {
            "StringLike": {
              "aws:userAgent": "*exec-env/AmazonQ-For-*"
            }
          }
        },
        {
          "Sid": "AllowWriteOperationsFromRegularCLI",
          "Effect": "Allow",
          "Action": [
            "s3:DeleteObject*",
            "ec2:TerminateInstances",
            "iam:DeleteUser"
          ],
          "Resource": "*",
          "Condition": {
            "StringNotLike": {
              "aws:userAgent": "*exec-env/AmazonQ-For-*"
            }
          }
        }
      ]
    }
    

    Note on User Agent Reliability: While AWS warns that user agents can be “spoofed,” this concern is reduced for Q Developer governance use cases. The user agent is automatically set by Q Developer’s tools, not manually controlled by users. Any spoofing would require deliberate effort and would be detectable through usage pattern analysis. This approach is designed for operational governance and policy differentiation, not as a sole security control.

    Additional Control Layer: Custom Agent Configuration

    For an additional layer of control, you can create a custom agent configuration that restricts which AWS services Amazon Q Developer can access using allowedServices and deniedServices parameters for the use_aws tool:

    {
      "toolsSettings": {
        "use_aws": {
          "allowedServices": ["s3", "lambda", "ec2"],
          "deniedServices": ["eks", "rds"]
        }
      }
    }
    

    This custom agent configuration works in conjunction with IAM policies to provide defense-in-depth governance of AI-assisted AWS operations. For more details, refer to the agent configuration documentation.

    Verification and Monitoring

    CloudTrail Event Analysis

    To verify that your policies are working correctly, examine CloudTrail events. Here’s what to look for:

    Amazon Q Developer Event

    {
      "eventTime": "2025-01-15T10:30:00Z",
      "eventName": "GetCallerIdentity",
      "userAgent": "aws-cli/2.27.17 md/awscrt#0.26.1 ua/2.1 os/macos#24.6.0 md/arch#x86_64 lang/python#3.13.3 md/pyimpl#CPython exec-env/AmazonQ-For-CLI-Version-1.15.0 cfg/retry-mode#standard md/installer#exe md/prompt#off md/command#sts.get-caller-identity",
      "sourceIPAddress": "203.0.113.12",
      "userIdentity": {
        "type": "IAMUser",
        "principalId": "AIDACKCEVSQ6C2EXAMPLE",
        "arn": "arn:aws:iam::123456789012:user/developer"
      }
    }
    

    Regular AWS CLI Event

    {
      "eventTime": "2025-01-15T10:35:00Z",
      "eventName": "GetCallerIdentity", 
      "userAgent": "aws-cli/2.27.17 md/awscrt#0.26.1 ua/2.1 os/macos#24.6.0 md/arch#x86_64 lang/python#3.13.3 md/pyimpl#CPython cfg/retry-mode#standard md/installer#exe md/prompt#off md/command#sts.get-caller-identity",
      "sourceIPAddress": "203.0.113.12",
      "userIdentity": {
        "type": "IAMUser",
        "principalId": "AIDACKCEVSQ6C2EXAMPLE", 
        "arn": "arn:aws:iam::123456789012:user/developer"
      }
    }
    

    Monitoring Script Example

    Create a simple monitoring script to track Amazon Q Developer usage:

    #!/bin/bash
    # Monitor Amazon Q Developer AWS API usage
    # Get events from last 24 hours and filter for Q Developer user agents
    aws cloudtrail lookup-events \
      --start-time $(date -u -v-24H '+%Y-%m-%dT%H:%M:%SZ') \
      --lookup-attributes AttributeKey=EventName,AttributeValue=GetCallerIdentity \
      --query 'Events[?contains(CloudTrailEvent, `AmazonQ-For-CLI`)].[EventTime,EventName,UserIdentity.userName]' \
      --output table
    

    Conclusion

    Amazon Q Developer’s built-in user agent markers provide a powerful foundation for implementing enterprise-grade security controls around AI-assisted AWS operations. By leveraging these markers in IAM policies, organizations can:

    • Distinguish between AI-assisted and manual AWS operations
    • Implement differentiated security policies based on operation source
    • Maintain detailed audit trails for compliance requirements
    • Enable secure Amazon Q Developer adoption in enterprise environments while maintaining strict controls over write operations that could impact production systems

    For organizations currently evaluating Amazon Q Developer adoption, implementing user agent marker-based controls is a key component of your deployment strategy. This approach enables you to realize the productivity benefits of AI-assisted development while maintaining the governance and security controls your organization requires.

    Experience the power of Amazon Q Developer as your AI-powered coding assistant, and implement the governance controls outlined in this post to ensure secure adoption in your enterprise environment. These built-in user agent markers enable you to maintain enterprise-grade security while unlocking the productivity benefits of AI-assisted development.

    To learn more about Amazon Q Developer’s features and capabilities, visit the Amazon Q Developer product page.

    About the Author

    kirankumar.jpeg

    Kirankumar Chandrashekar is a Generative AI Specialist Solutions Architect at AWS, focusing on Amazon Q Developer/Kiro and developer productivity. Bringing deep expertise in AWS cloud services, DevOps, modernization, and infrastructure as code, he helps customers accelerate their development cycles and elevate developer productivity through innovative AI-powered solutions. By leveraging Amazon Q Developer and Kiro, he enables teams to build applications faster, automate routine tasks, and streamline development workflows. Kirankumar is dedicated to enhancing developer efficiency while solving complex customer challenges, and enjoys music, cooking, and traveling.

    Unlock new possibilities: AWS Organizations service control policy now supports full IAM language

    Post Syndicated from Swara Gandhi original https://aws.amazon.com/blogs/security/unlock-new-possibilities-aws-organizations-service-control-policy-now-supports-full-iam-language/

    Amazon Web Service (AWS) recently announced that AWS Organizations now offers full AWS Identity and Access Management (IAM) policy language support for service control policies (SCPs). With this feature, you can use conditions, individual resource Amazon Resource Names (ARNs), and the NotAction element with Allow statements. Additionally, you can now use wildcards at the beginning or middle of the Action element strings and implement the NotResource element in both Allow and Deny statements in SCPs. This feature is now available across AWS commercial and AWS GovCloud (US) Regions.

    In this blog post, we walk through a set of newly supported SCP language capabilities that simplify permission management cases. These enhancements enable more intuitive and concise policy designs. We explore how these capabilities address past limitations to reduce operational overhead and improve policy readability. We also show what the previous implementation looked like and provide an example of how the new capability makes the intent clearer and implementation simpler.

    Overview of the newly supported elements

    The following table lists the supported SCP language elements along with their purpose and applicable effects. Elements and effects shown in bold indicate newly supported capabilities.

    Element Purpose Supported effects
    Version Specifies the language syntax rules to use for processing the policy. Allow, Deny
    Statement Serves as the container for policy elements. You can have multiple statements in an SCP. Allow, Deny
    Statement ID (Sid) (Optional) Provides a friendly name for the statement. Allow, Deny
    Effect Defines whether the SCP statement allows or denies access to the IAM users and roles in an account. Allow, Deny
    Action Specifies the AWS service and actions that the SCP allows or denies. Allow, Deny
    NotAction Specifies the AWS service and actions that are exempt from the SCP. Used instead of the Action element. Allow, Deny
    Resource Specifies the AWS resources that the SCP applies to. Allow, Deny
    NotResource Specifies the AWS resources that are exempt from the SCP. Used instead of the Resource element. Allow, Deny
    Condition Specifies the conditions for when the statement is in effect. Allow, Deny

    Additionally, you can now use the wildcard characters * and ? anywhere in the Action or NotAction element. Previously, these wildcards were only allowed by themselves or at the end of an element. For example, all of the following are now valid:

    • "servicename:action*"
    • "servicename:*action"
    • "servicename:some*action"
    • "servicename:*"

    Navigating new SCP language capabilities

    Let’s explore recommended policy strategies and best practices by walking through some examples.

    Using Deny with NotResource

    You can use the NotResource element to apply a policy across resources except those explicitly listed. This is especially useful for implementing broad deny-by-default policies with scoped exceptions, simplifying policy structure while enforcing strong boundaries.

    Example 1:

    The goal of this example is to enforce a resource perimeter that blocks access to resources outside the organization, except for a defined set of service-owned resources.

    • Previous implementation: The policy used a tag-based approach to manage exceptions. It required tagging IAM principals with dp:exclude:resource:s3=true to grant access to external resources. This created operational overhead in tag management and introduced potential security risks if tags were incorrectly applied.
    • Improved implementation: With support for NotResource in Deny statements, the updated SCP uses a single, consolidated Deny statement denying the action except for a defined set of AWS-owned resources.
    Policy structure before NotResource support Policy structure after NotResource support
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "EnforceResourcePerimeterAWSResourcesS3",
          "Effect": "Deny",
          "Action": "s3:GetObject",
          "Resource": "*",
          "Condition": {
            "StringNotEqualsIfExists": {
              "aws:ResourceOrgID": "<my-org-id>",
              "aws:PrincipalTag/dp:exclude:resource:s3": "true"
            }
          }
        }
      ]
    }

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "EnforceResourcePerimeterAWSResources",
          "Effect": "Deny",
          "Action": "s3:GetObject",
          "NotResource": [
            "arn:aws:s3:::service-owned-bucket/*",
            "arn:aws:s3:::service-owned-bucket2/*"
          ],
          "Condition": {
            "StringNotEquals": {
              "aws:ResourceOrgID": "<org-id>"
            }      
           }
        }
      ]
    }

    Example 2:

    This example denies access to Amazon Bedrock models except for one specific model.

    • Before this change: SCP relied on a broad permission baseline for AWS accounts within the organization by allowing access to Amazon Bedrock actions by default, while explicitly denying invocation of three specific models (examples: Deepseek, Anthropic, and meta). However, this approach requires continuous operational overhead to make sure policies are updated to deny access to newly added models to avoid exposure to potentially unwanted models.
    • Improved implementation: With support for NotResource in Deny statements, the updated SCP uses a single, consolidated Deny statement that denies actions except Amazon models.
    Policy structure before NotResource support Policy structure after NotResource support
    {
    	"Version": "2012-10-17",
    	"Statement": [
    		{
    			"Effect": "Allow",
    			"Action": "bedrock:*",
    			"Resource": "*"
    		},
    		{
    			"Effect": "Deny",
    			"Action": [
    				"bedrock:InvokeModel",
    				"bedrock:InvokeModelWithResponseStream",
    				"bedrock:PutFoundationModelEntitlement"
    			],
    			"Resource": [
    				"arn:aws:bedrock:*::foundation-model/deepseek.*",
    				"arn:aws:bedrock:*::foundation-model/anthropic.*",
    				"arn:aws:bedrock:*::foundation-model/meta.*"
    			]
    		}
    	]
    }

    {
    	"Version": "2012-10-17",
    	"Statement": [
    		{
    			"Effect": "Allow",
    			"Action": "bedrock:*",
    			"Resource": "*"
    		},
    		{
    			"Sid": "Statement1",
    			"Effect": "Deny",
    			"Action": [
    				"bedrock:InvokeModel",
    				"bedrock:InvokeModelWithResponseStream",
    				"bedrock:PutFoundationModelEntitlement"
    			],
    			"NotResource": [
    				"arn:aws:bedrock:*::foundation-model/amazon.*"
    			]
    		}
    	]
    }

    Using Allow with conditions

    By using the Condition element, you can specify the circumstances under which a policy statement is in effect. While optional, this element is now supported in Allow statements within SCPs, enabling more precise and scalable access control.

    Note: We recommend using explicit Deny statements when authoring SCPs in most cases. Using Deny statements help make sure that each control works independently and remains enforceable. Relying solely on allow statements and the implicit deny-by-default model can lead to unintended access, because broader or overlapping Allow statements can override more restrictive ones.

    The following example allows access to specific AWS services in certain AWS Regions.

    • Before this change: The policy uses a single Allow statement under the Sid: AllowSpecificServices. It lists broad service-level actions (for example, "ec2:", "s3:", and so on) in the Action element and applies them across resources ("Resource": "*"). Because AWS SCPs operate under a deny-by-default model, this setup effectively permits actions across the listed services while implicitly denying access to other services not included. For example, an explicit Deny restricts actions outside us-east-1, us-west-2, and eu-central-1 using a Region condition.
    • Improved implementation: In the updated example, the policy allows the same services, but only when they are requested in specific Regions (for example, "us-east-1", "us-west-2", and "eu-central-1"). This is achieved using the aws:RequestedRegion condition key in the Allow statement. This enhancement allows organizations to retain basic Allow logic while introducing contextual boundaries—such as limiting access by Region, account, or resource tag—previously only possible with Deny conditions.

    Note: We recommend using one broad Allow statement and multiple targeted Deny statements in your policies. Avoid writing additional Allow statements that might overlap, because doing so could lead to unintended access. The recommended approach is to start with a broad Allow statement and then use Deny statements to refine and restrict access as needed.

    Policy structure before support for Allow with conditions Policy structure after support for Allow with conditions
    {
       "Version":"2012-10-17",
       "Statement":[
          {
             "Sid":"AllowSpecificServices",
             "Effect":"Allow",
             "Action":[
                "ec2:*",
                "s3:*",
                "rds:*",
                "lambda:*",
                "cloudformation:*",
                "iam:*",
                "cloudwatch:*"
             ],
             "Resource":"*"
          },
          {
             "Sid":"AllowAccessOnlyTo3Regions",
             "Effect":"Deny",
             "Action":"*",
             "Resource":"*",
             "Condition":{
                "StringNotEquals":{
                   "aws:RequestedRegion":[
                      "us-east-1",
                      "us-west-2",
                      "eu-central-1"
                   ]
                }
             }
          }
       ]
    }

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "AllowServicesBasedOnRegion",
          "Effect": "Allow",
          "Action": [
            "ec2:*",
            "s3:*",
            "rds:*",
            "lambda:*",
            "cloudformation:*",
            "iam:*",
            "cloudwatch:*"
          ],
          "Resource": "*",
          "Condition": {
            "StringEquals": {
              "aws:RequestedRegion": [
                "us-east-1",
                "us-west-2",
                "eu-central-1"
              ]
            }
          }
        }
      ]
    }

    Other newly supported elements

    To bring SCPs to full IAM policy language support, additional elements are now supported. While technically valid, some of these constructs require additional considerations and testing in practice because of their potential for unintended access if not carefully managed.

    Newly supported feature Important considerations
    Action with wildcards (*, ?) Can help shorten policies but use with caution—new actions added by AWS will match existing wildcard patterns as designed, potentially granting unintended permissions.
    NotAction with wildcards (*, ?) We recommend using NotAction with a Deny statement if you want to deny all actions except those listed, which helps future-proof your controls (for example, denying everything in Amazon EC2 except actions that don’t match “*vpn*”.
    Allow with NotResource Limited use cases. While supported, Allow with NotResource can default to including all resources—potentially allowing access to new resources added later. Use with caution and prefer explicit Deny statements when possible.
    Allow with NotAction Limited use cases. While supported, Allow with NotAction can unintentionally permit access to new actions added by AWS. Use with caution and prefer explicit Deny statements to maintain control as services evolve.
    Allow with Resource other than wildcard “*”. When using Allow with specific resources (not "*"), make sure your policy design avoids conflicting or overlapping Allow statements. Start with a broad Allow, then use targeted Deny statements to restrict access—this helps prevent unintended access from overlapping Allow statements.

    Validate your policies with IAM Access Analyzer

    You can use AWS IAM Access Analyzer to validate your SCPs before applying them, using both policy validation and custom policy checks.

    IAM Access Analyzer validates your policy against IAM policy grammar and best practices. You can view policy validation check findings that include security warnings, errors, general warnings, and suggestions. These findings provide actionable recommendations to help you author policies that are both functional and aligned with security best practices.

    Custom policy checks are an IAM Access Analyzer capability that security teams can use to help them accurately and proactively identify critical permissions in their policies. Custom policy checks can determine whether a new version of a policy is more permissive than the previous version. They use automated reasoning—a form of static analysis—to provide a higher level of security assurance in the cloud.

    Custom policy checks can be embedded into continuous integration and continuous delivery (CI/CD) pipelines, so that policies can be checked without being deployed. Developers can also run custom policy checks from their local development environments and receive fast feedback on whether the policies they are authoring comply with your organization’s security standards. For more information refer to introducing IAM Access Analyzer custom policy checks.

    Conclusion

    The latest enhancements to AWS service control policies improve policy expressiveness and precision while reducing operational effort. By enabling constructs like Allow with conditions and specific resource ARNs, supporting NotResource in Deny statements, and expanding wildcard flexibility, you can simplify your policies and avoid layered or complex policies to achieve your goals. These updates bring SCPs in parity with IAM policy capabilities and empower organizations to implement cleaner, more intuitive access controls. As a best practice, it’s important to use these capabilities carefully—especially wildcard use—to avoid unintended permissions as AWS services evolve. We also encourage the implementation of explicit Deny statements as a best practice and using Allow statements when needed.


    If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

    Swara Gandhi

    Swara Gandhi

    Swara is a Senior Solutions Architect on the AWS Identity Solutions team. She works on building secure and scalable end-to-end identity solutions. She is passionate about everything identity, security, and cloud.

    Niti Prasad

    Niti Prasad

    Niti is a Senior Security Solutions Architect supporting Strategic Accounts. She supports customers as they look to secure and govern their AWS environment. Her enthusiasm for security drives her to continuously explore innovative ways to help customers protect their cloud workloads.

    AWS Weekly Roundup: SQS fair queues, CloudWatch generative AI observability, and more (July 28, 2025)

    Post Syndicated from Micah Walter original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-sqs-fair-queues-cloudwatch-generative-ai-observability-and-more-july-28-2025/

    To be honest, I’m still recovering from the AWS Summit in New York, doing my best to level up on launches like Amazon Bedrock AgentCore (Preview) and Amazon Simple Storage Service (S3) Vectors. There’s a lot of new stuff to learn!

    Meanwhile, it’s been an exciting week for AWS builders focused on reliability and observability. The standout announcement has to be Amazon SQS fair queues, which tackles one of the most persistent challenges in multi-tenant architectures: the “noisy neighbor” problem. If you’ve ever dealt with one tenant’s message processing overwhelming shared infrastructure and affecting other tenants, you’ll appreciate how this feature enables more balanced message distribution across your applications.

    On the AI front, we’re also seeing AWS continue to enhance our observability capabilities with the preview launch of Amazon CloudWatch generative AI observability. This brings AI-powered insights directly into your monitoring workflows, helping you understand infrastructure and application performance patterns in new ways. And for those managing Amazon Connect environments, the addition of AWS CloudFormation for message template attachments makes it easier to programmatically deploy and manage email campaign assets across different environments.

    Last week’s launches

    • Amazon SQS Fair Queues — AWS launched Amazon SQS fair queues to help mitigate the “noisy neighbor” problem in multi-tenant systems, enabling more balanced message processing and improved application resilience across shared infrastructure.
    • Amazon CloudWatch Generative AI Observability (Preview) — AWS launched a preview of Amazon CloudWatch generative AI observability, enabling users to gain AI-powered insights into their cloud infrastructure and application performance through advanced monitoring and analysis capabilities.
    • Amazon Connect CloudFormation Support for Message Template Attachments —AWS has expanded the capabilities of Amazon Connect by introducing support for AWS CloudFormation for Outbound Campaign message template attachments, enabling customers to programmatically manage and deploy email campaign attachments across different environments.
    • Amazon Connect Forecast Editing — Amazon Connect introduces a new forecast editing UI that allows contact center planners to quickly adjust forecasts by percentage or exact values across specific date ranges, queues, and channels for more responsive workforce planning.
    • Bloom Filters for Amazon ElastiCache — Amazon ElastiCache now supports Bloom filters in version 8.1 for Valkey, offering a space-efficient way to quickly check if an item is in a set with over 98% memory efficiency compared to traditional sets.
    • Amazon EC2 Skip OS Shutdown Option — AWS has introduced a new option for Amazon EC2 that allows customers to skip the graceful operating system shutdown when stopping or terminating instances, enabling faster application recovery and instance state transitions.
    • AWS HealthOmics Git Repository Integration — AWS HealthOmics now supports direct Git repository integration for workflow creation, allowing researchers to seamlessly pull workflow definitions from GitHub, GitLab, and Bitbucket repositories while enabling version control and reproducibility.
    • AWS Organizations Tag Policies Wildcard Support — AWS Organizations now supports a wildcard statement (ALL_SUPPORTED) in Tag Policies, allowing users to apply tagging rules to all supported resource types for a given AWS service in a single line, simplifying policy creation and reducing complexity.

    Blogs of note

    Beyond IAM Access Keys: Modern Authentication Approaches — AWS recommends moving beyond traditional IAM access keys to more secure authentication methods, reducing risks of credential exposure and unauthorized access by leveraging modern, more robust approaches to identity management.

    Upcoming AWS events

    AWS re:Invent 2025 (December 1-5, 2025, Las Vegas) — AWS’s flagship annual conference offering collaborative innovation through peer-to-peer learning, expert-led discussions, and invaluable networking opportunities.

    AWS Summits — Join free online and in-person events that bring the cloud computing community together to connect, collaborate, and learn about AWS. Register in your nearest city: Mexico City (August 6) and Jakarta (August 7).

    AWS Community Days — Join community-led conferences that feature technical discussions, workshops, and hands-on labs led by expert AWS users and industry leaders from around the world: Singapore (August 2), Australia (August 15), Adria (September 5), Baltic (September 10), and Aotearoa (September 18).

    Trusted identity propagation using IAM Identity Center for Amazon OpenSearch Service

    Post Syndicated from Sohaib Katariwala original https://aws.amazon.com/blogs/big-data/trusted-identity-propagation-using-iam-identity-center-for-amazon-opensearch-service/

    Enterprise customers of Amazon OpenSearch Service require comprehensive security controls with seamless authentication and authorization mechanisms when accessing data in provisioned domains and Amazon OpenSearch Serverless collections. Security teams within these organizations must not only maintain compliance with enterprise policies but also need to make sure that their users can access data securely, with robust identity management. AWS IAM Identity Center is a popular mechanism for identity management that provides single sign-on (SSO) capabilities for these enterprise customers. IAM Identity Center can use Security Assertion Markup Language (SAML) with both OpenSearch Service provisioned domains and OpenSearch Serverless. Now, by using trusted identity propagation, IAM Identity Center provides a new, direct method for accessing data in OpenSearch Service.

    In this post, we outline how you can take advantage of this new access method to simplify data access using the OpenSearch UI and still maintain robust role-based access control for your OpenSearch data.

    Trusted identity propagation overview

    Trusted identity propagation in IAM Identity Center adds the identity context of a user to a role when accessing OpenSearch Service, which in turn uses this context to authorize and scope OpenSearch data access. This simplifies the authentication and authorization flow for customers because the applications access the data on their behalf. Users or user agents need not be present between the application and the backend services for this authorization to happen, unlike methods like SAML where a user agent needs to be present between these entities as a go-between for exchanging assertions. This flexibility helps simplify accessing a wide variety of data sources such as data residing within the Amazon Virtual Private Cloud (Amazon VPC) of an OpenSearch Service domain, or an OpenSearch Serverless collection. By using the OpenSearch UI, you can additionally simplify the backend connections, resulting in seamless access to the data. The following figures shows how the identity propagation works with OpenSearch Service.

    Prerequisites

    Before starting to use IAM Identity Center with OpenSearch Service, there are a few options that you must enable. To start, set up an organization or account instance of IAM Identity Center following the instruction in this guide. For OpenSearch Service-provisioned domains, you must enable the IAM Identity Center (IDC) Authentication –new option. You can do this though AWS CloudFormation, OpenSearch REST API, AWS SDK, or the AWS Management Console.

    To enable Identity Center using the console

    1. To add the capability for an existing provisioned domain, go to the OpenSearch Service console and navigate to the Security configuration tab and choose Edit.

    Amazon OpenSearch Service console highlighting security settings, cluster health status, and VPC endpoint configuration

    1. After this step, or if you are creating new domain, select the check box for IAM Identity Center (IDC) Authentication – new.
    2. You have various options to choose for Subject Key and Roles Key depending upon how you want to establish your role-based access control discussed later in this post. For now, select UserName for Subject Key and GroupName for Roles Key.

    AWS IAM Identity Center SSO configuration interface with API authentication controls and identity mapping settings

    1. For OpenSearch Serverless, choose Serverless in the navigation pane, then Security and Authentication. Choose Edit in the IAM Identity Center (IdC) authentication – new section.

    AWS OpenSearch Service authentication management console with SAML providers list and IAM Identity Center integration status

    1. Select the checkbox for Authenticate with IAM Identity Center, and then choose Save.

    AWS IAM Identity Center configuration modal for enabling centralized authentication and resource access management

    1. Select the checkbox for Authentication with IAM Identity Center under Single sign-on authentication, when creating an OpenSearch UI application. For step-by-step instructions on how to create an OpenSearch UI application, see Creating an OpenSearch UI application

    AWS single sign-on setup interface with option to enable IAM Identity Center for unified access management

    After these steps, you’re ready to configure IAM Identity Center by creating new users and groups, or by using existing user identities.

    Propagating IAM Identity Center identities

    Currently, adding single sign-on authentication with IAM Identity Center can be done while setting up a new OpenSearch UI application. Use the following steps to create a new OpenSearch UI application. Note that single sign-on currently cannot be turned on after an application is created. After single sign-on is enabled, you should see an AWS managed application under Applications in the Identity Center console.

    AWS Identity Center interface showing managed applications tab with OpenSearch integration and service details

    Assigning users and groups

    After the application is created and the status shows as Active, you need to assign users and groups to the application. This assignment is important and recommended because these assignments determine the scope and permissions for data access within OpenSearch Service. To do this, select the application you created in the previous steps in the OpenSearch Service console. Here, you will see an option for IAM Identity Center user and groups under Single sign-on authentication. Choose Assign users and groups and select the appropriate Identity Center users and groups.

     AWS IAM Identity Center dashboard showing connected status, role details, and option to assign users and groups for SSO

    For OpenSearch Serverless, you must create a new data access policy or add a rule to an existing one to grant IAM Identity Center principals appropriate permissions to access the collections. For example, the following figure shows a data access policy that grants specific permissions to a one user with Rule 1 and provide a more restrictive permission to a group with Rule 2.

    Amazon OpenSearch Service console displaying sample data access policy with IAM principal and collection/index permissions

    At this point your OpenSearch Service domains, OpenSearch Serverless collections and OpenSearch UI are set up for identity propagation.

    Fine grained access control for IAM Identity Center identities

    Fine grained access control is a role-based access control for OpenSearch Service that provides security at index, document, and field levels for provisioned domains. You can choose what aspects of identity context you propagate to OpenSearch Service. You can choose between UserId, UserName, and Email for your Subject keys, and GroupId and GroupName for your Roles key. This configuration is important because the values of the properties in the identity context are used to match exactly with the user and backend role mapping within OpenSearch Service provisioned domains. Note that if IAM Identity Center sign-on isn’t enabled, OpenSearch Service can only evaluate the request signature with AWS signature Version 4. This means that the role your OpenSearch UI will use won’t contain identity context for authorization. To complete authorization, add the values of the identity context fields to the OpenSearch role mapping. See Mapping roles to users under Managing permissions. Role mapping can be done using OpenSearch REST API, AWS SDK, or using OpenSearch Dashboards.

    To map roles using OpenSearch Dashboards

    1. From the menu icon  on the top left corner or your screen, select Management, Security, Roles, <Your role>.
    2. Choose the Mapped Users tab and select Manage mapping.

    AWS security console showing Map User interface with internal user creation and backend role management for IAM

    1. When mapping the role, make sure that you enter the values corresponding to the Subject key. This value must be the same as in your identity context. Additionally, use the Roles key to assign access-based IAM Identity Center groups.

    With OpenSearch Serverless, the granularity of access control is at the index level so you will need to add additional rules in the Data Access Policy to control principals who can access collections or indices within a collection.

    Verifying identity propagation

    The final step is to verify identity propagation.

    1. Open the OpenSearch UI application and select IAM Identity Center from the Login drop down.

    1. After you complete the login process with IAM Identity Center, the OpenSearch UI will open. Choose the user icon in the lower left corner of the screen to verify that it’s your correct principal from Identity Center. It should match the Identity Center property you chose earlier.

    OpenSearch welcome interface showing available workspaces, documentation links, and user profile menu with logout option

    1. To verify correct identity propagation, choose the Dev tools  icon just above the user profile icon in the bottom left corner of the screen.
    2. Select the correct OpenSearch domain or OpenSearch Serverless collection data source in the top right corner of the screen and run a _search query. You should see results from the data source confirming that the identity is correctly propagated to OpenSearch Service.

    Dev tools showing successful search operation with 5ms response time and config details

    Conclusion

    In this post, we showed you how to use Trusted Identity Propagation using IAM Identity Center for Amazon OpenSearch Service, providing a streamlined approach to secure data access while maintaining robust access controls. This solution offers several key benefits:

    • Simplified authentication: By eliminating the need for user agents between applications and backend services, the solution streamlines the authentication process compared to traditional SAML-based approaches.
    • Enhanced security: The integration maintains comprehensive security controls while providing seamless authentication and authorization mechanisms for both OpenSearch Service provisioned domains and Amazon OpenSearch Serverless collections.
    • Flexible identity management: Organizations can use existing IAM Identity Center implementations to manage user access, making it easier to maintain compliance with enterprise security policies.
    • Fine-grained access control: The solution supports detailed access control at the index, document, and field level for provisioned domains, allowing organizations to implement precise security measures.

    Get started implementing this solution in your environment today!

    For more information about identity management and security best practices with OpenSearch Service, we recommend:


    About the authors

    Muthu Pitchaimani is a Search Specialist with Amazon OpenSearch Service. He builds large-scale search applications and solutions. Muthu is interested in the topics of networking and security, and is based out of Austin, Texas.

    Sohaib Katariwala is a Senior Specialist Solutions Architect at AWS focused on Amazon OpenSearch Service based out of Chicago, IL. His interests are in all things data and analytics. More specifically he loves to help customers use AI in their data strategy to solve modern day challenges.

    Beyond IAM access keys: Modern authentication approaches for AWS

    Post Syndicated from Mitch Beaumont original https://aws.amazon.com/blogs/security/beyond-iam-access-keys-modern-authentication-approaches-for-aws/

    When it comes to AWS authentication, relying on long-term credentials, such as AWS Identity and Access Management (IAM) access keys, introduces unnecessary risks; including potential credential exposure, unauthorized sharing, or theft. In this post, I present five common use cases where AWS customers traditionally use IAM access keys and present more secure alternatives that you should consider.

    AWS CLI access: Embrace CloudShell

    If you’re primarily using access keys for AWS Command Line Interface (AWS CLI) access, consider AWS CloudShell—a browser-based CLI that minimizes the need for local credential management while providing the same powerful CLI capabilities that you’re accustomed to.

    AWS CLI with enhanced security: IAM Identity Center

    If you need a more robust solution, AWS CLI v2 combined with AWS IAM Identity Center offers a superior authentication approach. This integration enables:

    • Centralized user management
    • Seamless multi-factor authentication (MFA) integration
    • Enhanced security controls

    Configuration is straightforward using the AWS CLI documentation, and MFA can be enabled following the IAM Identity Center MFA guide.

    Local development: IDE integration

    For developers working in local environments, modern integrated development environments (IDEs) such as Visual Studio Code, with AWS Toolkit support offer secure authentication through IAM Identity Center. This alleviates the need for static access keys while maintaining a smooth development experience. Learn more about AWS IDE integrations.

    AWS compute services and CI/CD access

    When your applications and automation pipelines need AWS resource access, whether running on AWS compute services (Amazon Elastic Compute Cloud (Amazon EC2), Amazon Elastic Container Service (Amazon ECS), or AWS Lambda) or through continuous integration and delivery (CI/CD) tools, IAM roles can provide the ideal solution. These roles automatically manage temporary credential rotation and follow security best practices.

    • For AWS compute services: Use standard IAM roles with your compute resources. Review the EC2 IAM roles documentation for implementation details.
    • For AWS-hosted CI/CD: When using AWS CodePipeline or AWS CodeBuild for example, use service-linked roles to manage permissions securely.
    • For CI/CD tools self-hosted on Amazon EC2: If you’re running tools such as Jenkins or GitLab on AWS resources, use the instance profile roles the same as you would with other compute services.

    For third-party CI/CD services (such as GitHub Actions, CircleCI, and so on), see External access requirements.

    External access requirements

    For scenarios involving third-party applications or on-premises workloads, AWS offers three methods:

    • Third-party applications: Implement temporary security credentials through IAM roles instead of static access keys. Never use root account access keys. See third-party access documentation.
    • On-premises workloads: Use AWS IAM Roles Anywhere to generate temporary credentials for non-AWS workloads. For more information, see Access for non AWS workloads.
    • CI/CD software as a service (SaaS): For cloud-based CI/CD services, use OpenID Connect (OIDC) integration with IAM roles to minimize the need for long-term credentials. This allows your CI/CD pipelines to obtain temporary credentials through trust relationships. See the AWS OIDC provider documentation for implementation details.

    Best practice: Principle of least privilege

    Regardless of your authentication method, always implement the principle of least privilege. This helps make sure that users and applications have only the permissions they need. For guidance on crafting precise IAM policies, see Techniques for writing least privilege IAM policies.

    Note: AWS also offers policy generation based on AWS CloudTrail logs, helping you create permission templates based on actual usage patterns. Learn about this feature in the IAM policy generation documentation.

    Conclusion

    As you’ve seen, there are numerous secure alternatives to IAM access keys that you can use to enhance your AWS authentication strategy while reducing security risks. By using tools such as CloudShell, IAM Identity Center, IDE integrations, IAM roles, and IAM Roles Anywhere, you can implement robust authentication mechanisms that align with modern security best practices.Key takeaways:

    • Prefer temporary credentials over long-term access keys
    • Choose the authentication method that best fits your use case
    • Implement the principle of least privilege across all access methods
    • Take advantage of the built-in tools provided by AWS for policy generation and management
    • Regularly review and update your authentication methods as new solutions become available

    By making these changes, you can not only improve your security posture but also streamline your authentication processes across your AWS environment. Start small by identifying your current IAM access key use cases and gradually transition to these more secure alternatives. Your future self—and your security team—will thank you.

    If you have feedback about this post, submit comments in the Comments section below.

    Mitch Beaumont

    Mitch Beaumont

    Mitch is a Principal Solutions Architect for Amazon Web Services based in Sydney, Australia. Mitch works with some of Australia’s largest financial services customers, helping them to continually raise the security bar for the products and features that they build and ship. Outside of work, Mitch enjoys spending time with his family, photography, and surfing.

    Implement secure hybrid and multicloud log ingestion with Amazon OpenSearch Ingestion

    Post Syndicated from Xiaoxue Xu original https://aws.amazon.com/blogs/big-data/implement-secure-hybrid-and-multicloud-log-ingestion-with-amazon-opensearch-ingestion/

    Running applications across hybrid or multicloud environments creates a common challenge: fragmented logs scattered across different platforms. This fragmentation complicates monitoring, slows troubleshooting, and reduces operational visibility. To address this, many organizations seek to implement secure log ingestion from all environments into a centralized platform.

    Amazon OpenSearch Service provides a unified solution for real-time search, analytics, and log management across your entire infrastructure. Amazon OpenSearch Ingestion, a fully managed data collector, simplifies data processing with built-in capabilities to filter, transform, and enrich your logs before analysis.

    However, securely sending logs from non-AWS environments presents a challenge. Every request to OpenSearch Ingestion requires AWS Signature Version 4 (AWS SigV4) authentication, traditionally requiring long-term credentials that introduce security risks. AWS Identity and Access Management Roles Anywhere solves this problem by providing temporary credentials for workloads running outside AWS.

    In this post, we demonstrate how to configure Fluent Bit, a fast and flexible log processor and router supported by various operating systems, to securely send logs from any environment to OpenSearch Ingestion using IAM Roles Anywhere. This approach alleviates the need for long-term credentials while providing a comprehensive view of your application logs across all environments—improving security, simplifying operations, and enhancing your ability to quickly resolve issues.

    Solutions overview

    The solution in this post uses Fluent Bit to collect logs, retrieve temporary credentials from IAM Roles Anywhere, and sign HTTP log ingestion requests with AWS SigV4 before sending them to the OpenSearch Ingestion pipeline. The following diagram shows the architecture.

    Architecture for log ingestion with AWS IAM Roles Anywhere

    This solution provisions the following key components:

    • Certificate authority – For this post, we use AWS Private Certificate Authority (AWS Private CA) as the certificate authority (CA) source. Alternatively, you can integrate with an external CA; for more details, see IAM Roles Anywhere with an external certificate authority. Certificates issued from public CAs can’t be used as trust anchors for IAM Roles Anywhere.
    • X.509 Certificate – We use a sample private certificate stored in AWS Certificate Manager (ACM) and issued by AWS Private CA.
    • IAM Roles Anywhere configuration – This includes the following:
      • Trust anchor – Establishes trust between IAM Roles Anywhere and the specified CA.
      • IAM role – Grants permissions for log ingestion and trusts the IAM Roles Anywhere service principal. At minimum, this role must be granted permission for the osis:Ingest action.
      • Profile – Defines which roles IAM Roles Anywhere can assume and the maximum permissions granted with the temporary credentials.
    • OpenSearch Service domain – For this post, we use an OpenSearch Service domain, which is an AWS provisioned equivalent of an open source OpenSearch cluster. We create the domain within a virtual private cloud (VPC); see VPC versus public domains for more information. Alternatively, you can use an Amazon OpenSearch Serverless collection, which is an OpenSearch cluster that scales compute capacity based on your application’s needs.
    • OpenSearch Ingestion – This is configured to receive logs over HTTP as the pipeline source and forward them to the OpenSearch Service domain as the pipeline sink.

    Connectivity between AWS and your hybrid or multicloud environments

    You can access your OpenSearch Ingestion pipelines using an interface VPC endpoint with push-based HTTP source, which provides private IP address connectivity. For production environments, we recommend using these private connections through interface endpoints for enhanced security.

    Setting up this connectivity requires additional configuration, such as creating an AWS Site-to-Site VPN connection with your hybrid and multicloud network. Although this post focuses on the log ingestion solution, you can find detailed guidance on network connectivity in the following resources:

    How Fluent Bit retrieves temporary credentials using IAM Roles Anywhere

    Using the HTTP output plugin, Fluent Bit can send logs to the OpenSearch Ingestion pipeline. The following diagram is a simplified view of how Fluent Bit retrieves AWS credentials.

    How Fluent Bit retrieve AWS credentials

    On Linux systems, Fluent Bit can use an AWS Command Line Interface (AWS CLI) profile that uses the credential_process parameter to trigger an external process. This external process is invoked to generate or retrieve credentials not directly supported by the AWS CLI.

    The following are two common mechanisms for the external process:

    Although both options are viable, this post focuses on IAM Roles Anywhere. In this setup, the AWS IAM Roles Anywhere Credential Helper is executed to handle the signing process for the CreateSession API. This returns credentials in a JSON format that Fluent Bit can consume.

    As of this writing, the Fluent Bit aws_profile configuration is supported only on Linux. It is untested on other Unix-based systems (such as macOS) and is not implemented for Windows.

    Prerequisites

    Before you begin this walkthrough, make sure you have the following:

    Deploy AWS resources with AWS CloudFormation

    Follow these steps to deploy AWS resources required for this solution:

    1. Choose Launch Stack:


    2. Enter a unique name for Stack name. The default value is osis-with-iamra.
    3. Configure the stack parameters. Default values are provided in the following table.
    Parameter Default value Description
    CACommonName example.com Common Name for the CA
    CACountry US Organization for the CA
    CAOrganization Example Org Country for the CA
    CAValidityInDays 1826 Validity period in days for the CA certificate
    VPCCIDR 10.0.0.0/16 IPv4 CIDR range for the VPC used for OpenSearch Service domain
    PublicSubnetCIDR 10.0.0.0/24 IPv4 CIDR range for public subnet
    PrivateSubnet1CIDR 10.0.1.0/24 IPv4 CIDR range for private subnet
    PrivateSubnet2CIDR 10.0.2.0/24 IPv4 CIDR range for private subnet
    DomainName test-domain Name of the OpenSearch Service domain
    PipelineName test-pipeline Name of the OpenSearch Ingestion pipeline
    PipelineIngestionPath /test-ingestion-path Ingestion path for the OpenSearch Ingestion pipeline
    1. Select the acknowledgement check box and choose Create Stack.
      Stack deployment takes about 30 minutes to complete.
    2. When stack creation is complete, navigate to the Outputs tab on the AWS CloudFormation console and note down the values for the resources created.
      The following table summarizes the output values.
    Output Description Example value
    ACMCertificateArn Amazon Resource Name (ARN) of the ACM certificate. You will use this for exporting certificate and private key files using the AWS CLI in a later step. arn:aws:acm:aa-example-1:111122223333:certificate/a1b2c3d4-5678-90ab-cdef-EXAMPLE11111
    CertificateAuthorityArn ARN of the Private CA. arn:aws:acm-pca:aa-example-1:111122223333:certificate-authority/a1b2c3d4-5678-90ab-cdef-EXAMPLE22222
    TrustAnchorArn ARN of the IAM Roles Anywhere profile. You will use this value for configuring credential_process for IAM Roles Anywhere in a later step. arn:aws:rolesanywhere:aa-example-1:111122223333:trust-anchor/a1b2c3d4-5678-90ab-cdef-EXAMPLE33333
    IngestionRoleArn ARN of the OpenSearch Ingestion role. You will use this value for configuring credential_process for IAM Roles Anywhere in a later step. arn:aws:iam::111122223333:role/role-name-with-path
    ProfileArn ARN of the IAM Roles Anywhere profile. You will use this value for configuring credential_process for IAM Roles Anywhere in a later step. arn:aws:rolesanywhere:aa-example-1:111122223333:profile/a1b2c3d4-5678-90ab-cdef-EXAMPLE44444
    OpenSearchDomainEndpoint Endpoint of the VPC OpenSearch domain. You will use this public endpoint for querying your index after ingestion. vpc-my-domain-123456789012.aa-example-1.es.amazonaws.com
    PipelineEndpoint Endpoint of the OpenSearch Ingestion pipeline. You will use this public endpoint in the Fluent Bit configuration. my-pipeline-123456789012.aa-example-1.osis.amazonaws.com
    PipelineIngestionPath Ingestion path for the OpenSearch Ingestion pipeline. /test-ingestion-path

    Export a sample private certificate using CloudShell

    Follow these steps to export the sample private certificate created by the CloudFormation stack:

    1. Open CloudShell. For more details, see Navigating the AWS CloudShell interface.
    2. Export the certificate ARN from the CloudFormation outputs. If you changed the stack name in the previous step, use that value for <stack-name>, otherwise use the default value osis-with-iamra.
    export CERT_ARN=$(aws cloudformation describe-stacks \
        --stack-name <stack-name> \
        --query 'Stacks[0].Outputs[?OutputKey==`ACMCertificateArn`].OutputValue' \
        --output text)
    1. Extract the certificate and private key files:
    # Generate and save the passphrase
    export PASSPHRASE=$(openssl rand -base64 32)
    
    # Export certificate using environment variables
    aws acm export-certificate \
        --certificate-arn $CERT_ARN \
        --passphrase $(echo -n "$PASSPHRASE" | base64) \
        > cert_export.json
    
    # Extract components to separate files
    jq -r '.Certificate' cert_export.json > certificate.pem
    jq -r '.PrivateKey' cert_export.json > encrypted_private_key.pem
    
    # Decrypt the private key
    openssl rsa -in encrypted_private_key.pem -out private_key.pem -passin pass:"$PASSPHRASE"
    
    # Clear environment variables
    unset PASSPHRASE CERT_ARN
    1. Download the extracted certificate and private key files from CloudShell:
      1. /home/cloudshell-user/certificate.pem
      2. /home/cloudshell-user/private_key.pem

    Configure an AWS CLI profile

    Follow these steps to configure an AWS CLI profile for your log ingestion environment:

    1. Store the downloaded certificate and private key to your environment. For an automated approach to generate and rotate certificates, see Set up AWS Private Certificate Authority to issue certificates for use with IAM Roles Anywhere.
    2. Create a new profile named osis-pipeline-credentials that invokes the credential process. Replace the placeholders with your specific values. Find the values for trusted-anchor-arn, profile-arn, and ingestion-role-arn in your CloudFormation stack outputs.
    aws configure set profile.osis-pipeline-credentials.credential_process "</path/to/aws_signing_helper> credential-process \
          --certificate </path/to/certificate.pem> \
          --private-key </path/to/private_key.pem> \
          --trust-anchor-arn <trusted-anchor-arn> \
          --profile-arn <profile-arn> \
          --role-arn <ingestion-role-arn>"
    1. Verify your configuration. Open the ~/.aws/config file and confirm it contains a profile named osis-pipeline-credentials similar to the following:
    [profile osis-pipeline-credentials]
    credential_process = </path/to/aws_signing_helper> credential-process       --certificate </path/to/certificate.pem>       --private-key </path/to/private_key.pem>       --trust-anchor-arn <trusted-anchor-arn>       --profile-arn <profile-arn>       --role-arn <ingestion-role-arn>

    Configure Fluent Bit

    Run the following command to create a Fluent Bit configuration. Replace the placeholders with your specific values. Find the osis-pipeline-endpoint and pipeline-ingestion-path values in your CloudFormation stack outputs.

    cat << 'EOF' > ~/fluent-bit.conf
    [INPUT]
      name                  tail
      path                  /var/log/syslog
      read_from_head        true
      refresh_interval      5
    [OUTPUT]
      name 			http
      match	   		*
      aws_service 		osis
      host 			<osis-pipeline-endpoint>
      port 			443
      uri 			<pipeline-ingestion-path> 
      format 		json
      aws_auth	 	true
      aws_region 		<aa-example-1>   
      aws_profile 		osis-pipeline-credentials
      tls 			On
    EOF

    This example configuration includes the following:

    • Uses the tail input plugin to monitor the /var/log/syslog file
    • Uses the http output plugin to flush log records to the OpenSearch Ingestion pipeline endpoint
    • Uses the osis-pipeline-credentials profile to obtain temporary AWS credentials for SigV4 authentication (aws_auth set to true)

    Test the solution

    Follow these steps to test the setup:

    1. Start the Fluent Bit client with the configuration file fluent-bit.conf that you created in the previous step. Replace the placeholder with the value applicable to your environment. For Ubuntu 24.04, the default path of the Fluent Bit client is /opt/fluent-bit/bin/fluent-bit. Adjust the path if using other distributions.

    sudo AWS_CONFIG_FILE=~/.aws/config <path-to-fluent-bit> -c ~/fluent-bit.conf

    1. Because the solution in this post launched the OpenSearch Service domain within a VPC, you will need an environment that has connectivity to the VPC. For this post, we create a CloudShell VPC environment to run the commands in the next step. Find the VPC, subnet, and security group to use from your CloudFormation stack outputs.
    2. The solution that you deployed through AWS CloudFormation dynamically creates indexes based on ingestion timestamps, format logs-%{yyyy.MM.dd}. You can specify your preferred naming using OpenSearch Ingestion index management. You can query your OpenSearch index using your preferred tool to see the ingested logs from Fluent Bit. We use awscurl in a CloudShell environment as shown in the following example. Replace the placeholders with your specific values. Find the opensearch-domain-endpoint value in your CloudFormation stack outputs.
    pip install awscurl
    
    export OPENSEARCH_DOMAIN_ENDPOINT=https://<opensearch-domain-endpoint>
    
    # List indices matching logs-%{yyyy.MM.dd} format and get most recent one to query
    export INDEX=$(awscurl --service es "$OPENSEARCH_DOMAIN_ENDPOINT/_cat/indices?v" | grep -E "logs-[0-9]{4}\.[0-9]{2}\.[0-9]{2}" | sort -r | head -1 | awk '{print $3}')
    
    awscurl --service es $OPENSEARCH_DOMAIN_ENDPOINT/$INDEX/_search \
            -X GET -H "Content-Type: application/json" \
            -d '{
                "size": 10,
                "sort": [
                  {"@timestamp": {"order": "desc"}}
                ],
                "query": { "match_all": {} }
              }' | jq '.hits.hits[]._source'

    The following is an example of the expected output:

    {
      "date": 1732039662.399506,
      "log": "2024-11-19T18:07:42.399375+00:00 test-server fluent-bit[9986]: 200 OK",
      "@timestamp": "2024-11-19T18:07:42.812Z"
    }
    {
      "date": 1732039662.399501,
      "log": "2024-11-19T18:07:42.399224+00:00 test-server fluent-bit[9986]: [2024/11/19 18:07:42] [ info] [output:http:http.0] test-pipeline-123456789012.us-east-2.osis.amazonaws.com:443, HTTP status=200",
      "@timestamp": "2024-11-19T18:07:42.812Z"
    }
    ...

    Clean up

    To avoid future charges, remove the deployed resources:

    1. Delete the CloudFormation stack.
    2. Remove generated files from CloudShell:

    rm cert_export.json encrypted_private_key.pem certificate.pem private_key.pem

    Conclusion

    In this post, we demonstrated how to obtain temporary credentials from IAM Roles Anywhere and securely ingest logs from hybrid or multicloud environments into OpenSearch Service using OpenSearch Ingestion. This approach minimizes the risk of credential exposure while enabling centralized log collection from distributed workloads. This solution is particularly valuable for organizations managing complex infrastructures across multiple environments and looking to consolidate observability data in OpenSearch Service. For additional details, refer to the following resources:

    If you have questions or feedback about this post, please leave them in the comments section.


    About the Authors

    Xiaoxue Xu is a Solutions Architect for AWS based in Toronto. She primarily works with financial services customers to help secure their workload and design scalable solutions on the AWS Cloud.

    Simran Singh is a Senior Solutions Architect at AWS. In this role, he assists our large enterprise customers in meeting their key business objectives using AWS. His areas of expertise include artificial intelligence and machine learning, security, and improving the experience of developers building on AWS.