Tag Archives: Advanced (300)

Real-time CDC from Aurora PostgreSQL to Amazon S3 Tables using Debezium and Firehose

Post Syndicated from Chintan Agrawal original https://aws.amazon.com/blogs/big-data/real-time-cdc-from-aurora-postgresql-to-amazon-s3-tables-using-debezium-and-firehose/

Enterprises running transactional workloads on Amazon Aurora PostgreSQL-Compatible Edition (Aurora PostgreSQL) need their operational data available for analytics. However, analytical queries and cross-database joins compete for resources on OLTP-optimized clusters. Batch exports introduce latency, and when data spans multiple Aurora clusters, there’s no straightforward way to join datasets or run cross-domain analytics. Real-time change data capture (CDC) addresses this by streaming row-level changes into a separate analytics layer. However, most CDC approaches write append-only records that require downstream consumers to reconstruct current state from the change log.

In this post, we show you how to build a CDC pipeline that delivers query-ready Iceberg tables directly. The pipeline captures inserts, updates, and deletes from Aurora PostgreSQL and applies them as row-level operations in Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3). The destination tables always reflect the current state of the source database. You use Debezium on Amazon MSK Connect for change capture and Amazon Managed Streaming for Apache Kafka (Amazon MSK) for streaming. You also use AWS Lambda to transform CDC events and resolve operation semantics, and Amazon Data Firehose to deliver records into Iceberg tables. You deploy the infrastructure using the AWS Cloud Development Kit (AWS CDK).

Apache Iceberg supports row-level updates, deletes, ACID transactions, schema evolution, and time travel natively. S3 Tables handles Iceberg snapshot management and compaction automatically. With AWS Lake Formation for access control, multiple teams can query the tables through Amazon Athena, Amazon Redshift, or Amazon SageMaker Unified Studio.

Solution overview

The following diagram shows the architecture of the CDC pipeline.

Figure 1. CDC pipeline architecture from Aurora PostgreSQL to Amazon S3 Tables.

Figure 1. CDC pipeline architecture from Aurora PostgreSQL to Amazon S3 Tables.

The pipeline uses six components:

  1. Aurora PostgreSQL to Debezium. Debezium runs on MSK Connect in your VPC and uses PostgreSQL’s native logical replication to stream row-level changes from the write-ahead log (WAL), with minimal impact on query performance.
  2. Debezium to Amazon MSK. The ByLogicalTableRouter SMT reroutes CDC events from multiple tables into a single topic (aurora.cdc.all-tables), retaining the source table name in each message.
  3. Amazon MSK to Firehose. Firehose connects to the MSK cluster using the IAM access control over AWS PrivateLink and continuously polls the topic for new messages.
  4. Firehose to Lambda. For each batch, Firehose invokes the Lambda function to decode the Kafka message, flatten the Debezium envelope, and set otfMetadata routing with the destination table and operation type.
  5. Firehose to S3 Tables. Firehose reads the otfMetadata, routes each record to the correct Iceberg table, and performs the appropriate row-level operation using configured unique keys (for example, order_id for orders). S3 Tables handles compaction and snapshot management automatically.
  6. Query and access control. After data lands in S3 Tables, you can query the Iceberg tables with Amazon Athena, Amazon Redshift, or Amazon SageMaker Unified Studio, with AWS Lake Formation managing fine-grained access control.

Firehose supports one MSK topic per delivery stream. The single-topic routing pattern uses a Debezium SMT to consolidate multiple tables into one topic, and a Lambda function to route records to the correct destination. With this, you can serve multiple tables through one Firehose stream, reducing cost and operational complexity.

Debezium event transformation

Debezium produces CDC events in an envelope structure containing both the previous and current state of a row, along with metadata about the source database, table, and operation type. However, Firehose expects records in a flattened JSON format with routing metadata that indicates the target table and operation type.

The Lambda function bridges this gap by performing three operations on each record:

  1. Decode. When Firehose uses Amazon MSK as a source, it delivers the Kafka message value as a base64-encoded string in the kafkaRecordValue field. The function base64-decodes this field to obtain the raw Debezium JSON payload.
  2. Flatten and extract. Pulls the row data from the Debezium envelope. For inserts and updates, the function uses the after field (the row after the change). For deletes, it uses the before field, because the after field is null when a row is removed.
  3. Route. Sets the otfMetadata block with destinationTableName (extracted from the Debezium source.table field) and operation (mapped from Debezium’s single-character codes to Firehose’s operation types).

The following table shows how Debezium operation codes map to Firehose Iceberg operations:

Debezium code Meaning Firehose operation
c Row created (insert) insert
u Row updated update
d Row deleted delete
r Snapshot read (initial load) insert

When Debezium starts with snapshot.mode=initial, it reads all existing rows and emits them as r (read) events. These represent rows that existed before CDC began, so they are mapped to insert to establish the baseline state in the destination tables.

For example, the function transforms this Debezium envelope:

{
"op": "c",
"before": null,
"after": {"order_id": 1, "customer_id": 1, "total_amount": 299.99},
"source": {"table": "orders", "db": "cdcdemo"}
}

Into a response record with routing metadata:

{
"recordId": "<original-record-id>",
"result": "Ok",
"kafkaRecordValue": "<base64-encoded flattened row JSON>",
"metadata": {
"otfMetadata": {
"destinationDatabaseName": "aurora_cdc",
"destinationTableName": "orders",
"operation": "insert"
}
}
}

The kafkaRecordValue contains the base64-encoded flattened row data (for example, {"order_id": 1, "customer_id": 1, "total_amount": 299.99}), and the otfMetadata block tells Firehose which table to write to and which operation to perform.

With this routing metadata, a single Firehose stream can write to multiple destination tables. For more information, see Route incoming records to different Iceberg tables.

Walkthrough

The following sections walk you through building the CDC pipeline end to end. Before you begin, complete the prerequisites.

Prerequisites

Before you begin, make sure you have the following:

Step 1: Enable CDC in Aurora PostgreSQL

PostgreSQL supports change data capture through its logical replication framework, which allows database changes to be streamed from the write-ahead log (WAL). Debezium uses this mechanism to continuously read row-level changes and publish them to Kafka topics.

To enable logical replication in Aurora PostgreSQL, configure a custom DB cluster parameter group:

  1. Create a custom parameter group and set the following parameter: rds.logical_replication = 1.
  2. Apply the parameter group to your Aurora cluster and reboot the cluster for the change to take effect.
  3. Connect to your Aurora PostgreSQL cluster and create the source tables:
CREATE TABLE public.orders (
    order_id SERIAL PRIMARY KEY,
    customer_id INTEGER,
    order_date VARCHAR(50),
    total_amount DECIMAL(12,2),
    status VARCHAR(50),
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);
CREATE TABLE public.products (
    product_id SERIAL PRIMARY KEY,
    product_name VARCHAR(255),
    category VARCHAR(100),
    price DECIMAL(10,2),
    stock_quantity INTEGER,
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);
  1. Create a publication that defines which tables are included in the change stream. Debezium automatically creates the logical replication slot when the connector starts for the first time, so you don’t need to create one manually.
CREATE PUBLICATION dbz_publication FOR TABLE public.orders, public.products;
  1. Verify the publication was created:
SELECT * FROM pg_publication WHERE pubname = 'dbz_publication';

You should see one row returned, confirming the publication is active.

Important: When the Debezium connector starts (Step 6), it creates a replication slot named debezium_slot. This slot retains WAL segments until consumed. If the connector is stopped for an extended period, WAL segments can accumulate and increase storage usage on the Aurora cluster. Monitor the ReplicationSlotDiskUsage Amazon CloudWatch metric for your Aurora cluster.

Step 2: Build and register the Debezium plugin

MSK Connect runs connectors using custom plugins that you upload to Amazon S3. In this step, you download the Debezium PostgreSQL connector, package it as a ZIP file, upload it to S3, and register it with MSK Connect.

First, create an S3 bucket for the plugin, or use an existing metadata management bucket:

aws s3 mb s3://<your-plugin-bucket> --region <your-region>

Download and package the Debezium connector:

DEBEZIUM_VERSION=2.7.3.Final
curl -LO "https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/${DEBEZIUM_VERSION}/debezium-connector-postgres-${DEBEZIUM_VERSION}-plugin.tar.gz"
mkdir -p debezium-plugin
tar -xzf debezium-connector-postgres-${DEBEZIUM_VERSION}-plugin.tar.gz -C debezium-plugin/
cd debezium-plugin && zip -r ../debezium-postgres-connector.zip . && cd ..
aws s3 cp debezium-postgres-connector.zip s3://<your-plugin-bucket>/plugins/

Register the plugin with MSK Connect:

aws kafkaconnect create-custom-plugin \
    --custom-plugin-name debezium-postgres-connector \
    --content-type ZIP \
    --location "s3Location={bucketArn=arn:aws:s3:::<your-plugin-bucket>,fileKey=plugins/debezium-postgres-connector.zip}"

Create a worker configuration that tells MSK Connect to serialize Kafka messages as JSON without schemas:

aws kafkaconnect create-worker-configuration \
    --name debezium-worker-config \
    --properties-file-content "$(echo -n 'key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false' | base64)"

Note the customPluginArn and workerConfigurationArn from the output. You need these for the CDK configuration in the next step.

Note: The custom plugin and worker configuration are created through the AWS CLI because the Debezium connector JARs must be downloaded from the Debezium project and packaged manually. The remaining infrastructure is deployed using the AWS CDK in the following steps.

Step 3: Configure the CDK project

Clone the sample repository and install dependencies:

git clone https://github.com/aws-samples/sample-aurora-cdc-s3tables.git
cd sample-aurora-cdc-s3tables/cdk
npm install

Open cdk/lib/v2/config.ts and update the configuration values to match your environment:

export const CONFIG = {
account: '<your-account-id>',
region: '<your-region>',
// VPC - must match your Aurora cluster's VPC
vpcId: '<your-vpc-id>',
subnetIds: ['<subnet-1>', '<subnet-2>'],
auroraSecurityGroupId: '<aurora-security-group-id>',
// Aurora connection details
auroraEndpoint: '<aurora-cluster-endpoint>',
auroraPort: '5432',
auroraDbName: '<database-name>',
auroraUser: '<db-user>',
auroraSecretArn: '<secrets-manager-arn>',
// Debezium - use the ARNs from Step 2
debeziumPluginArn: '<customPluginArn-from-step-2>',
debeziumWorkerConfigArn: '<workerConfigurationArn-from-step-2>',
debeziumPluginBucket: '<your-plugin-bucket-name>',
debeziumTopicPrefix: 'aurora.cdc',
debeziumTables: 'public.orders,public.products',
// S3 Tables - the table bucket name must be globally unique
s3TablesBucketName: '<your-table-bucket-name>',
s3TablesNamespace: 'aurora_cdc',
tables: ['orders', 'products'],
tableKeys: { orders: 'order_id', products: 'product_id' },
// Firehose - general purpose S3 bucket for failed record backup
firehoseBackupBucket: '<your-backup-bucket-name>',
};

Key configuration notes:

  • auroraSecurityGroupId. The security group attached to your Aurora cluster. The CDK creates an MSK security group with ingress rules allowing traffic from this security group, and a reverse rule allowing MSK Connect workers to reach Aurora on port 5432.
  • tableKeys. The primary key column for each table. Firehose uses these to match incoming records against existing rows for update and delete operations in the Iceberg tables.
  • s3TablesBucketName. The name for your S3 table bucket. Table bucket names must be unique for your account in the chosen Region.

Step 4: Deploy the CDK stacks

Deploy all six stacks with a single command. The CDK resolves the dependency order automatically:

npx cdk --app "npx ts-node bin/app-v2.ts" deploy --all

When prompted, review the AWS Identity and Access Management (IAM) changes and confirm the deployment. The CDK deploys the following stacks:

Stack What it creates
CdcMskCluster Amazon MSK cluster (2x kafka.m5.large brokers) with dual authentication (IAM for Firehose, unauthenticated for Debezium), custom configuration with auto.create.topics.enable=true, security groups with ingress rules for Aurora and MSK Connect workers
CdcMskConnectIam MSK Connect service execution role with permissions for Kafka cluster operations, VPC networking, S3 plugin access, and AWS Secrets Manager; Amazon CloudWatch Logs group for connector logs
CdcS3Tables S3 table bucket, aurora_cdc namespace, two Iceberg tables (orders, products) with column schemas
CdcLambdaTransform Lambda function for CDC event transformation and multi-table routing
CdcFirehoseRole Firehose IAM role with permissions for Amazon MSK, S3 Tables, AWS Glue Data Catalog, AWS Lake Formation, VPC networking, and Lambda invocation
CdcFirehose Firehose delivery stream with MSK as source (private connectivity through AWS PrivateLink), Lambda processing, Apache Iceberg Tables as destination with two table configurations, and S3 backup bucket for failed records

The MSK cluster takes approximately 25 minutes to create. The Debezium connector takes approximately 5 minutes after the cluster is ready. You can monitor the deployment progress in the AWS CloudFormation console.

After the deployment completes, you can verify the resources in the AWS console. The S3 table bucket shows the two Iceberg tables in the aurora_cdc namespace.

Figure 2. S3 table bucket showing the orders and products Iceberg tables in the aurora_cdc namespace.

Figure 2. S3 table bucket showing the orders and products Iceberg tables in the aurora_cdc namespace.

The Firehose delivery stream shows the MSK source, Lambda transformation, and Apache Iceberg Tables destination.

Figure 3. Amazon Data Firehose delivery stream with MSK source, Lambda transformation, and Apache Iceberg Tables destination.

Figure 3. Amazon Data Firehose delivery stream with MSK source, Lambda transformation, and Apache Iceberg Tables destination.

The MSK cluster uses dual authentication (IAM for Firehose, unauthenticated for Debezium through TLS_PLAINTEXT), multi-VPC private connectivity for Firehose PrivateLink access, and auto.create.topics.enable=true so Debezium can create topics on first connect. VPC connectivity and the cluster resource policy are configured as CLI steps in Step 5.

Step 5: Enable MSK VPC connectivity, grant Lake Formation permissions, and apply MSK cluster policy

After the CDK deployment completes, enable multi-VPC private connectivity with IAM on the MSK cluster. Firehose requires this to create an AWS PrivateLink endpoint to the MSK brokers. This setting can’t be configured during cluster creation and must be applied as an update, which triggers a rolling broker restart (approximately 20–30 minutes).

# Get the cluster ARN and current version from the CdcMskCluster stack outputs
MSK_ARN=<msk-cluster-arn>
CLUSTER_VERSION=$(aws kafka describe-cluster-v2 \
    --cluster-arn $MSK_ARN \
    --region <your-region> \
    --query 'ClusterInfo.CurrentVersion' --output text)
# Enable VPC connectivity with IAM
aws kafka update-connectivity \
    --cluster-arn $MSK_ARN \
    --current-version $CLUSTER_VERSION \
    --connectivity-info '{"VpcConnectivity":{"ClientAuthentication":{"Sasl":{"Iam":{"Enabled":true}}}}}' \
    --region <your-region>

Wait for the cluster state to return to ACTIVE before proceeding:

aws kafka describe-cluster-v2 \
    --cluster-arn $MSK_ARN \
    --region <your-region> \
    --query 'ClusterInfo.State'

Next, grant the Firehose IAM role permissions through AWS Lake Formation. S3 Tables uses a sub-catalog format for the CatalogId parameter, which differs from the standard AWS Glue Data Catalog. These permissions require a data lake administrator identity.

Grant database-level and table-level permissions to the Firehose role:

# Grant database-level permissions
aws lakeformation grant-permissions \
    --region <your-region> \
    --principal '{"DataLakePrincipalIdentifier": "<firehose-role-arn>"}' \
    --resource '{"Database": {"CatalogId": "<account-id>:s3tablescatalog/<table-bucket-name>", "Name": "aurora_cdc"}}' \
    --permissions '["ALL"]'
# Grant table-level permissions (wildcard for the tables in the namespace)
aws lakeformation grant-permissions \
    --region <your-region> \
    --principal '{"DataLakePrincipalIdentifier": "<firehose-role-arn>"}' \
    --resource '{"Table": {"CatalogId": "<account-id>:s3tablescatalog/<table-bucket-name>", "DatabaseName": "aurora_cdc", "TableWildcard": {}}}' \
    --permissions '["ALL"]'

Note the CatalogId format: <account-id>:s3tablescatalog/<table-bucket-name>. This is specific to S3 Tables and tells Lake Formation to look up permissions in the S3 Tables catalog rather than the default Glue Data Catalog. For more information, see Integrating Amazon S3 Tables with AWS analytics services.

Next, attach a resource-based policy to the MSK cluster that grants the Firehose service principal permission to create VPC connections:

aws kafka put-cluster-policy \
    --cluster-arn <msk-cluster-arn> \
    --region <your-region> \
    --policy '{
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"Service": "firehose.amazonaws.com"},
"Action": ["kafka:CreateVpcConnection", "kafka:GetBootstrapBrokers", "kafka:DescribeClusterV2"],
"Resource": "<msk-cluster-arn>"
}]
}'

You can find the <msk-cluster-arn> in the CdcMskCluster stack outputs from Step 4, and the <firehose-role-arn> in the CdcFirehoseRole stack outputs.

Step 6: Create the Debezium connector

With the MSK cluster running and Lake Formation permissions in place, create the Debezium connector using the MSK Connect API. The connector reads changes from Aurora PostgreSQL and publishes them to the MSK topic.

Firehose supports only one MSK topic per delivery stream, so each source table would otherwise need its own Firehose stream and VPC connection. To avoid this, the connector uses the Debezium ByLogicalTableRouter Single Message Transform (SMT) to route changes from multiple tables into a single topic (aurora.cdc.all-tables). The Lambda function then uses the source table name in each message to direct records to the correct Iceberg table. This single-topic pattern uses one Firehose stream for multiple tables, reducing cost and operational complexity.

First, retrieve the MSK bootstrap servers from the cluster:

aws kafka get-bootstrap-brokers \
    --cluster-arn <msk-cluster-arn> \
    --region <your-region>

Note the BootstrapBrokerString value (the PLAINTEXT brokers). Then create the connector:

aws kafkaconnect create-connector --cli-input-json '{
"connectorName": "aurora-postgres-debezium-connector",
"kafkaCluster": {
"apacheKafkaCluster": {
"bootstrapServers": "<bootstrap-servers>",
"vpc": {
"subnets": ["<subnet-1>", "<subnet-2>"],
"securityGroups": ["<msk-security-group-id>"]
}
}
},
"kafkaClusterClientAuthentication": {"authenticationType": "NONE"},
"kafkaClusterEncryptionInTransit": {"encryptionType": "PLAINTEXT"},
"kafkaConnectVersion": "2.7.1",
"plugins": [{"customPlugin": {"customPluginArn": "<custom-plugin-arn>", "revision": 1}}],
"serviceExecutionRoleArn": "<msk-connect-service-role-arn>",
"capacity": {"provisionedCapacity": {"mcuCount": 2, "workerCount": 2}},
"workerConfiguration": {"workerConfigurationArn": "<worker-config-arn>", "revision": 1},
"connectorConfiguration": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"tasks.max": "1",
"database.hostname": "<aurora-cluster-endpoint>",
"database.port": "5432",
"database.user": "<db-user>",
"database.password": "<db-password>",
"database.dbname": "<database-name>",
"database.server.name": "aurora_cdc",
"plugin.name": "pgoutput",
"slot.name": "debezium_slot",
"publication.name": "dbz_publication",
"table.include.list": "public.orders,public.products",
"topic.prefix": "aurora.cdc",
"schema.history.internal.kafka.topic": "schema-changes.aurora",
"schema.history.internal.kafka.bootstrap.servers": "<bootstrap-servers>",
"decimal.handling.mode": "string",
"time.precision.mode": "adaptive_time_microseconds",
"tombstones.on.delete": "false",
"snapshot.mode": "initial",
"publication.autocreate.mode": "filtered",
"transforms": "Reroute",
"transforms.Reroute.type": "io.debezium.transforms.ByLogicalTableRouter",
"transforms.Reroute.topic.regex": "aurora\\\\\\\\.cdc\\\\\\\\.public\\\\\\\\.(.*)",
"transforms.Reroute.topic.replacement": "aurora.cdc.all-tables"
},
"logDelivery": {
"workerLogDelivery": {
"cloudWatchLogs": {
"enabled": true,
"logGroup": "/aws/msk-connect/aurora-cdc-debezium"
}
}
}
}'

The <msk-security-group-id> and <msk-connect-service-role-arn> can be found in the CdcMskCluster and CdcMskConnectIam stack outputs respectively. The ByLogicalTableRouter Single Message Transform routes CDC events from the monitored tables into a single topic (aurora.cdc.all-tables).

Step 7: Verify the Debezium connector

After creating the connector, verify that it is running and has completed its initial snapshot.

aws kafkaconnect list-connectors --region <your-region> \
    --query 'connectors[?connectorName==`aurora-postgres-debezium-connector`].{Name:connectorName,State:connectorState}' \
    --output table

The connector state should show RUNNING, as shown in the following figure.

Figure 4. Debezium connector running on Amazon MSK Connect.

Figure 4. Debezium connector running on Amazon MSK Connect.

Check the CloudWatch Logs to confirm the snapshot completed:

aws logs tail /aws/msk-connect/aurora-cdc-debezium --follow --region <your-region>

You should see messages indicating the transition to streaming mode:

Finished exporting 0 records for table 'public.orders' (1 of 2 tables)
Finished exporting 0 records for table 'public.products' (2 of 2 tables)
Snapshot completed
Starting streaming

If the tables were empty when the connector started, the export count is 0. If you had existing data, the snapshot captures the existing rows as r (read) operations, which the Lambda function maps to insert operations in the Iceberg tables.

Verify that the Firehose delivery stream is active:

aws firehose describe-delivery-stream \
    --delivery-stream-name msk-to-s3tables-firehose \
    --region <your-region> \
    --query 'DeliveryStreamDescription.DeliveryStreamStatus'

The status should return ACTIVE.

Step 8: Test the pipeline

Insert test data into the Aurora PostgreSQL source tables. Each insert triggers a CDC event that flows through the pipeline: Aurora WAL to Debezium to MSK topic to Firehose to Lambda transform to S3 Tables.

-- Insert orders
INSERT INTO public.orders (customer_id, order_date, total_amount, status)
VALUES
(1, '2026-01-20', 299.99, 'shipped'),
(2, '2026-01-21', 149.50, 'processing'),
(1, '2026-01-22', 89.99, 'delivered');
-- Insert products
INSERT INTO public.products (product_name, category, price, stock_quantity)
VALUES
('Wireless Headphones', 'Electronics', 79.99, 150),
('Running Shoes', 'Sports', 129.99, 75),
('Coffee Maker', 'Kitchen', 49.99, 200);

This creates six records across two tables. Each record generates a Debezium CDC event with operation type c (create), which the Lambda function maps to an insert operation in the corresponding Iceberg table.

Step 9: Verify data delivery

Check the Firehose IncomingRecords metric to confirm records are flowing through the delivery stream:

aws cloudwatch get-metric-statistics \
    --namespace AWS/Firehose \
    --metric-name IncomingRecords \
    --dimensions Name=DeliveryStreamName,Value=msk-to-s3tables-firehose \
    --start-time $(date -u -v-10M +%Y-%m-%dT%H:%M:%S) \
    --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
    --period 60 --statistics Sum \
    --region <your-region>

You should see a Sum value of 6 or more. If the value is 0, wait another minute and retry. There can be a short delay between MSK topic delivery and Firehose metric reporting.

If records aren’t appearing, check the Firehose error output in the backup S3 bucket and the Lambda function’s CloudWatch Logs for transformation errors.

Step 10: Query data using Amazon Athena

With data delivered to S3 Tables, you can query the Iceberg tables using Amazon Athena. S3 Tables integrates with the AWS Glue Data Catalog as a sub-catalog, so you reference tables using the S3 Tables catalog format.

Tip: If records aren’t appearing in Athena, check the Firehose IncomingRecords CloudWatch metric and the Lambda function’s CloudWatch Logs for transformation errors.

Open the Athena console, select the AwsDataCatalog data source, and run the following queries:

SELECT * FROM "s3tablescatalog/<table-bucket-name>"."aurora_cdc"."products" LIMIT 10;
SELECT * FROM "s3tablescatalog/<table-bucket-name>"."aurora_cdc"."orders" LIMIT 10;

Replace <table-bucket-name> with your S3 table bucket name. You should see the records from the initial snapshot that Debezium captured when the connector started.

The following figures show the initial state of both tables as queried through Athena. At this point, the products table contains seven records and the orders table contains seven records, captured during the Debezium initial snapshot.

Figure 5. Initial state of the products table in Amazon Athena, showing seven records captured from Aurora PostgreSQL through the CDC pipeline.

Figure 5. Initial state of the products table in Amazon Athena, showing seven records captured from Aurora PostgreSQL through the CDC pipeline.

Figure 6. Initial state of the orders table in Amazon Athena, showing seven records captured from Aurora PostgreSQL through the CDC pipeline.

Figure 6. Initial state of the orders table in Amazon Athena, showing seven records captured from Aurora PostgreSQL through the CDC pipeline.

Now test that update and delete operations propagate correctly. Run the following statements in Aurora:

-- Insert new records
INSERT INTO public.products (product_name, category, price, stock_quantity)
VALUES ('Bluetooth Speaker', 'Electronics', 129.99, 90), ('Standing Desk', 'Furniture', 799.99, 20);
INSERT INTO public.orders (customer_id, order_date, total_amount, status)
VALUES (201, '2026-04-03', 149.99, 'NEW'), (202, '2026-04-03', 249.50, 'NEW'), (203, '2026-04-03', 79.90, 'NEW');
-- Update existing records
UPDATE public.products SET stock_quantity = 30, price = 549.99 WHERE product_name = 'Ergonomic Chair';
UPDATE public.orders SET status = 'DELIVERED' WHERE order_id = 201;
-- Delete a record
DELETE FROM public.products WHERE product_name = 'Test Widget';

Wait for the changes to propagate through the pipeline, then query Athena again. The following figures show the results after the insert, update, and delete operations have been applied.

In the products table, the Test Widget record (product_id 100) is no longer present because it was removed by the delete operation. The Ergonomic Chair row now reflects the updated price (549.99) and stock quantity (30). Two new records, Bluetooth Speaker and Standing Desk, appear with a later created_at timestamp, confirming they were inserted after the initial snapshot.

Figure 7. Products table after CDC operations. The Ergonomic Chair, Headphones, and Desk Lamp rows reflect updated values. Bluetooth Speaker and Standing Desk are newly inserted records. The Test Widget record has been removed by the delete operation.

Figure 7. Products table after CDC operations. The Ergonomic Chair, Headphones, and Desk Lamp rows reflect updated values. Bluetooth Speaker and Standing Desk are newly inserted records. The Test Widget record has been removed by the delete operation.

In the orders table, order 100 now shows a status of SHIPPED and order 201 shows DELIVERED, reflecting the update operations. Three new orders (301, 302, 303) appear with status NEW and a later timestamp, confirming they were inserted after the initial load.

Figure 8. Orders table after CDC operations. Orders 100 and 201 reflect updated status values. Orders 301, 302, and 303 are newly inserted records.

Figure 8. Orders table after CDC operations. Orders 100 and 201 reflect updated status values. Orders 301, 302, and 303 are newly inserted records.

This confirms that the pipeline correctly handles the three CDC operation types: inserts, updates, and deletes are captured from the Aurora WAL by Debezium, routed through the single MSK topic, transformed by the Lambda function, and applied as row-level Iceberg operations by Firehose.

S3 Tables handles compaction and snapshot management for Iceberg tables automatically, including compaction of small data files and expiration of old snapshots. You don’t need to run manual maintenance operations.

You can also use Iceberg’s time travel capability to query the table as it existed before the updates:

SELECT * FROM "s3tablescatalog/<table-bucket-name>"."aurora_cdc"."orders"
FOR TIMESTAMP AS OF current_timestamp - interval '5' minute;

This returns the original data before the update, demonstrating the time travel capability that Apache Iceberg provides through S3 Tables.

Cleaning up

To avoid ongoing charges, delete the resources in reverse dependency order.

Delete the CDK stacks:

cd cdk
npx cdk --app "npx ts-node bin/app-v2.ts" destroy --all

Delete the Debezium custom plugin and worker configuration that were created through the AWS CLI in Step 2:

aws kafkaconnect delete-custom-plugin --custom-plugin-arn <plugin-arn>
aws kafkaconnect delete-worker-configuration --worker-configuration-arn <worker-config-arn>

Clean up the Aurora PostgreSQL replication resources:

SELECT pg_drop_replication_slot('debezium_slot');
DROP PUBLICATION dbz_publication;

Important: The replication slot (debezium_slot) was created automatically by Debezium. If you plan to redeploy the pipeline later, you don’t need to drop the slot and publication. However, the replication slot continues to retain WAL segments while the connector isn’t running, which can increase storage usage on the Aurora cluster. The MSK cluster is the largest cost component of this solution and can’t be paused. It can only be deleted and recreated.

Conclusion

In this post, we showed you how to build a near real-time CDC pipeline from Aurora PostgreSQL to Apache Iceberg tables in Amazon S3 Tables. The key architectural decisions include:

  • Single-topic routing with multi-table delivery. The Debezium ByLogicalTableRouter SMT routes CDC events from multiple tables through one MSK topic, and the Lambda otfMetadata routing directs each record to the correct Iceberg table. This reduces VPC connection costs by using a single Firehose stream for inserts, updates, and deletes across multiple destination tables.
  • Fully managed CDC pipeline. MSK Connect runs Debezium, Firehose handles delivery with automatic retries, and S3 Tables manages Iceberg compaction and snapshots. The Lambda transform preserves CDC semantics by mapping Debezium operations to Iceberg row-level operations.
  • Governed lakehouse access. Lake Formation controls fine-grained access to the Iceberg tables, and data from multiple isolated Aurora clusters can be unified in a single S3 Tables namespace for cross-domain analytics.
  • Infrastructure as code. Six AWS CDK stacks deploy the core pipeline, with Lake Formation permissions, MSK cluster policy, and Debezium connector configured through documented CLI steps.

To get started, clone the sample repository and follow the walkthrough steps. For more information about the services used in this solution, see the Amazon MSK Developer Guide, Amazon Data Firehose Developer Guide, and Amazon S3 Tables User Guide.

We encourage you to try this solution and adapt it to your own CDC workloads. If you have questions or feedback, leave a comment on this post.


About the author

Chintan Agrawal

Chintan Agrawal

Chintan is a Solutions Architect with over 7 years of experience, with a specialization in Analytics and Healthcare domain. He possesses a strong enthusiasm for assisting clients in discovering valuable insights from their data. Through his expertise, he constructs innovative solutions that empower businesses to arrive at informed, data-driven choices.

Upgrade PySpark from Spark 3.5 to Spark 4.0 with AWS Spark Upgrade Agent

Post Syndicated from Prasad Nadig original https://aws.amazon.com/blogs/big-data/upgrade-pyspark-from-spark-3-5-to-spark-4-0-with-aws-spark-upgrade-agent/

Upgrading Apache Spark applications across major versions means tracking down breaking changes, manually debugging failures from log files, and running repeated test cycles. This process can stretch across weeks for complex code bases.

In this post, we walk through a hands-on PySpark migration from Spark 3.5 to Spark 4.0 on Amazon EMR Serverless, using the AWS Spark Upgrade Agent. You’ll see how the agent iteratively validates your application on a live Amazon EMR Serverless application, automatically diagnosing and resolving failures from Amazon CloudWatch logs until the job succeeds. By the end, you have a multi-pipeline PySpark application running on Spark 4.0 with four distinct breaking changes resolved. The fixes include configuration key removals, codec renames, and stricter charset validation, all driven through natural language interaction in the Integrated Development Environment (IDE).

This is part 2 of a three-part series on how the AWS Spark Upgrade Agent can automate and simplify Spark upgrades.

In Part 1, we introduced the agent’s architecture and capabilities. This post walks through a complete PySpark migration from Spark 3.5 to Spark 4.0 on Amazon EMR Serverless.

In the sections that follow, you will set up the prerequisites and infrastructure, explore the sample application, run the iterative validation workflow on EMR Serverless, review data quality results, and generate a comprehensive upgrade summary.

Note: Because this upgrade is performed using the AWS Spark Upgrade Agent Model Context Protocol (MCP) server, an agentic artificial intelligence (AI) system, the agent might take different paths to reach the same successful outcome. The workflow demonstrated here represents one successful upgrade path. The key takeaway is the end-to-end workflow: generating an upgrade plan, iteratively validating on Amazon EMR Serverless, and producing a comprehensive upgrade summary.

1. Prerequisites and setup

This section covers the tools, infrastructure, and IDE configuration you need before starting the upgrade. To follow along, you need an AWS account with an AWS Identity and Access Management (AWS IAM) user or role that has permissions to deploy AWS CloudFormation stacks, create AWS IAM roles and policies, and create Amazon EMR Serverless applications. Intermediate knowledge of AWS Command Line Interface (AWS CLI), AWS CloudFormation, and Python is helpful.

1.1 Install Kiro CLI and local tools

In this post, we use Kiro CLI to demonstrate the upgrade workflow. You can use an MCP-compatible IDE or framework. Examples include VS Code with Cline, Cursor, Windsurf, and Claude Desktop, among others. To follow along with Kiro CLI, install it on your workstation. For more details on the installation and setup, refer to Setup for Upgrade Agent:

curl -fsSL https://cli.kiro.dev/install | bash

Run the following command and use your builder ID to log in:

kiro-cli login --use-device-flow

With the Kiro CLI installed and logged in, rather than installing the remaining tools manually, use Kiro CLI to set up and verify your prerequisites with the following prompt:

kiro-cli chat
> Install AWS CLI, Python 3.10, and uv on my system if they are not already installed

Kiro CLI output showing successful installation of AWS CLI, Python, and uv

Output of AWS CLI and local tools install step.

These tools are needed for the upgrade workflow:

1.2 Infrastructure setup (AWS CloudFormation)

Two AWS CloudFormation stacks create the required resources: an AWS IAM role, an Amazon Simple Storage Service (Amazon S3) staging bucket, an Amazon EMR Serverless application (Spark 4.0.1), and its execution role.

Stack 1 – AWS IAM role and Amazon S3 staging bucket:

The spark-upgrade-mcp-setup template creates the AWS IAM role and Amazon S3 staging bucket required by the upgrade agent. Choose the Launch Stack button for your Region. For additional Regions, see the full region list.

# Region Launch
1 US East (N. Virginia) Launch Stack
2 US East (Ohio) Launch Stack
3 US West (Oregon) Launch Stack
4 Europe (Ireland) Launch Stack

After deployment, open the AWS CloudFormation Outputs tab, copy the ExportCommand value, and run it in your terminal. This sets SMUS_MCP_REGION, IAM_ROLE, and STAGING_BUCKET_PATH automatically.

CloudFormation Outputs tab showing ExportCommand with SMUS_MCP_REGION, IAM_ROLE, and STAGING_BUCKET_PATH values

Outputs tab of the CloudFormation stack.

# Sets SMUS_MCP_REGION, IAM_ROLE, and STAGING_BUCKET_PATH
export SMUS_MCP_REGION=<YOUR-REGION> && export IAM_ROLE=arn:aws:iam::<YOUR-ACCOUNT-ID>:role/spark-upgrade-role-* && export STAGING_BUCKET_PATH=<YOUR-BUCKET>

Then configure the AWS CLI profile:

aws configure set profile.spark-upgrade-profile.role_arn ${IAM_ROLE}
aws configure set profile.spark-upgrade-profile.source_profile default
aws configure set profile.spark-upgrade-profile.region ${SMUS_MCP_REGION}

Stack 2 – Amazon EMR Serverless target application and execution role:

git clone https://github.com/aws-samples/sample-amazon-emr-spark4-examples
cd sample-amazon-emr-spark4-examples/pyspark/AWSSpark4AutoUpgradeDemo

The PySpark sample lives at resources/global_logistics_platform/. The AWS CloudFormation template lives at resources/cloudformation/.

Deploy the AWS CloudFormation template to create the source and target Amazon EMR Serverless applications and a shared execution role:

aws cloudformation deploy \
  --template-file resources/cloudformation/emr-serverless-target-setup.yaml \
  --stack-name spark-emr-serverless-upgrade \
  --region ${SMUS_MCP_REGION} \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
    StagingBucketName=${STAGING_BUCKET_PATH} \
    SourceReleaseLabel=emr-7.0.0 \
    TargetReleaseLabel=emr-spark-8.0-preview \
    SourceApplicationName=spark-upgrade-source \
    TargetApplicationName=spark-upgrade-target

This creates two Amazon EMR Serverless applications: a source (Spark 3.5.0) for data quality baseline and a target (Spark 4.0.1) for upgrade validation, with a shared execution role. Both applications auto-stop after 15 minutes of idle time, so there is no cost when not in use. To upgrade between different Spark versions, override SourceReleaseLabel and TargetReleaseLabel with your target Amazon EMR release labels.

After the stack completes deployment, note the outputs:

aws cloudformation describe-stacks \
  --stack-name spark-emr-serverless-upgrade \
  --region ${SMUS_MCP_REGION} \
  --query "Stacks[0].Outputs" --output table

This gives you the SourceApplicationId, TargetApplicationId, and ExecutionRoleArn needed for the upgrade prompt. Make a note of them.

1.3 IDE and MCP server configuration

Configure the spark-upgrade MCP server. For Kiro CLI:

kiro-cli-chat mcp add \
    --name "spark-upgrade" \
    --command "uvx" \
    --args '[
      "mcp-proxy-for-aws@latest",
      "https://sagemaker-unified-studio-mcp.'${SMUS_MCP_REGION}'.api.aws/spark-upgrade/mcp",
      "--service", "sagemaker-unified-studio-mcp",
      "--profile", "spark-upgrade-profile",
      "--region", "'${SMUS_MCP_REGION}'",
      "--read-timeout", "180"
    ]' \
    --timeout 180000 \
    --scope global

For other MCP clients, refer to your IDE’s MCP configuration documentation and use the same server parameters shown previously.

Verify the connection: Start Kiro CLI and confirm the spark-upgrade tools are loaded:

$ kiro-cli chat
...
spark-upgrade (MCP):
- generate_spark_upgrade_plan          * not trusted
- update_build_configuration           * not trusted
- fix_upgrade_failure                  * not trusted
- run_validation_job                   * not trusted
- check_job_status                     * not trusted
...

Tip: After Kiro CLI and the MCP server are configured, you can ask the agent to verify your setup. For example: “Check if I have AWS CLI, Python 3.10+, and uv installed, and confirm the spark-upgrade MCP server is connected.”

Kiro CLI output confirming spark-upgrade MCP server connection and tool availability

Output showing the status of each tool, AWS CLI, and MCP server.

Tip: Trust mode vs. confirm mode: When running the upgrade agent in Kiro CLI, you have two options:

Trust mode: Type t when prompted to approve a tool. The agent auto-approves subsequent uses of that tool without asking for confirmation. You can also use /tools trust-all to trust every tool at once for a fully autonomous experience.

Confirm mode: Type y for each individual tool invocation. This lets you review, verify, and approve every action before the agent runs it. If this is your first time using the agent, use confirm mode for full visibility.

2. Hands-on PySpark upgrade from Spark 3.5 to Spark 4.0

This section walks through the complete migration of a representative PySpark application from Amazon EMR Serverless 7.0.0 (Spark 3.5.0) to EMR Serverless with the emr-spark-8.0-preview release label (Spark 4.0.1), using the global_logistics_platform sample.

2.1 Sample project: global logistics platform

The sample application is a multi-domain PySpark data processing application with three pipelines:

  • Fleet management: Processes vehicle telemetry data (GPS tracking, fuel consumption, driver behavior scoring) using window functions, lag/lead operations, and statistical aggregations. Writes Parquet with lz4raw compression.
  • International shipping: Handles cross-border shipment documents with multi-language address standardization using character encoding functions (encode/decode with charsets like Shift_JIS, GB2312, EUC-KR), and processes carrier manifests with ISO-8859-1 encoding.
  • Historical compliance: Processes regulatory audit records spanning centuries (including pre-1582 Julian calendar dates), requiring legacy datetime rebasing for Parquet writes.

Project structure:

global_logistics_platform/
├── main.py                          # Orchestrator - runs all 3 pipelines
├── src/
│   ├── utils/
│   │   └── spark_config.py          # Spark session config & logging
│   └── domain/                      # Application code that needs migration
│       ├── fleet_management/
│       │   └── telemetry_processor.py
│       ├── international_shipping/
│       │   └── shipment_processor.py
│       └── historical_compliance/
│           └── compliance_processor.py
└── data/                             # Sample dataset for the workflow
    └── sample/
        ├── fleet_telemetry.csv
        ├── international_shipments.csv
        └── compliance_records.csv

2.2 The four Spark 4.0 incompatibilities

Before diving into the upgrade, here are the four specific breaking changes present in this code base that the agent discovers and resolves entirely through runtime validation:

# Incompatibility File(s)
1 Legacy Parquet configuration key removed: spark.sql.legacy.parquet.datetimeRebaseModeInWrite removed in Spark 4.0. Must use spark.sql.parquet.datetimeRebaseModeInWrite. spark_config.py
2 Parquet compression codec rename: lz4raw codec renamed to lz4_raw in Spark 4.0. telemetry_processor.py
3 Stricter charset encoding validation: Spark 4.0 tightened encode() behavior. Encoding CJK (Chinese, Japanese, Korean) characters to ISO-8859-1 now throws MALFORMED_CHARACTER_CODING. In Spark 3.x this silently replaced unmappable chars with ?. Restored via spark.sql.legacy.codingErrorAction. spark_config.py
4 Character encoding restrictions: encode()/decode() in Spark 4.0 supports US-ASCII, ISO-8859-1, UTF-8, UTF-16BE, UTF-16LE, UTF-16, and UTF-32. Code uses Shift_JIS, GB2312, EUC-KR. shipment_processor.py

The agent resolves each of these through iterative runtime validation on EMR Serverless: submitting the job, diagnosing failures from Amazon CloudWatch logs, applying fixes, and resubmitting until the job succeeds.

Architecture diagram showing the iterative validation workflow between the IDE, MCP server, and Amazon EMR Serverless

2.3 Step 1: Invoke the upgrade agent

Open the project in Kiro CLI and enter the following prompt:

Upgrade my Spark application in the current directory from EMR serverless version 7.0.0 to EMR serverless version 8.0.0.
Use Amazon EMR Serverless target app-id <YOUR-TARGET-APP-ID> and execution role
<YOUR-EXECUTION-ROLE-ARN> for validation.
Use source Amazon EMR Serverless app-id <YOUR-SOURCE-APP-ID> for data quality baseline.
Store artifacts at s3://${STAGING_BUCKET_PATH}/spark4-upgrade/python/
Enable data quality validation

Tip: The SourceApplicationId, TargetApplicationId, and ExecutionRoleArn are in the Outputs of the spark-emr-serverless-upgrade AWS CloudFormation stack you deployed in Section 1.2.

The agent invokes generate_spark_upgrade_plan, scans the project structure, identifies the Spark version mapping (EMR 7.0.0 → Spark 3.5.0, EMR 8.0.0 → Spark 4.0.1), and produces a structured upgrade plan with an Analysis ID for traceability.

The agent presents the plan and asks for confirmation. Type y to approve the tool invocation, or t to trust that tool for the rest of the session.

You have an option to save the plan as a local JSON file for future reference or to resume the upgrade at a later point, so go ahead and ask Kiro to save it locally. Provide the AWS CLI profile that you have configured on your system. Use the following prompt to provide these inputs:

Yes I would like to save the plan to a local file and use spark-upgrade-profile

2.4 Step 2: Build and package

The agent validates the Python project compiles successfully, then packages it for Amazon EMR Serverless deployment:

  • Runs py_compile on each .py file to verify syntax.
  • Creates src.zip containing the src/ directory (preserving the import structure used by from src.utils import ...).
  • Uploads src.zip, main.py, and sample input data to the Amazon S3 staging path.
# What the agent does behind the scenes:
zip -r src.zip src/
aws s3 cp main.py s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/source/main.py
aws s3 cp src.zip s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/source/src.zip
aws s3 cp data/sample/ s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/input/ --recursive

No external dependencies (no requirements.txt), so no virtual environment is needed. If your project has external dependencies in a requirements.txt, the agent will package them into a virtual environment archive and include it in the EMR Serverless submission parameters.

2.5 Step 3: Data quality baseline on source application

Before migrating the code, the agent establishes a data quality baseline by running the original (pre-upgrade) code on the source Amazon EMR Serverless application (Spark 3.5.0 / EMR 7.0.0). This captures the expected output that the upgraded application must match.

The agent submits the job to the source application with data quality check enabled:

{
  "executionRoleArn": "arn:aws:iam::<YOUR-ACCOUNT-ID>:role/<YOUR-EXECUTION-ROLE>",
  "jobDriver": {
    "sparkSubmit": {
      "entryPoint": "s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/source/main.py",
      "entryPointArguments": [
        "s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/input/",
        "s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/output/source/"
      ],
      "sparkSubmitParameters": "--py-files s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/source/src.zip"
    }
  },
  "configurationOverrides": {
    "monitoringConfiguration": {
      "cloudWatchLoggingConfiguration": {
        "enabled": true,
        "logGroupName": "/aws/emr-serverless"
      }
    }
  }
}

The agent monitors the source run via check_job_status until it completes successfully. This baseline output is stored for comparison after the target validation succeeds.

2.6 Step 4: Iterative runtime validation on target application

This is the core of the upgrade. The agent submits the unmodified application to the target Amazon EMR Serverless application (Spark 4.0.1), and every incompatibility is discovered, diagnosed, and fixed through runtime failures. The agent drives the entire fix cycle by submitting to EMR, reading errors from Amazon CloudWatch logs, applying fixes, rebuilding, and resubmitting.

The agent presents the proposed Amazon EMR Serverless job configuration for your review before each submission. Type y to approve.

2.6.1 Fix 1: Legacy Parquet configuration key removed (iteration 1)

The first submission fails immediately at SparkSession initialization:

org.apache.spark.sql.AnalysisException:
The SQL config 'spark.sql.legacy.parquet.datetimeRebaseModeInWrite' was removed
in the version 4.0.0. Use 'spark.sql.parquet.datetimeRebaseModeInWrite' instead.

The Historical Compliance pipeline configures spark.sql.legacy.parquet.datetimeRebaseModeInWrite for handling pre-1582 Julian calendar dates. Spark 4.0 removed the legacy. prefix from this configuration key.

The agent calls fix_upgrade_failure, which identifies the migration rule and recommends the fix:

File: src/utils/spark_config.py

# Before
.config("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "LEGACY")

# After
.config("spark.sql.parquet.datetimeRebaseModeInWrite", "LEGACY")

After applying the fix, the agent rebuilds src.zip, re-uploads to Amazon S3, and resubmits the job.

2.6.2 Fix 2: Parquet compression codec rename (iteration 2)

The resubmitted job fails with a new error, which confirms progress:

pyspark.errors.exceptions.captured.IllegalArgumentException:
[CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION]
The codec lz4raw is not available.
Available codecs are brotli, uncompressed, lzo, snappy, lz4_raw, none, zstd, lz4, gzip.
SQLSTATE: 56038

The Fleet Management pipeline’s telemetry_processor.py uses lz4raw as the Parquet compression codec. Spark 4.0 renamed this to lz4_raw (with an underscore).

The recommended fix:

File: src/domain/fleet_management/telemetry_processor.py

# Before
.option("compression", "lz4raw")

# After
.option("compression", "lz4_raw")

The agent applies the change, rebuilds, and resubmits.

2.6.3 Fix 3: Stricter charset encoding validation (iteration 3)

The next submission surfaces a different failure:

org.apache.spark.SparkRuntimeException:
[MALFORMED_CHARACTER_CODING]
Invalid value found when performing `encode` with ISO-8859-1
SQLSTATE: 22000

The International Shipping pipeline’s process_carrier_manifests() method uses encode(..., 'ISO-8859-1') on data containing CJK (Chinese, Japanese, Korean) characters. Although ISO-8859-1 is in Spark 4.0’s supported charset list, it is a single-byte encoding that cannot represent CJK characters. In Spark 3.x, the Java charset encoder silently replaced unmappable characters with ?. Spark 4.0 tightened this behavior to throw MALFORMED_CHARACTER_CODING for unmappable characters.

The agent identifies the migration rule and adds a legacy compatibility configuration:

File: src/utils/spark_config.py

# Added to SparkSession builder
.config("spark.sql.legacy.codingErrorAction", "true")

This restores the Spark 3.x behavior where unmappable characters are silently replaced instead of throwing errors.

With the configuration added, the agent rebuilds and resubmits.

2.6.4 Fix 4: Character encoding restrictions (iteration 4)

The fourth submission fails with yet another encoding error:

org.apache.spark.SparkIllegalArgumentException:
[INVALID_PARAMETER_VALUE.CHARSET]
The value of parameter(s) `charset` in `encode` is invalid:
expects one of the iso-8859-1, us-ascii, utf-16, utf-16be, utf-16le, utf-32, utf-8,
but got Shift_JIS. SQLSTATE: 22023

The International Shipping pipeline’s standardize_addresses_with_charset() method uses Shift_JIS, GB2312, and EUC-KR charsets in encode()/decode() calls. Spark 4.0 restricts these functions to seven standard charsets. These regional charsets are not in the supported list.

The agent replaces the unsupported charsets with UTF-8:

File: src/domain/international_shipping/shipment_processor.py

Before (Spark 3.5.0):

df = df.withColumn(
    "shipper_address_normalized",
    when(col("origin_country") == "JP",
         expr("decode(encode(shipper_address, 'Shift_JIS'), 'UTF-8')"))
    .when(col("origin_country") == "CN",
         expr("decode(encode(shipper_address, 'GB2312'), 'UTF-8')"))
    .when(col("origin_country") == "KR",
         expr("decode(encode(shipper_address, 'EUC-KR'), 'UTF-8')"))
    .otherwise(col("shipper_address"))
)

After (Spark 4.0.1):

df = df.withColumn(
    "shipper_address_normalized",
    when(col("origin_country") == "JP",
         expr("decode(encode(shipper_address, 'UTF-8'), 'UTF-8')"))
    .when(col("origin_country") == "CN",
         expr("decode(encode(shipper_address, 'UTF-8'), 'UTF-8')"))
    .when(col("origin_country") == "KR",
         expr("decode(encode(shipper_address, 'UTF-8'), 'UTF-8')"))
    .otherwise(col("shipper_address"))
)

The same transformation is applied to consignee_address_normalized.

The agent rebuilds and resubmits one final time.

2.6.5 Final submission: success

The fifth submission completes successfully:

{"success": true, "message": "EMR SERVERLESS job completed successfully",
"compute_run_id": "<JOB-RUN-ID>", "status": "SUCCESS",
"application_type": "EMR-Serverless"}

The three pipelines (Fleet Management, International Shipping, and Historical Compliance) complete on EMR Serverless with the emr-spark-8.0-preview release label (Spark 4.0.1).

2.7 Summary of the iterative runtime validation

The runtime validation loop is the core value of the upgrade agent. Here’s the complete iteration history:

Table showing the four validation iterations with error types and fixes applied

Each iteration follows the same cycle:

Diagram showing the submit, diagnose, fix, rebuild, and resubmit cycle

Failures that would normally require manual log analysis, root cause investigation, and code patching are resolved automatically by the agent in this workflow.

3. Data quality validation

With both the source baseline (Section 2.5) and the upgraded target run (Section 2.6) completed successfully, the agent performs data quality validation to verify the migration hasn’t changed your application’s output. This is the key advantage of including the source application in your upgrade prompt: the agent can compare outputs from both Spark versions side by side.

3.1 Data quality comparison

The agent invokes get_data_quality_summary to compare the outputs across four dimensions:

  • Schema validation: Confirms column names, data types, and column ordering match between source and target outputs.
  • Row count validation: Verifies no data loss or duplication during migration.
  • Nullability validation: Detects changes in null handling.
  • Statistical summary validation: Compares numeric and string column distributions (min, max, mean, count, distinct values).

The agent presents the comparison results:

Data quality summary showing schema, row count, and nullability checks passing with a statistical mismatch in shipper_address

The preceding image shows the data quality summary.

Three of four checks pass cleanly. The statistical summary validation detects a mismatch in the shipper_address column of the customs_declarations output: the max and min summary values differ between source and target.

3.2 Understanding and resolving the mismatch

This mismatch is a direct consequence of Fix 4 (Section 2.6.4). The original code ran addresses through a Shift_JIS/GB2312/EUC-KR → UTF-8 roundtrip that produced garbled text, because the intermediate regional charset corrupted multi-byte UTF-8 characters. The upgraded code uses UTF-8 → UTF-8, preserving addresses faithfully. The mismatch reflects improved data quality, not a regression.

Schema, row counts, and nullability matched exactly: the difference is limited to string values that were previously garbled. No further action is needed. The upgraded application is production-ready.

Expected behavior: Character encoding migrations might change string values, although they preserve semantic meaning. When data quality validation reports mismatches, trace each one back to a specific code change. If the mismatch is explained by a required migration fix (as here), verify the new behavior is correct and document it. If a mismatch cannot be explained, investigate before promoting to production.

4. Upgrade summary

After the agent completes the entire upgrade workflow, it produces a comprehensive upgrade summary following a structured template. This summary lets you review the job configuration updates, code modifications with diffs and file references, relevant migration rules applied, and data quality validation status.

Here is the summary the agent produced for this upgrade:

Upgrade plan

  • Compile and build project with current Spark 3.5.0: validated that Python files compile successfully.
  • Run baseline validation on source EMR Serverless (00g4vhvt1lhtrs09) with Spark 3.5.0: established data quality baseline.
  • Run target validation on target EMR Serverless (00g4vhvt3np1bj09) with Spark 4.0.1: fixed 4 issues iteratively across 4 validation attempts.
  • Compare data quality between source and target runs: detected expected mismatch in shipper_address.
  • Generate and persist upgrade summary.

Upgrade result

Upgrade completed with data validation enabled. Data validation detected an expected mismatch in the shipper_address column because of the charset encoding migration from unsupported charsets (Shift_JIS, GB2312, EUC-KR) to UTF-8.

Dependency changes

No external dependencies were changed in this project (no requirements.txt).

Job configuration changes

  • Parquet datetime rebase configuration key renamed.
    • Change: spark.sql.legacy.parquet.datetimeRebaseModeInWrite → spark.sql.parquet.datetimeRebaseModeInWrite.
    • Migration rule: In Spark 4.0, the legacy datetime rebasing SQL configurations with the prefix spark.sql.legacy are removed. The SQL configuration spark.sql.legacy.parquet.datetimeRebaseModeInWrite was removed in the version 4.0.0. Use spark.sql.parquet.datetimeRebaseModeInWrite instead.
  • Legacy coding error action enabled.
    • Change: Added spark.sql.legacy.codingErrorAction set to true.
    • Migration rule: In Spark 4.0, the encode() and decode() functions raise MALFORMED_CHARACTER_CODING error when handling unmappable characters. In Spark 3.5 and earlier versions, these characters are replaced with garbled text. To restore the previous behavior, set spark.sql.legacy.codingErrorAction to true.

Code changes

  • Validation attempt 1: Legacy Parquet configuration key.
    • Validation run: EMR-Serverless job_run_id 00g4vm14v118vg0b.
    • Error: The SQL config 'spark.sql.legacy.parquet.datetimeRebaseModeInWrite' was removed in the version 4.0.0.
    • Applied changes: src/utils/spark_config.py: Changed .config("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "LEGACY") to .config("spark.sql.parquet.datetimeRebaseModeInWrite", "LEGACY").
  • Validation attempt 2: Parquet compression codec.
    • Validation run: EMR-Serverless job_run_id 00g4vm5pm1hig00b.
    • Error: [CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION] The codec lz4raw is not available.
    • Applied changes: src/domain/fleet_management/telemetry_processor.py: Changed .option("compression", "lz4raw") to .option("compression", "lz4_raw").
  • Validation attempt 3: Stricter charset encoding.
    • Validation run: EMR-Serverless job_run_id 00g4vm8sh4sp0g0b.
    • Error: [MALFORMED_CHARACTER_CODING] Invalid value found when performing encode with ISO-8859-1.
    • Applied changes: src/utils/spark_config.py: Added .config("spark.sql.legacy.codingErrorAction", "true") to the SparkSession builder.
  • Validation attempt 4: Unsupported charsets.
    • Validation run: EMR-Serverless job_run_id 00g4vmc668ng6o0b.
    • Error: [INVALID_PARAMETER_VALUE.CHARSET] charset in encode is invalid: expects one of iso-8859-1, us-ascii, utf-16, utf-16be, utf-16le, utf-32, utf-8, but got Shift_JIS.
    • Applied changes: src/domain/international_shipping/shipment_processor.py: Replaced Shift_JIS, GB2312, EUC-KR with UTF-8 for shipper and consignee address encoding.

Data validation result

# Validation Status
1 Schema validation (column names, types, ordering) Passed (no difference)
2 Row count validation (no data loss) Passed (no difference)
3 Nullability validation (null handling changes) Passed (no difference)
4 Statistical summary validation (numeric/string distributions) Failed (with difference)

Data mismatch: 1. The shipper_address column max summary value changed in customs_declarations output. This is expected because of the charset encoding migration from Shift_JIS/GB2312/EUC-KR to UTF-8. 2. The shipper_address column min summary value changed in customs_declarations output for the same expected cause.

5. Conclusion

The AWS Spark Upgrade Agent turns a traditionally time-consuming PySpark migration into an automated, iterative workflow. For the Global Logistics Platform sample, the agent identified and resolved four distinct Spark 4.0 breaking changes: legacy Parquet configuration key removal, compression codec renames, stricter charset encoding validation, and character encoding restrictions. Each fix was applied across three domain processors, through natural language interaction in the IDE.

Every incompatibility was discovered through runtime validation on Amazon EMR Serverless. The agent submitted the unmodified application to the target application, and each failure revealed the next breaking change:

  • The spark.sql.legacy.parquet.datetimeRebaseModeInWrite configuration removal, which crashes SparkSession initialization.
  • The lz4raw → lz4_raw codec rename, which fails when Parquet writes run.
  • ISO-8859-1 encoding of CJK characters: ISO-8859-1 is a valid Spark 4.0 charset, so the failure surfaces only when the code runs against real multi-language data, because Spark 4.0 tightened charset encoding validation to reject unmappable characters.
  • Shift_JIS/GB2312/EUC-KR charsets removed from Spark 4.0’s supported charset list entirely.

The agent diagnosed each error from Amazon CloudWatch logs, applied the fix, rebuilt, and resubmitted without manual intervention beyond approving each step. The data quality validation then confirmed that the upgraded application produces equivalent output on Spark 4.0.1: schema, row counts, and nullability matched exactly. The one difference, in the shipper_address column, resulted from the charset migration from regional encodings to UTF-8, which actually improved data quality by eliminating garbled text from incorrect encoding roundtrips. With each mismatch traced back to a specific, understood code change, the upgraded application is production-ready.

# Category Spark 3.x behavior Spark 4.0 change Agent fix
1 Parquet datetime configuration spark.sql.legacy.parquet.datetimeRebaseModeInWrite legacy. prefix removed from key name Update configuration key
2 Parquet compression lz4raw codec name Renamed to lz4_raw (with underscore) Update codec name
3 Charset + CJK data ISO-8859-1 silently replaced unmappable CJK chars with ? Stricter charset validation throws MALFORMED_CHARACTER_CODING for unmappable characters Add spark.sql.legacy.codingErrorAction=true
4 Character encoding encode()/decode() supported Java charsets Restricted to 7 standard charsets Replace unsupported charsets with UTF-8

Next steps after your first upgrade:

  1. Apply the agent to your production PySpark code base.
  2. Integrate the upgrade workflow into your CI/CD pipeline.
  3. Explore Scala application upgrades (see Part 3 of this series).

To get started with your own PySpark migration:

  • Deploy the AWS CloudFormation templates from Section 1.2 for one-time AWS IAM, Amazon S3, and Amazon EMR Serverless setup.
  • Configure the spark-upgrade MCP server in your MCP-compatible IDE.
  • Point the agent at your PySpark project and let it handle the rest.

For more information, see the Amazon EMR Serverless documentation, the Apache Spark 4.0 migration guide, and the AWS Spark Upgrade Agent setup guide.

6. Clean up resources

To avoid ongoing costs, delete the resources you created:

  1. Delete the Amazon EMR Serverless stack:
    aws cloudformation delete-stack --stack-name spark-emr-serverless-upgrade --region ${SMUS_MCP_REGION}

  2. Delete the AWS IAM and Amazon S3 staging stack:
    aws cloudformation delete-stack --stack-name spark-upgrade-mcp-setup --region ${SMUS_MCP_REGION}

  3. If the Amazon S3 staging bucket contains objects, empty it before deleting the stack:
    aws s3 rm s3://${STAGING_BUCKET_PATH} --recursive


About the authors

Prasad Nadig

Prasad Nadig

Prasad Nadig is a Senior Analytics Specialist Solutions Architect at AWS, specializing in data and AI, including data lakes, data warehousing, and analytics services such as Amazon Redshift, Amazon EMR, and AWS Glue. He helps customers architect, migrate, and modernize their data and analytics workloads to achieve scalable, performant, and cost-effective solutions on AWS.

Karthik Prabhakar

Karthik Prabhakar

Karthik is a Data Processing Engines Architect for Amazon EMR at Amazon Web Services (AWS). He specializes in distributed systems architecture and query optimization, working with customers to solve complex performance challenges in large-scale data processing workloads. His focus spans engine internals, cost-optimization strategies, and architectural patterns that enable customers to run petabyte-scale analytics efficiently.

Bezuayehu Wate

Bezuayehu Wate

Bezuayehu is a Specialist Solutions Architect at AWS, specializing in big data analytics and AI solutions. She works closely with customers to modernize analytics platforms using AWS data and AI services. With a passion for emerging technologies and customer success, she thrives on designing innovative cloud solutions that deliver measurable business impact and drive organizational transformation.

Chuhan Liu

Chuhan Liu

Chuhan is a Software Development Engineer at AWS.

Keerthi Chadalavada

Keerthi Chadalavada

Keerthi is a Senior Software Development Engineer in the AWS analytics organization. She focuses on combining generative AI and data integration technologies to design and build comprehensive solutions for customer data and analytics needs.

Pradeep Patel

Pradeep Patel

Pradeep is a Sr. Software Engineer at AWS Glue. He is passionate about helping customers solve their problems by using the power of the AWS Cloud to deliver highly scalable and robust solutions. In his spare time, he loves to hike and play with web applications.

Migrate JMS applications to Amazon MQ for RabbitMQ with minimal changes

Post Syndicated from Vinodh Kannan Sadayamuthu original https://aws.amazon.com/blogs/big-data/migrate-jms-applications-to-amazon-mq-for-rabbitmq-with-minimal-changes/

Running JMS applications on on-premises brokers or Apache ActiveMQ requires manual patching cycles, capacity planning for peak loads, and maintaining high availability across multiple data centers. With Amazon MQ version 4 and above, you can migrate your existing JMS applications without rewriting your messaging layer, removing weeks of rewrite work.

This post shows you how to migrate your JMS applications and walks through a complete setup, from creating the broker to sending and receiving messages. You will also see a real-world scenario: migrating an existing Apache ActiveMQ workload to an Amazon MQ broker running RabbitMQ. The post covers configuration changes, monitoring with Amazon CloudWatch, and validation steps to make sure that your migration succeeds.

Amazon MQ version 4 and above includes built-in support for the RabbitMQ JMS Client and the JMS Topic Exchange plugin. The RabbitMQ JMS Client and JMS Topic Exchange plugin work together, allowing your existing JMS applications to connect using familiar JMS APIs. You update the connection factory configuration and broker endpoint. Your business logic, message producers, consumers, and listeners stay exactly as written.

Understanding JMS and AMQP

How the RabbitMQ JMS Client works

Use the RabbitMQ JMS Client to connect your Java application to Amazon MQ. The client translates your JMS API calls (javax.jms or jakarta.jms) into AMQP 0-9-1 messages that the broker understands.

Advanced Message Queuing Protocol (AMQP) defines how messages are formatted and transmitted across the network at the wire level. This means that non-Java services can consume the same messages using native AMQP clients, making the protocol language-agnostic

Architecture diagram showing JMS to AMQP translation: Java and Spring applications use the JMS API and RabbitMQ JMS Client Library to communicate with an Amazon MQ broker running RabbitMQ via the AMQP 0-9-1 protocol.

JMS version support

Migrate at the JMS version that your application already uses. The client supports JMS 1.1, 2.0, and 3.1 (Jakarta Messaging), so you don’t need to upgrade your application code before migrating brokers. The client integrates with Spring Framework and Spring Boot applications without requiring custom bean factories or application context configuration.

Because the JMS abstraction layer sits between your application and the broker, most migrations require only a connection factory change, not a logic rewrite.

RabbitMQ JMS Topic Exchange plugin

Your existing publish/subscribe patterns work without client-side routing logic. The JMS Topic Exchange plugin adds server-side support for JMS topic semantics, handling topic routing and SQL-based message selection directly in the broker.

The plugin handles SQL-based message selection (JMS selectors like OrderType = `Electronics` AND Priority > 5) and topic hierarchies with wildcard pattern matching (* for single level, # for multiple levels). Your application uses standard JMS topic APIs (createTopic(), setMessageSelector()) without additional filtering logic.

Getting started

This walkthrough shows you how to set up Amazon MQ and connect your existing JMS application. You will create a broker, configure the connection factory, and send and receive messages.

Prerequisites

You need an existing JMS application built on Apache ActiveMQ or another JMS provider to migrate. If you don’t have one, you can still follow Steps 1–5 to create a broker and test the connection pattern. Before you begin, confirm that you have the following in place:

  • An active AWS account
  • AWS Command Line Interface (AWS CLI) installed. For instructions, see Installing the AWS CLI.
  • Java 11 or later installed on your local development environment.
  • An AWS Identity and Access Management (IAM) principal (user or role) with the AmazonMQFullAccess managed policy attached.
  • Maven or Gradle for dependency management.

Amazon MQ broker charges apply based on instance type and usage. Review the Amazon MQ pricing page before you start.

Step 1: Create an Amazon MQ for RabbitMQ broker

The following command creates a single-instance broker running RabbitMQ 4.2 on an mq.m7g.medium instance.

aws mq create-broker \ 
--broker-name my-rabbitmq-broker \ 
--engine-type rabbitmq \ 
--engine-version 4.2 \ 
--deployment-mode SINGLE_INSTANCE \ 
--host-instance-type mq.m7g.medium \ 
--auto-minor-version-upgrade \ 
--publicly-accessible \ 
--users "Username=admin,Password=[PASSWORD]" \ 
--region us-west-2

Replace <broker-name> with the name that you want to give to the broker. Replace <username> and <password> as described in the create-broker CLI documentation. After the command runs successfully, the command line displays the BrokerArn and BrokerId.

Note: This command creates a publicly accessible broker for demonstration purposes only. For production workloads, create brokers in private subnets within your VPC and restrict access using security groups. Don’t use the –publicly-accessible flag. For more information, see Security best practices for Amazon MQ.

The command returns output similar to:

{
    "BrokerArn": "arn:aws:mq:us-west-2:111122223333:broker:my-rabbitmq-broker:b-c8352341-ec91-4a78-ad9c-a57f23f235bb",
    "BrokerId": "b-c8352341-ec91-4a78-ad9c-a57f23f235bb"
}

Save the BrokerId value for the next step.

The broker takes approximately 15–20 minutes to reach the Running state. Run the following command every 2 minutes to check the status:

aws mq describe-broker --broker-id <BrokerId> --region us-west-2 --query 'BrokerState' 

Proceed to the next step after the broker state is RUNNING.

To get the broker endpoints, run:

aws mq describe-broker --broker-id <BrokerId> --region us-west-2 --query 'BrokerInstances'

Note the ConsoleURL and Endpoints from the output. The command returns output similar to:

[{
    "ConsoleURL": "https:// b-c8352341-ec91-4a78-ad9c-a57f23f235bb.mq.us-west-2.on.aws",
    "Endpoints": ["amqps://b-c8352341-ec91-4a78-ad9c-a57f23f235bb.mq.us-west-2.on.aws:5671"]
}]

Step 2: Add the RabbitMQ JMS Client dependency

Choose the dependency that matches your application’s current JMS version. If your imports reference javax.jms packages, use version 2.12.0. If your imports reference jakarta.jms packages (JMS 3.1 / Jakarta EE 9+), use version 3.4.0.

For JMS 1.1 and 2.0 (javax.jms):

<dependency>
<groupId>com.rabbitmq.jms</groupId>
<artifactId>rabbitmq-jms</artifactId>
<version>2.12.0</version>
</dependency>

For JMS 3.1 / Jakarta JMS 3.1 / Jakarta Messaging (jakarta.jms):

<dependency>
<groupId>com.rabbitmq.jms</groupId>
<artifactId>rabbitmq-jms</artifactId>
<version>3.4.0</version>
</dependency>

Step 3: Configure the connection factory

Store your broker credentials in AWS Secrets Manager before configuring the connection factory. This keeps credentials out of your source code and configuration files.

Create the secret:

aws secretsmanager create-secret \
--name dev-rabbitmq \
--description "Amazon MQ broker credentials" \
--secret-string '{"username":"admin","password":"[PASSWORD]"}' \ 
--region us-west-2

Add the AWS SDK for Secrets Manager to your pom.xml:

<dependency> 
<groupId>software.amazon.awssdk</groupId> 
<artifactId>secretsmanager</artifactId> 
<version>2.20.0</version> </dependency>

<dependency> 
<groupId>com.fasterxml.jackson.core</groupId> 
<artifactId>jackson-databind</artifactId> 
<version>2.15.0</version> 
</dependency>

Replace your existing broker URL with the Amazon MQ endpoint. In most cases, this is the only change required in your application configuration:

import com.rabbitmq.jms.admin.RMQConnectionFactory;
import software.amazon.awssdk.services.secretsmanager.SecretsManagerClient;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueRequest;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueResponse;
import software.amazon.awssdk.regions.Region;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.Map;
import javax.jms.*; 

// Retrieve credentials from AWS Secrets Manager
Map<String, String> creds;
try (SecretsManagerClient secretsClient = SecretsManagerClient.builder().region(Region.US_WEST_2).build()) {    
    GetSecretValueResponse response = secretsClient.getSecretValue(GetSecretValueRequest.builder().secretId("dev-rabbitmq").build());        
    ObjectMapper objectMapper = new ObjectMapper();    
    creds = objectMapper.readValue(response.secretString(),new TypeReference<Map<String, String>>() {});
    } 

// Create and configure the connection factory
RMQConnectionFactory connectionFactory = new RMQConnectionFactory();
connectionFactory.setHost("b-c8352341-ec91-4a78-ad9c-a57f23f235bb.mq.us-west-2.on.aws");
connectionFactory.setPort(5671);connectionFactory.setVirtualHost("/");
connectionFactory.useSslProtocol();
connectionFactory.setUsername(creds.get("username"));
connectionFactory.setPassword(creds.get("password")); 

Replace the host value with your broker endpoint from Step 1.

The connection factory requires four parameters:

  • Host: Your broker endpoint from the describe-broker output
  • Port: 5671 for AMQP over TLS (Amazon MQ requires encryption in transit)
  • VirtualHost: “/” (the default RabbitMQ virtual host)
  • UseSslProtocol: true (required by Amazon MQ)

Step 4: Send messages

The following examples show how to send messages to a queue and a topic using the JMS 2.0 simplified API.

Point-to-point (queue) for one-to-one delivery:

try (JMSContext context = connectionFactory.createContext()) {
Queue queue = context.createQueue("orders");
context.createProducer().setProperty("OrderType", "Electronics").send(queue, "Order #12345");
System.out.println("Sent message to queue: orders");
}

Publish/subscribe (topic) for one-to-many broadcast:

try (JMSContext context = connectionFactory.createContext()) {
	Topic topic = context.createTopic("orders.electronics");
	context.createProducer().setProperty("MessageType", "Broadcast").send(topic, "New electronics order received!");
	System.out.println("Published message to topic: orders.electronics");
}

Message properties(OrderType, MessageType) are JMS headers that consumers can use for filtering. These properties become AMQP message headers when transmitted to the broker.

Step 5: Receive messages asynchronously

To receive messages asynchronously, attach a MessageListener to a consumer. The listener fires each time a message arrives.

Queue consumer:

Asynchronous consumers process messages in a background thread without blocking your main application logic. The MessageListener callback fires each time a message arrives, allowing your application to handle messages as they’re delivered rather than polling with receive().

try (JMSContext context = connectionFactory.createContext()) {
	Queue queue = context.createQueue("orders");
	JMSConsumer consumer = context.createConsumer(queue);
	consumer.setMessageListener(message -> {
		if (message instanceof TextMessage) {
			try {
				System.out.println("Received: " + ((TextMessage) message).getText());
			} catch (JMSException e) {
				e.printStackTrace();
			}
		}
	});

	System.out.println("Listening for messages on queue: orders");

	// Keep the consumer active for 30 seconds
	Thread.sleep(30000);
}

Topic subscriber:

try (JMSContext context = connectionFactory.createContext()) {
	Topic topic = context.createTopic("orders.electronics");
	JMSConsumer consumer = context.createConsumer(topic);
	consumer.setMessageListener(message -> {
		if (message instanceof TextMessage) {
			try {
				System.out.println("Subscriber received: " + ((TextMessage) message).getText());
			} catch (JMSException e) {
				e.printStackTrace();
			}
		}
	});

	System.out.println("Subscribed to topic: orders.electronics");

	// Keep the consumer active for 30 seconds
	Thread.sleep(30000);}

The Thread.sleep(30000) call keeps the consumer active for 30 seconds.

Use case: Migrating an ActiveMQ Workload to Amazon MQ for RabbitMQ

Migrate your Apache ActiveMQ applications to Amazon MQ by updating four configuration points. Your business logic, message producers, consumers, and listeners stay exactly as written. This walkthrough uses a real JMS 1.1 application with a centralized broker configuration class to show precisely which lines change and which remain identical.

Apache ActiveMQ powers messaging infrastructure for thousands of Java applications worldwide. If you run JMS applications on ActiveMQ, you can migrate to Amazon MQ for RabbitMQ with minimal code changes. The following steps demonstrate a complete migration using an application that includes a centralized broker configuration class, a message producer, and a message consumer.

Step 1: Update the Maven dependency

Replace the ActiveMQ client dependencies with the RabbitMQ JMS client in your pom.xml. The rabbitmq-jms artifact includes the RabbitMQ AMQP client and JMS API as transitive dependencies, so a single entry replaces both ActiveMQ artifacts.

Before (ActiveMQ):

<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-client</artifactId>
<version>5.18.6</version>
</dependency>

<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-pool</artifactId>
<version>5.18.6</version>n>5.18.6</version>
</dependency>

After (Amazon MQ):

<dependency>
<groupId>com.rabbitmq.jms</groupId>
<artifactId>rabbitmq-jms</artifactId>
<version>2.12.0</version>
</dependency>

The rabbitmq-jms artifact pulls in the RabbitMQ AMQP client and the JMS API as transitive dependencies, so a single entry replaces both ActiveMQ artifacts.

Step 2: Update the broker configuration

If your application centralizes connection details in a shared configuration class, that class is the only file that needs to change. The queue name and everything else your application references remain the same.

Before (ActiveMQ):

// BrokerConfig.java - ActiveMQ version
public final class BrokerConfig {
	// OpenWire endpoint
	public static final String BROKER_URL = "tcp://localhost:61616";
	public static final String USERNAME = "[PASSWORD]";
	public static final String PASSWORD = "[PASSWORD]";
	public static final String QUEUE_NAME = "demo.queue";
	private BrokerConfig() {}}

After (Amazon MQ):

// BrokerConfig.java - Amazon MQ version

import com.fasterxml.jackson.core.type.TypeReference;import com.fasterxml.jackson.databind.ObjectMapper;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.secretsmanager.SecretsManagerClient;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueRequest;
import software.amazon.awssdk.services.secretsmanager.model.GetSecretValueResponse;
import java.util.Map;

public final class BrokerConfig {
	// AMQPS endpoint (TLS required by Amazon MQ)
	public static final String BROKER_URL = "amqps://b-c8352341-ec91-4a78-ad9c-a57f23f235bb.mq.us-west-2.on.aws:5671";

	// Queue name carries over unchanged
	public static final String QUEUE_NAME = "demo.queue";

	// Secret name in AWS Secrets Manager
	private static final String SECRET_ID = "dev-rabbitmq";
	private static final Map<String, String> CREDENTIALS = loadCredentials();
	public static String getUsername() {return CREDENTIALS.get("username");}
	public static String getPassword() {return CREDENTIALS.get("password");}
	private static Map<String, String> loadCredentials() { 
		try (SecretsManagerClient client = SecretsManagerClient.builder().region(Region.US_WEST_2).build()) {
			GetSecretValueResponse response = client.getSecretValue(GetSecretValueRequest.builder().secretId(SECRET_ID).build());
			ObjectMapper mapper = new ObjectMapper();
			return mapper.readValue(response.secretString(), new TypeReference<Map<String, String>>() {});
		} catch (Exception e) {
			throw new RuntimeException("Failed to load broker credentials from Secrets Manager", e);
		}
	}
	private BrokerConfig() {}}

Two things changed in this file compared to the ActiveMQ version: the protocol prefix (tcp:// to amqps://) and the host and port (OpenWire on 61616 to AMQP over TLS on 5671). The queue name is identical. Credentials are no longer stored as static string constants. Instead, loadCredentials() retrieves them from AWS Secrets Manager at startup, and getUsername() and getPassword() expose them to the rest of the application. This follows AWS security best practices and streamlines credential rotation.

Step 3: Update the message producer

The producer requires two changes: the import statement and the factory instantiation. Every JMS API call after the factory (createConnection, createSession, createProducer, send) is identical to the ActiveMQ version.

Before (ActiveMQ):

import org.apache.activemq.ActiveMQConnectionFactory;
import javax.jms.*;

public class MessageProducer {
	public static void main(String[] args) {
		Connection connection = null;
		try {
			ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory(BrokerConfig.USERNAME,BrokerConfig.PASSWORD,BrokerConfig.BROKER_URL);
			connection = factory.createConnection();
			connection.start();
			Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
			Destination destination = session.createQueue(BrokerConfig.QUEUE_NAME);
			javax.jms.MessageProducer producer = session.createProducer(destination);
			producer.setDeliveryMode(DeliveryMode.PERSISTENT);
			for (int i = 1; i <= 5; i++) {
				TextMessage message = session.createTextMessage("Hello from ActiveMQ - message #" + i);
				producer.send(message);
				System.out.println("Sent: " + message.getText());
			}
			producer.close();
			session.close();
		} catch (JMSException e) {
			e.printStackTrace();
		} finally {
			if (connection != null) {
				try { 
					connection.close();
				} catch (JMSException ignored) {}}
			}
		}
	}

After (Amazon MQ):

import com.rabbitmq.jms.admin.RMQConnectionFactory;
import javax.jms.*;

public class MessageProducer {

	public static void main(String[] args) {
		Connection connection = null;
		try {
			RMQConnectionFactory factory = new RMQConnectionFactory();
			factory.setUri(BrokerConfig.BROKER_URL);
			factory.setUsername(BrokerConfig.getUsername());
			factory.setPassword(BrokerConfig.getPassword());

			// Everything below this line is identical to the ActiveMQ version
			connection = factory.createConnection();
			connection.start();
			Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
			Destination destination = session.createQueue(BrokerConfig.QUEUE_NAME);
			javax.jms.MessageProducer producer = session.createProducer(destination);
			producer.setDeliveryMode(DeliveryMode.PERSISTENT);
			for (int i = 1; i <= 5; i++) {
				TextMessage message = session.createTextMessage("Hello from Amazon MQ - message #" + i);
				producer.send(message);
				System.out.println("Sent: " + message.getText());
			}
			producer.close();
			session.close();
		} catch (JMSException e) {
			e.printStackTrace();
		} finally {
			if (connection != null) {
				try {
					connection.close();
				} catch (JMSException ignored) {}
			}
		}
	}
}

The import changes from org.apache.activemq.ActiveMQConnectionFactory to com.rabbitmq.jms.admin.RMQConnectionFactory. The factory construction switches from a constructor that accepts credentials and URL to a no-arg constructor with explicit setter calls. Credentials are now retrieved from AWS Secrets Manager through BrokerConfig.getUsername() and BrokerConfig.getPassword(). That is the complete change set for the producer.

Step 4: Update the message consumer

The consumer follows the same pattern as the producer. Swap the factory class and import, update the credential calls, and keep everything else.

Before (ActiveMQ):

import org.apache.activemq.ActiveMQConnectionFactory;
import javax.jms.*;
public class MessageConsumer {
	public static void main(String[] args) {
		Connection connection = null;
		try {
			ActiveMQConnectionFactory factory = new ActiveMQConnectionFactory(BrokerConfig.USERNAME, BrokerConfig.PASSWORD, BrokerConfig.BROKER_URL);
			connection = factory.createConnection();
			connection.start();
			Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
			Destination destination = session.createQueue(BrokerConfig.QUEUE_NAME);
			javax.jms.MessageConsumer consumer = session.createConsumer(destination);
			System.out.println("Waiting for messages on queue: " + BrokerConfig.QUEUE_NAME);
			Message message;
			while ((message = consumer.receive(10000)) != null) {
				if (message instanceof TextMessage) {
					TextMessage textMessage = (TextMessage) message;
					System.out.println("Received: " + textMessage.getText());
				}
			}
			consumer.close();
			session.close();
		} catch (JMSException e) {
			e.printStackTrace();
		} finally {
			if (connection != null) {
				try { 
					connection.close(); 
				} catch (JMSException ignored) {}
			}
		}
	}
}

After (Amazon MQ):

import com.rabbitmq.jms.admin.RMQConnectionFactory;
import javax.jms.*;

public class MessageConsumer {
	public static void main(String[] args) {
		Connection connection = null;
		try {
			RMQConnectionFactory factory = new RMQConnectionFactory();
			factory.setUri(BrokerConfig.BROKER_URL);
			factory.setUsername(BrokerConfig.getUsername());
			factory.setPassword(BrokerConfig.getPassword());

			// Everything below this line is identical to the ActiveMQ version
			connection = factory.createConnection();
			connection.start();
			Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
			Destination destination = session.createQueue(BrokerConfig.QUEUE_NAME);
			javax.jms.MessageConsumer consumer = session.createConsumer(destination);
			System.out.println("Waiting for messages on queue: " + BrokerConfig.QUEUE_NAME);
			Message message;
			while ((message = consumer.receive(10000)) != null) {
				if (message instanceof TextMessage) {
					TextMessage textMessage = (TextMessage) message;
					System.out.println("Received: " + textMessage.getText());
				}
			}
			consumer.close();
			session.close();
		} catch (JMSException e) {
			e.printStackTrace();
		} finally {
			if (connection != null) {
				try { 
					connection.close(); 
				} catch (JMSException ignored) {}
			}
		}
	}
}

The import changes from org.apache.activemq.ActiveMQConnectionFactory to com.rabbitmq.jms.admin.RMQConnectionFactory. The factory construction switches to a no-arg constructor with explicit setter calls, and BrokerConfig.USERNAME / BrokerConfig.PASSWORD are replaced with BrokerConfig.getUsername() / BrokerConfig.getPassword(). The session creation, queue lookup, consumer setup, and message processing loop are identical to the ActiveMQ version.

Configuration

The following table summarizes the changes required when migrating from Apache ActiveMQ.

 

ActiveMQ Amazon MQ for RabbitMQ
Maven dependency activemq-client 5.18.6 rabbitmq-jms 2.12.0
Connection factory class ActiveMQConnectionFactory RMQConnectionFactory
Import package org.apache.activemq com.rabbitmq.jms.admin
Broker URL format tcp://host:61616 amqps://broker-id.mq.region.on.aws:5671
Protocol OpenWire AMQP 0-9-1
Port 61616 (OpenWire) 5671 (AMQP over TLS)
TLS Optional Required
Credentials Plain text / JNDI AWS Secrets Manager (recommended)
Virtual host N/A / (default)
JMS version support JMS 1.1 JMS 1.1, 2.0, 3.1 (Jakarta)
Queue/Topic names demo.queue demo.queue (no change)
JMS API calls Standard JMS 1.1 Standard JMS 1.1 (no change)

Validating the migration

Run your application against Amazon MQ for RabbitMQ in a staging environment before directing production traffic to the new broker. Verify that messages flow correctly, consumers process as expected, and no data loss occurs during cutover.

The RabbitMQ management console provides real-time visibility into broker operations. Access it through the ConsoleURL from your broker details. The console shows queue depths, consumer counts, and message rates. Use it during testing to identify routing or throughput issues before production deployment

The console displays jms.durable.queues and jms.durable.topic exchanges. The JMS client creates these automatically when your application creates queues and topics, so no manual exchange configuration is required.

RabbitMQ management console showing jms.durable.queues and jms.durable.topic exchanges created automatically by the JMS client.

Monitoring with Amazon CloudWatch

Amazon MQ publishes broker metrics to Amazon CloudWatch with no additional configuration needed. This gives you persistent monitoring and alerting that works alongside the rest of your AWS observability setup, beyond what the RabbitMQ management console provides in real time.

After your JMS messages reach the Amazon MQ for RabbitMQ broker, they’re transported as AMQP messages, which means standard RabbitMQ operational best practices apply. Keep queue depth low to avoid memory pressure and consumer lag. Follow message durability and reliability guidelines to prevent message loss during broker restarts. For connection management, review broker setup and connection best practices to avoid connection churn.

Set Amazon CloudWatch alarms on MessageCount and ConnectionCount first. A rising queue depth with a stable or dropping consumer count is an early signal of a processing bottleneck. A sudden drop in connections can indicate a client configuration issue that’s more straightforward to catch before it affects production traffic.

Clean up

To avoid ongoing charges after testing, delete the Amazon MQ broker and Secrets Manager secret using the AWS CLI.

Delete the broker:

aws mq delete-broker --broker-id <your-broker-id> --region us-west-2

Delete the Secrets Manager secret:

aaws secretsmanager delete-secret \
--secret-id dev-rabbitmq \
--force-delete-without-recovery \
--region us-west-2

Broker deletion is permanent and can’t be undone. Amazon MQ removes all messages, configurations, and user credentials. Leaving the broker running incurs hourly charges based on the instance type, plus storage costs for message data retained on the broker.

Conclusion

In this post, we walked you through how to migrate your JMS applications. We also walked through a complete setup, from creating the broker to sending and receiving messages. Migrating the broker is the straightforward part. The more significant question is what you do next. After your JMS application is running on Amazon MQ for RabbitMQ, you have access to native AMQP clients, which means non-Java services can start consuming the same messages without a JMS layer. A Java-centric messaging system becomes a shared event backbone that service can participate in. The migration is a starting point, not just a lift-and-shift.

Next Steps

To get started with your migration, create your first Amazon MQ for RabbitMQ broker. For detailed technical guidance, see the Amazon MQ Developer Guide and explore the RabbitMQ JMS Client documentation.


About the authors


Vinodh Kannan is a Senior Specialist Solutions Architect at Amazon Web Services (AWS). His expertise centers on AWS messaging and streaming services, where he provides architectural best practices consultation to AWS customers.
Akhil Melakunta
Akhil Melakunta is a Senior Solutions Architect at Amazon Web Services (AWS) with over 12 years of industry experience. His expertise spans AI/ML, serverless, and messaging services, where he guides enterprise customers through large-scale cloud transformations on AWS.

Vignesh Selvam is the Principal Product Manager for Amazon MQ at AWS. He works with customers to solve their messaging needs and with the open-source communities for innovating with message brokers. Prior to joining AWS, he built products for security and analytics.

Customize federated sign-in with new Amazon Cognito Lambda trigger

Post Syndicated from Abrom Douglas original https://aws.amazon.com/blogs/security/customize-federated-sign-in-with-new-amazon-cognito-lambda-trigger/

You can use Amazon Cognito user pools to add sign-up and sign-in functionality to your web and mobile applications. You can authenticate users directly with Amazon Cognito managed accounts using passwords, passwordless flows, or custom authentication flows, or let users federate in through external identity providers (IdP) using SAML, OpenID Connect, or social providers such as Google, Facebook, Sign in with Apple, or Login with Amazon. For consumers, identity federation means fewer passwords to remember and a smoother sign-in experience. For business-to-business (B2B) software as a service (SaaS) providers, it means your tenants’ organizations keep control of their own identities rather than managing credentials on their behalf. But federation can also introduce challenges for enterprises and application developers. What happens when your enterprise customer’s SAML provider sends hundreds of group memberships that exceed attribute size limits? Or when your ecommerce customer forgets they already have an account and tries to sign in with a different social provider, creating duplicate records?

In this blog post, I introduce the inbound federation Lambda trigger for Amazon Cognito, a new feature that gives you programmatic control over federated authentication flows. This AWS Lambda trigger intercepts the federated authentication response immediately after your external identity provider responds to Cognito, so you can transform, filter, and enrich user attributes before the user profile is created and user attributes are mapped in your user pool.

Understanding the inbound federation Lambda trigger

The inbound federation Lambda trigger is invoked after your Amazon Cognito user pool has received and verified the response from the external IdP. The request payload for the federated IdP response is then sent from Cognito to your Lambda function and you will receive the following information:

  • The common parameters of Amazon Cognito Lambda triggers (including userPoolId and clientId)
  • Which external IdP was used (for example, providerName)
  • The providerType (SAML, OIDC, Login with Amazon, and so on)
  • Attribute data from the external IdP specific to the user signing in

The specific format of this attribute data depends on the provider type, view the Inbound federation Lambda trigger parameters section in the docs to learn more. If the external IdP is a SAML provider, you will receive a JSON key-pair listing of the user’s attributes from the IdP assertion. If the external IdP is an OIDC provider (or social provider), you will receive the access token and attribute data from the /userinfo endpoint, along with an ID token if one was provided. See Figure 1 for a detailed flow of a federated sign-in with an Amazon Cognito user pool configured to use the inbound federation Lambda trigger.

Figure 1: Sequence flow of a federated login configured with the inbound federation Lambda trigger

Figure 1: Sequence flow of a federated login configured with the inbound federation Lambda trigger

  1. The user begins using the application but is required to sign in first.
  2. The managed login is rendered, and the user can select which IdP they want to sign in with. If identifiers are used with SAML or OIDC providers, the user enters their email address and Amazon Cognito looks up the domain of their provided email and routes them to the appropriate IdP.
  3. Alternatively, the managed login can be bypassed by the client providing the identity_provider request parameter.
  4. Amazon Cognito sends the authentication request to the appropriate IdP.
  5. The external IdP challenges the user to sign in.
  6. The user completes the sign-in process required by the external identity provider.
  7. The challenge response is sent to the external IdP.
  8. The IdP verifies that the sign-in is successful. If there are any subsequent challenges, such as multi-factor authentication (MFA), additional rounds of authentication challenges and responses take place. This is determined by the configuration and settings of the external IdP.
  9. The external IdP sends a response to the Amazon Cognito user pool, and Cognito validates the cryptographic signature and that it hasn’t been tampered with.
  10. Amazon Cognito sends attribute data from the IdP to the inbound federation Lambda function
  11. Attribute data for the authenticated user and the common parameters for Amazon Cognito are available for the Lambda function to add, modify, or suppress according to your requirements.
  12. Your added, modified, or suppressed attributes are returned to Amazon Cognito. These are attribute values that map to the user’s profile in Cognito—whether the user profile was just created or is being updated for a returning user.
  13. Continuing the OAuth 2.0 authorization code grant, Amazon Cognito sends an authorization code to the client.
  14. The client then calls the /token endpoint with the authorization code.
    Note: It’s a security best practice to use confidential clients and to use OAuth 2.0 Proof Key for Code Exchange (PKCE) extension whenever possible.
  15. An access, ID, and refresh token is returned to the client.
  16. The user has signed into the application. ID tokens can be used to identify who the user is (authentication), and access tokens can be used to determine what the user can do (authorization).

Common federation challenges and use cases

Federation introduces complexity that varies depending on your use case. For B2B and SaaS applications, you’re often not in control of your customers’ IdPs, including what attributes they send or how they format them. As an example, an enterprise customer will configure their SAML response to include every group a user belongs to. This could be hundreds of groups or long group identifiers, and if the group membership of the user is mapped to an Amazon Cognito attribute, this can lead to a scenario where the Cognito attribute size limit is exceeded, causing federated sign-ins to fail.

Challenges for business-to-customer (B2C) applications can differ from B2B use cases. For B2C applications, organizations shouldn’t be required to think about identity providers. The ability to sign-up and sign-in should be seamless for consumer-facing applications. Customers visiting a consumer-facing application might create an account with email and password, forget they created created it, and then later try signing in with Facebook (or other social provider). Without proper account linking in Amazon Cognito, you then have multiple user records for the same user, which could lead to fragmented purchase history and a frustrating customer experience.

Both B2B and B2C use cases might need to look up external data just prior to completing the sign-in process, such as additional roles and access for B2B users or looking up active orders for B2C users. Another example could be the need to normalize data just prior to storing it in the user profile within the Amazon Cognito user pool or even discarding personally identifiable information (PII) prior to storing it in your Cognito user pool.

With the inbound federation Lambda trigger, you can handle these B2B and B2C use cases programmatically, and do so without requiring modification of your applications or coordinating IdP-specific changes with external IdPs. In this section, I dive deeper into two common use cases: oversized group attributes, common with B2B customers, and automated account linking, common with B2C customers.

Use case 1: Filtering oversized group attributes

If you have B2B and SaaS use cases, it’s a common practice to use group membership from the IdP to determine the level of access you have within the SaaS service. This is a great way to still provide some access control back to the enterprise customers themselves. The groups can be used to represent the roles a user will have or for some form of coarse-grained authorization. However, your customers might inadvertently send a large number of groups a user is a member of, thus leading to an oversized attribute payload.

Another common scenario is where the syntax and format of group name a user belongs to can arrive in various formats across different IdPs; such as a canonical name (for example, example.com/groups/myApp-readOnly), a distinguished name (common with LDAP based systems and such as cn=myApp-readOnly,OU=groups,DC=example,DC=com), or a plain text string (such as myApp-readOnly). Instead of having downstream authorization logic to accommodate different variations of a group name, you can now normalize how groups are represented prior to storing the user’s attribute data using the inbound federation Lambda trigger.

To expand this, imagine your enterprise customer uses a SAML IdP, such as Active Directory Federation Services (AD FS), in front of Active Directory (AD). When their users authenticate, AD FS sends a groups attribute containing every AD group the user belongs to. For users in large organizations, this can be hundreds of groups, and the attribute is mapped to an Amazon Cognito attribute, this could result in a string that exceeds 2,048-character limit per attribute of Cognito. Authentication would fail in this scenario, ultimately leading to support tickets because enterprise customers would be unable to sign in. Even if certain users didn’t exceed this limit, because of a smaller number of group memberships, this would result in the collection and storing of unnecessary data in your Cognito user pool.

Previously, you would need to work with your customer’s IT department to modify their SAML configuration to filter groups at the source—a process that could take weeks and require multiple approval cycles because it involves a change to the federation configuration. Especially for SaaS customers, this isn’t a scalable approach because you could integrate with hundreds of external IdPs. With the inbound federation Lambda trigger, you can solve this by filtering the groups to only those relevant to your application and normalizing the nomenclature of these groups. The following Lambda function filters the groups attribute to include only groups relevant to your application and normalizes the names of groups.

// Configure the group prefix to filter on (e.g. "App1-", "myApp-", etc.)
// Change this to match the prefix your IdP uses for relevant group names.
const GROUP_PREFIX = process.env.GROUP_PREFIX || 'myApp-';

// The SAML attribute/claim name that contains group membership.
// Common values: "groups", "memberOf", "http://schemas.xmlsoap.org/claims/Group", etc.
const GROUP_ATTRIBUTE = process.env.GROUP_ATTRIBUTE || 'groups';

/**
 * Extracts the short group name from common IdP formats:
 *   - Plain text:       "myApp-readOnly"
 *   - Leading slash:    "/myApp-readOnly"
 *   - Canonical/URL:    "example.com/groups/myApp-readOnly"
 *   - Distinguished name (DN): "cn=myApp-readOnly,OU=groups,DC=example,DC=com"
 * Returns the last meaningful segment so all formats normalize to "myApp-readOnly".
 */

function extractGroupName(raw) {
  let name = raw.trim();

  // Some IdPs prefix group names with "/" to indicate a top level group — strip it before format detection
  if (name.startsWith('/')) {
    name = name.substring(1);
  }

  // DN format — extract the CN (common name) value
  if (/^cn=/i.test(name) || /,\s*(ou|dc)=/i.test(name)) {
    const cnMatch = name.match(/^cn=([^,]+)/i);
    return cnMatch ? cnMatch[1].trim() : name;
  }

  // URL / path format — take the last segment after the final "/"
  if (name.includes('/')) {
    const segments = name.split('/').filter(Boolean);
    return segments[segments.length - 1];
  }

  return name;
}
export const handler = async (event) => {
  try {
    console.log('Full event:', JSON.stringify(event, null, 2));
    console.log('Provider type:', event.request?.providerType);

    // Initialize the response structure
    event.response = event.response || {};

    if (event.request?.providerType?.toLowerCase() === "saml") {
      const samlResponse = event.request.attributes?.samlResponse;

      if (samlResponse) {
        console.log('Original SAML Attributes:', JSON.stringify(samlResponse, null, 2));

        // Build the attribute map — you MUST include every attribute you want Cognito to retain. Anything omitted from userAttributesToMap is dropped.
        const mappedAttributes = {};

        Object.keys(samlResponse).forEach(key => {
          if (key === GROUP_ATTRIBUTE) {
            // Parse the groups JSON string from the SAML assertion
            let groupsArray = [];
            try {
              groupsArray = JSON.parse(samlResponse[GROUP_ATTRIBUTE]);
            } catch (error) {
              console.error(`Error parsing ${GROUP_ATTRIBUTE}:`, error);
            }

            // Normalize each group name, then filter to the configured prefix
            const normalizedGroups = groupsArray.map(extractGroupName);
            const filteredGroups = normalizedGroups.filter(group =>
              group.startsWith(GROUP_PREFIX)
            );

            console.log(`Original ${GROUP_ATTRIBUTE}:`, groupsArray);
            console.log(`Normalized ${GROUP_ATTRIBUTE}:`, normalizedGroups);
            console.log(`Filtered ${GROUP_ATTRIBUTE}:`, filteredGroups);

            // Only include the groups attribute if there are matching groups
            if (filteredGroups.length > 0) {
              mappedAttributes[GROUP_ATTRIBUTE] = filteredGroups.map(group => `'${group}'`).join(', ');
            }
          } else {
            // Pass all other SAML attributes through unchanged
            mappedAttributes[key] = samlResponse[key];
          }
        });

        event.response.userAttributesToMap = mappedAttributes;
        console.log('Response to Cognito:', JSON.stringify(event.response, null, 2));
      }
    }

    // For any unhandled provider type (or missing samlResponse), this intentionally does NOT set userAttributesToMap and tells Cognito to keep all original IdP attributes unchanged (no-op).

    // To handle OIDC or social providers, add additional logic here using event.request.attributes.idToken, .userInfo, and/or .tokenResponse.

    return event;
  } catch (error) {
    console.error('Error in Lambda:', error);
    throw error;
  }
};

This approach reduces a large group list to only what is applicable to your application. Authentication succeeds, and you maintain control over your user pool’s data without depending on external configuration changes.

Use case 2: Automatic account linking

The second use case addresses a challenge that’s particularly common in B2C facing ecommerce or any consumer-facing applications; although it can also be applicable to B2B scenarios. Imagine you’re running an online retail store. A customer creates an account with their email and password to make a purchase. A few months later, they return to your site but forgot they already created an account and they see the Login with Amazon button and decide to sign in this way. Without account linking, Amazon Cognito creates a new federated user because these are technically distinct accounts, and now this customer has two separate accounts with different purchase histories and saved preferences.

This fragmentation creates a poor customer experience and complicates your business operations. You can’t see the customer’s complete purchase history, loyalty points are split across accounts, and your analytics show two distinct customers instead of one.

The inbound federation Lambda trigger can be used to solve this by automatically linking federated identities to existing local accounts based on email address. While account linking can also be implemented in a pre-sign-up Lambda trigger, the inbound federation trigger runs on every federated sign-in, not just the first, giving you access to the latest IdP attributes and the ability to apply linking logic continuously rather than only at initial account creation. If no local Amazon Cognito account exists, you can create one and then link the social provider account to it. The local account can serve as the primary identity, ensuring consistent JSON Web Tokens (JWTs) regardless of how the user signs in. The following is an example of an inbound federation Lambda trigger that can help address this use case.

import { 
  CognitoIdentityProviderClient, 
  ListUsersCommand,
  AdminCreateUserCommand,
  AdminLinkProviderForUserCommand
} from "@aws-sdk/client-cognito-identity-provider";

const client = new CognitoIdentityProviderClient();

export const handler = async (event) => {
  try {
    console.log('Full event:', JSON.stringify(event, null, 2));
    
    const { userPoolId, request, userName } = event;
    const { providerName, providerType, attributes } = request;
    
    // Extract email and profile attributes based on provider type
    const { email, givenName, surname } = extractAttributes(providerType, attributes);
    
    if (!email) {
      console.error('No email found in federated response');
      return event;
    }
    
    console.log(`Processing federated login for email: ${email}, provider: ${providerName} (${providerType})`);
    
    // Check if a local user exists with this email
    const existingUser = await findLocalUserByEmail(userPoolId, email);
    
    if (existingUser) {
      console.log(`Found existing local user: ${existingUser.Username}`);
      if (isAlreadyLinked(existingUser, providerName, userName)) {
        console.log(`Federated identity ${providerName}:${userName} is already linked to ${existingUser.Username}, skipping link`);
      } else {
        await linkFederatedUser(userPoolId, existingUser.Username, providerName, userName);
      }
    } else {
      console.log('No existing local user found, creating new one');
      const newUsername = await createLocalUser(userPoolId, email, givenName, surname);
      await linkFederatedUser(userPoolId, newUsername, providerName, userName);
    }
    
    return event;
    
  } catch (error) {
    console.error('Error in account linking Lambda:', error);
    throw error;
  }
};


/**
 * Check if the federated identity is already linked to the local user by inspecting the identities attribute from the ListUsers response.
 */
function isAlreadyLinked(user, providerName, federatedUsername) {
  const identities = user.Attributes?.find(a => a.Name === 'identities');
  if (!identities?.Value) return false;

  try {
    const parsed = JSON.parse(identities.Value);
    return parsed.some(id => id.providerName === providerName && id.userId === federatedUsername);
  } catch {
    return false;
  }
}

/**
 * Extract email and profile attributes based on provider type.
 * - SAML: attributes come from samlResponse
 * - OIDC/Social: attributes come from userInfo, falling back to idToken (if one exists)
 */
function extractAttributes(providerType, attributes) {
  if (providerType?.toLowerCase() === 'saml') {
    const saml = attributes?.samlResponse;
    return {
      email: saml?.email || null,
      givenName: saml?.givenName || '',
      surname: saml?.surname || ''
    };
  }

  // OIDC and social providers: prefer userInfo, fall back to idToken
  const userInfo = attributes?.userInfo;
  const idToken = attributes?.idToken;

  const source = userInfo?.email ? userInfo : idToken;

  return {
    email: source?.email || null,
    givenName: source?.given_name || '',
    surname: source?.family_name || ''
  };
}

/**
 * Find a local Cognito user (not EXTERNAL_PROVIDER) by email address.
 */
async function findLocalUserByEmail(userPoolId, email) {
  try {
    const command = new ListUsersCommand({
      UserPoolId: userPoolId,
      Filter: `email = "${email}"`
    });
    
    const response = await client.send(command);
    console.log('ListUsers response:', JSON.stringify(response, null, 2));
    
    if (!response.Users || response.Users.length === 0) {
      return null;
    }

    // Find the first user that is a true local account (not a federated-only profile)
    const localUser = response.Users.find(u => u.UserStatus !== 'EXTERNAL_PROVIDER');
    return localUser || null;
  } catch (error) {
    console.error('Error finding user by email:', error);
    throw error;
  }
}

/**
 * Create a new local Cognito user without a password.
 * With passwordless (email OTP) enabled on the user pool, the user is created with UserStatus=CONFIRMED and no FORCE_CHANGE_PASSWORD state.
 */
async function createLocalUser(userPoolId, email, givenName, surname) {
  try {
    const userAttributes = [
      { Name: 'email', Value: email }
    ];

    if (givenName) userAttributes.push({ Name: 'given_name', Value: givenName });
    if (surname) userAttributes.push({ Name: 'family_name', Value: surname });

    const command = new AdminCreateUserCommand({
      UserPoolId: userPoolId,
      Username: email,
      UserAttributes: userAttributes,
      MessageAction: 'SUPPRESS'
    });
    
    const response = await client.send(command);
    console.log(`Created local user: ${email}`, JSON.stringify(response, null, 2));
    
    return email;
  } catch (error) {
    console.error('Error creating local user:', error);
    throw error;
  }
}

/**
 * Link a federated user identity to a local Cognito user.
 * The local user becomes the primary profile — all future JWTs will represent this local user regardless of sign-in method.
 */
async function linkFederatedUser(userPoolId, localUsername, providerName, federatedUsername) {
  try {
    const command = new AdminLinkProviderForUserCommand({
      UserPoolId: userPoolId,
      DestinationUser: {
        ProviderName: 'Cognito',
        ProviderAttributeValue: localUsername
      },
      SourceUser: {
        ProviderName: providerName,
        ProviderAttributeName: 'Cognito_Subject',
        ProviderAttributeValue: federatedUsername
      }
    });
    
    const response = await client.send(command);
    console.log(`Linked federated user ${federatedUsername} to local user ${localUsername}`);
    console.log('Link response:', JSON.stringify(response, null, 2));
    
    return response;
  } catch (error) {
    if (error.name === 'AliasExistsException' || error.message?.includes('already linked')) {
      console.log(`User already linked: ${error.message}`);
      return;
    }
    console.error('Error linking federated user:', error);
    throw error;
  }
}

Every federated sign-in will invoke the inbound federation Lambda trigger, and the logic is straightforward. When a user authenticates with an external identity provider, the trigger extracts their email from the federated response and searches the user pool for a local Cognito account with that same email. If one exists—such as if the user originally signed up with email and password—the Lambda function links the federated identity to that existing local account. If no local account exists, the trigger creates one on the fly as a passwordless account (confirmed, suppressing any emails, and ready for passwordless email one-time passcode (OTP) sign-in), then links the federated identity to it. In both cases, the local account is set as the primary profile. This means the user’s JWTs always carry the same sub-claim regardless of how they sign in—directly, or through Google, Facebook, or SAML—your application sees one consistent identity. The preceding Lambda trigger is also smart enough to check whether a linked account already exists before making the call, so returning users who’ve already been linked don’t generate unnecessary API calls. And because the local account supports passwordless authentication, a user who first arrived through federation can later sign in directly with an emailed OTP—or even add a password later through your applications account settings. The local account is always the anchor.

Best practices

As you implement these patterns, keep a few best practices in mind. Your Lambda function must be completed within 5 seconds, so optimize for speed to help ensure the federated sign-in process is able to successfully complete. If you’re making external calls within the inbound federation Lambda function, like Amazon DynamoDB queries or API requests, implement caching where possible. Handle errors gracefully—if your Lambda function throws an exception or an error, authentication could fail for the user. Consider logging the error and returning the original event back to Amazon Cognito rather than failing authentication for a legitimate user attempting to sign in. Here are some additional best practices for working with Lambda functions.

For the account linking use case, automatic linking relies on matching the email from the federated identity to a local account. However, there are scenarios where this match won’t exist. For example, Apple’s Hide My Email feature generates a unique alias for each app, so the federated email won’t match any existing local account. This is an effective privacy feature but it also blocks the ability to automatically link accounts. In cases like these, your application will need to implement a user-initiated account linking flow, such as prompting the user to verify ownership of both email addresses before calling the AdminLinkProviderForUser API to complete the link.

Monitor your Lambda function performance using Amazon CloudWatch metrics. Set up alarms for errors, timeouts, and throttling so you can respond quickly if issues arise. I also recommend capturing sample event payloads from a CloudWatch log group during your initial development and deployment—these will be valuable for local testing and debugging which can lead to quicker resolution if issues arise in your production environment. This is especially important as different IdPs (namely SAML and OIDC providers) may respond with varying attribute and value syntaxes. Consider implementing CloudWatch alarms to alert your security and operational teams if authentication failures spike, which could indicate an attempted attack, misconfiguration, or provide insight into further optimization of your inbound federation Lambda trigger.

Conclusion

In this post, you learned about the new inbound federation Lambda trigger for Amazon Cognito and how it can solve various use cases. You walked through two common federation challenges and reviewed some sample code to help resolve those challenges. For B2B and SaaS applications, the inbound federation Lambda trigger gives you control when dealing with oversized attributes from external identity providers (such as group membership) without requiring coordination with enterprise IT teams. For B2C and consumer-facing applications, it enables seamless account linking across multiple authentication methods, creating a unified customer experience.

The new Lambda trigger works with SAML, OIDC, and supported social providers, and is available now in AWS Regions where Amazon Cognito is available. To learn more about the new Lambda trigger and others, see the Amazon Cognito Developer Guide.

What federation challenges are you facing in your applications? I’d love to hear about your use cases in the comments below and over at AWS re:Post.

Abrom-Douglas-author

Abrom Douglas

Abrom is a Senior Solutions Architect within AWS Identity with over 20 years of software engineering and security experience, specializing in the identity and access management space. He loves speaking with customers about how identity and access management can provide secure outcomes that enable both business and technology initiatives. In his free time, he enjoys cheering for Arsenal FC, photography, travel, volunteering, and competing in duathlons.

Accelerate SQL development with SageMaker Data Agent in Query Editor

Post Syndicated from Jason Ramos original https://aws.amazon.com/blogs/big-data/accelerate-sql-development-with-sagemaker-data-agent-in-query-editor/

When you develop SQL against Amazon Redshift and Amazon Athena, you spend time finding the right tables across hundreds of databases, writing complex joins and aggregations, debugging failed queries without context from previous attempts, and re-specifying filters for every new question. Amazon SageMaker Data Agent in Query Editor takes a different approach. You describe what you need in natural language, and the Data Agent generates the SQL. It references your actual tables through AWS Glue Data Catalog, proposes step-by-step plans for complex questions, retains context across your session, and offers one-click error recovery with Fix with AI. In this post, you learn how to use Data Agent in Query Editor to explore data, build multi-step analyses, recover from errors, and summarize results using a public education dataset.

Solution overview

You can go from a natural language question to executable SQL in seconds. Data Agent in Query Editor provides a conversational interface with direct access to your AWS data environment, so you spend less time on query mechanics and more time on analysis. Data Agent in Query Editor focuses specifically on SQL development against Amazon Redshift and Amazon Athena. (For Python, SQL, and PySpark across broader analytical and machine learning (ML) workloads, use Data Agent in notebooks.)

Data Agent provides four key capabilities:

  • Catalog-aware SQL generation. You don’t need to browse catalog structures or memorize schema details. Data Agent reads your table metadata directly.
  • Querybook and session context. You build on previous work. Data Agent uses context from your earlier queries and results.
  • Step-by-step planning. You review and approve a structured plan before Data Agent generates SQL.
  • Fix with AI. You recover from failed queries with one click.

Data Agent integrates with AWS Glue Data Catalog and reads your actual table names, column types, descriptions, and relationships, so generated SQL references your real tables. Each follow-up question builds on your current Query Editor session—the SQL cells in your querybook, the active connection, your selected cell, and execution results from previously run cells. For complex requests, Data Agent produces a structured plan that specifies which data to retrieve, how to aggregate it, and what filters to apply. You review and approve each step before Data Agent proceeds. When a query fails, choose Fix with AI to get a corrected query based on the error and the failed cell’s context.

Query Editor Fix with AI panel showing a corrected SQL query ready for review

[Figure 1: The Query Editor Fix with AI panel, showing a corrected SQL query ready for your review.]

Walkthrough: Education data analysis

In this section, you use Data Agent in Query Editor to analyze California schools data and identify where SAT improvement investment has the most impact. The walkthrough covers four tasks:

  • Explore available data.
  • Build a multi-step analysis plan.
  • Summarize insights from your queries.
  • Recover from a failed query.

The same workflow applies to your own data, whether you are analyzing sales figures, operational metrics, or financial records.

The California schools dataset contains SAT score results, school demographic information, and county-level data for public schools across California. The dataset includes tables that organize SAT scores by subject (reading, writing, math), school details (name, address, county, district), and enrollment figures. After you upload the data into your project database, you directly access the tables from Query Editor through your Amazon Athena or Amazon Redshift Lakehouse connection.

Prerequisites

To complete this walkthrough, you need intermediate SQL knowledge and basic familiarity with the AWS Management Console. You don’t need prior AWS Glue experience, but familiarity with data catalogs (centralized metadata repositories) helps.

You can choose one of two setup paths:

  • Quick start (5 minutes). SageMaker Unified Studio provides a sample database (sagemaker_sample_db) with pre-loaded data. To explore it, choose Data in the navigation pane, expand AwsDataCatalog, and select sagemaker_sample_db.
  • Full setup (30–45 minutes). Upload the California schools dataset into your project’s Lakehouse database. This dataset is publicly available from the California Department of Education. Download the SAT scores, school information, and county-level data files, then upload them through the SageMaker Unified Studio UI. In your project, go to Build, choose Query editor, right-click your project database in the Data explorer, and choose Create table. Drag and drop each CSV file to create the tables. SageMaker Unified Studio stores the data in the project-managed Amazon Simple Storage Service (Amazon S3) location, registers it in AWS Glue Data Catalog, and applies AWS Lake Formation governance automatically.

Running queries against Amazon Athena or Amazon Redshift might incur costs. For pricing details, refer to Amazon Athena pricing and Amazon Redshift pricing. For detailed setup instructions, refer to AWS Identity and Access Management (IAM)-based domains and projects. Before starting the walkthrough, you must have a SageMaker Unified Studio IAM-based domain with a project using the SQL analytics or All Capabilities project profile. The project automatically provisions an AWS Glue database, the required IAM role, and Athena or Redshift Lakehouse connections.

Data Explorer panel in Query Editor showing the california_schools_db and sagemaker_sample_db tables

[Figure 2: The Data Explorer panel in Query Editor, showing the california_schools_db and sagemaker_sample_db tables.]

Explore available data. To start, enter the following prompt in the Data Agent panel:

Query my SAT scores from my california_schools_db

Data Agent searches AWS Glue Data Catalog, locates the relevant tables, and generates an initial exploratory query that retrieves SAT score records. It adds a SQL cell directly to your querybook.

  • Review the generated SQL in the comparison view, which highlights the proposed code.
  • Choose Accept, Reject, or Accept and run.
  • After you run the cell, the results appear inline, giving you a view of the data (column names, score ranges, and the number of records) before you write SQL.

Data Agent returns an exploratory query for the california_schools_db tables, ready for review

[Figure 3: Data Agent returns an exploratory query for the california_schools_db tables, ready for your review.]

SQL query results appear beneath the cell after choosing Accept and run

[Figure 4: The SQL query results appear beneath the cell after you choose Accept and run.]

Build a multi-step analysis plan. With the data explored, enter a more complex analytical question:

Identify which subjects need investment to improve SAT scores in the lowest-performing counties. Include school-level details with addresses.

Data Agent proposes a step-by-step plan before generating SQL. For this request, Data Agent breaks the question into three steps:

  1. Aggregate SAT scores by county and subject to find performance patterns.
  2. Filter to counties with a sufficient number of schools and rank the lowest performers.
  3. Join school address data to produce a final detailed list.

Review the plan in the Data Agent panel and choose Run step-by-step to proceed.

Data Agent proposes a multi-step plan with Cancel plan and Run step-by-step options

[Figure 5: Data Agent proposes a multi-step plan with options to Cancel plan or Run step-by-step.]

Data Agent generates SQL for each step and adds it as a separate querybook cell. Review each cell’s SQL in the comparison view, then choose Accept and run to execute it. The results from each step are visible inline, so you can verify the intermediate output (county-level aggregations, the filtered ranking, and the final school list) before moving to the next step. When the steps are complete, your querybook contains the full analytical progression from raw scores to a detailed investment list.

Each plan step produces a separate querybook cell that can be reviewed and run independently

[Figure 6: Each plan step produces a querybook cell that you can review and run independently.]

Summarize insights from your queries. After running the analysis, enter the following prompt:

Summarize the insights from my queries

Data Agent has context on your querybook, including the SQL and the query results from each cell. It generates a natural language summary: which counties are underperforming, which subjects (reading, writing, or math) need the most attention in each county, and how many schools appear on the investment list. This summary provides a starting point for a report or presentation.

Data Agent summarizes insights from the accumulated query results in the querybook

[Figure 7: Data Agent summarizes insights from the accumulated query results in the querybook.]

Recover from a failed query. During the analysis, a generated query might produce an error, for example, referencing a column name that doesn’t match the schema or a join condition that returns unexpected results. When a cell fails, Query Editor displays the error message and a Fix with AI option.

Choose Fix with AI, and Data Agent reads the error in the context of the failed cell, then generates corrected SQL and updates the querybook cell. Run the corrected cell to verify the fix.

After choosing Fix with AI, Data Agent generates a corrected query for the failed cell

[Figure 8: After you choose Fix with AI, Data Agent is prompted to generate a corrected query for the failed cell.]

Data Agent returns corrected SQL for review

[Figure 9: Data Agent returns corrected SQL for you to review.]

Security and governance

Data Agent operates within your AWS environment and only accesses data that your IAM policies explicitly permit. Your existing IAM access controls and AWS Lake Formation permissions determine what data Data Agent can reach. To use Data Agent, your project role must have permissions to invoke specific Amazon DataZone APIs. For more information, refer to Actions, resources, and condition keys for Amazon DataZone.

Data Agent includes content filtering that prevents it from responding to off-topic requests, requests to reveal its system prompt, and requests for internal technical implementation details. Data Agent is restricted to AWS-related topics and English-language output.

Amazon SageMaker stores your natural language prompts and generated SQL in the AWS Region where you created your SageMaker Unified Studio domain. Data Agent doesn’t store your data, querybook context, or catalog metadata.

To opt out of data usage for service improvement, configure an AI services opt-out policy for Amazon DataZone in AWS Organizations. For more information, refer to Data storage in the SageMaker Data Agent, Service improvement, and AI services opt-out policies.

Clean up

The walkthrough creates querybook cells in your Query Editor session but doesn’t provision standalone infrastructure. To remove the generated SQL cells, delete them from your querybook or delete the querybook itself.

If you uploaded the California schools dataset specifically for this walkthrough, remove the following resources to avoid ongoing charges:

  • SageMaker Unified Studio domain. If you created a domain solely for this walkthrough, delete it to stop incurring charges. Refer to the SageMaker Unified Studio administration guide for deletion steps.
  • Uploaded tables. In the Data explorer, right-click each table you created and choose Delete table to remove the data from your project database and the underlying S3 storage.
  • Amazon Athena query results. Amazon Athena stores query results in an S3 output location. Delete the query result files from that bucket, or delete the bucket if you created it solely for this walkthrough.
  • Amazon CloudWatch logs. If Amazon Athena queries generated CloudWatch log groups, delete those log groups to avoid storage charges.

Conclusion

Data Agent in Query Editor brings conversational, catalog-aware SQL development to your Amazon Redshift and Amazon Athena workloads. In this post, you explored unfamiliar data, built a multi-step investment analysis, recovered from query errors, and summarized findings through natural language prompts.

Data Agent works within your existing IAM and AWS Lake Formation security controls, keeps your data within your AWS environment, and retains context across your analytical workflow so each question builds on the last.

Get started with these next steps:

  1. Run your first prompt. Open Query Editor in your SageMaker Unified Studio domain and enter Show me the top 10 tables in my catalog with the most columns. For setup, refer to the SageMaker Unified Studio getting started guide.
  2. Add descriptions to your AWS Glue Data Catalog. Table descriptions and column-level business metadata improve the quality of generated SQL. For best practices, refer to Populating the AWS Glue Data Catalog.
  3. Try a multi-step analysis. Enter Which product categories had declining revenue quarter-over-quarter, and which regions drove the decline? and review Data Agent’s plan step by step.

For more information, refer to the Amazon SageMaker Data Agent documentation, the What’s New blog post, Amazon Redshift documentation, and Amazon Athena documentation. To learn how Data Agent works in notebooks, refer to Accelerate context-aware data analysis and ML workflows with Amazon SageMaker Data Agent.


About the authors

Jason Ramos

Jason Ramos

Jason is a Front-End Engineer on the Amazon SageMaker Unified Studio team. He builds the scalable frontend experiences that power SageMaker Data Agent, bringing conversational AI capabilities to data scientists, analysts, and engineers across SageMaker Unified Studio. Outside of work, he enjoys playing piano and exploring the Bay Area food scene.

Olena Mursalova

Olena Mursalova

Olena is a Software Development Engineer on the Amazon SageMaker Unified Studio team, where she develops the SageMaker Data Agent — an intelligent assistant that turns natural language prompts into code, visualizations, and data insights for data engineers and analysts.

Jessica Cheng

Jessica Cheng

Jessica is a Front-End Engineer on the Amazon SageMaker Unified Studio team based in the Bay Area, where she builds intelligent data agent experiences. At work, she is passionate about creating accessible, easy-to-use experiences at scale. Outside of work, her passions lie in finding the best swimming hole in California.

Sanjana Sekar

Sanjana Sekar

Sanjana is a Software Development Engineer on the Amazon SageMaker Unified Studio team. She was one of the engineers who built the SageMaker Data Agent, bringing conversational AI-powered SQL generation and debugging to Query Editor. She is focused on improving data agent capabilities and the compute blueprints experience within SageMaker Unified Studio. Outside of work, she enjoys hiking and biking.

Siddharth Gupta

Siddharth Gupta

Siddharth is heading Generative AI within SageMaker’s Unified Experiences. His focus is on driving agentic experiences, where AI systems act autonomously on behalf of users to accomplish complex tasks. An alumnus of the University of Illinois at Urbana-Champaign, he brings extensive experience from his roles at Yahoo, Glassdoor, and Twitch.

Schedule notebook runs in Amazon SageMaker Unified Studio

Post Syndicated from Shivani Mehendarge original https://aws.amazon.com/blogs/big-data/schedule-notebook-runs-in-amazon-sagemaker-unified-studio/

If you build notebooks for recurring tasks such as daily customer analysis, weekly report generation, or data quality checks in Amazon SageMaker Unified Studio, you’ve likely wanted to run them automatically on a schedule. Until now, there wasn’t a native way to do this. Teams had to manage orchestration separately, even though the interactive notebook experience was already in place. Now, notebook scheduling is available, so you can configure your production workloads to run automatically with minimal manual intervention.

In this post, we walk you through the new scheduling and orchestrating capabilities for notebooks in Amazon SageMaker Unified Studio. You will learn how to:

  • Trigger on-demand background runs, such as a model re-training job, without waiting at your desk.
  • Create recurring schedules for tasks such as nightly data freshness checks or weekly business reviews.
  • Parameterize notebooks so a single template can generate reports across different AWS Regions or customer segments.
  • Orchestrate multi-notebook workflows where one notebook’s output feeds into the next. For example, an extract, transform, and load (ETL) pipeline followed by a summary dashboard refresh.
  • Debug failed runs with AI-assisted troubleshooting.

Sample use case overview

In this walkthrough, you will take on the role of a logistics analyst who monitors shipping performance across carriers. The notebook loads shipping data from the ShippingLogs.csv dataset, identifies late deliveries, and generates a performance summary. You want to run this notebook every morning without manual intervention, reuse it across different carriers, and know when something goes wrong.

You will start by running a notebook in the background and viewing the results. Next, you will create a recurring schedule for daily runs, then parameterize the notebook to generate reports for different carriers. You will also orchestrate the notebook in a multi-step workflow and debug a failed run using AI-assisted troubleshooting.

Prerequisites

Before you begin, you need:

  • An Amazon SageMaker Unified Studio project with Notebooks enabled. See Set up IAM-based domains for permission requirements.
  • A sample dataset. We use the ShippingLogs.csv dataset, which contains shipping data including estimated and actual delivery times, carriers, and origins. You can download it from the Workshop Studio (the file is named ShippingLogs.csv on the linked page).

Setting up the notebook

Start by creating a new notebook in your SageMaker Unified Studio project. If you haven’t already, upload the ShippingLogs.csv file under the Shared tab in the Files panel.

SageMaker Unified Studio Notebook Files panel showing the Shared tab with the ShippingLogs.csv dataset uploaded

In the first cell, we load and explore the dataset. To reference the file in code, select the file in the Shared tab and copy the Amazon Simple Storage Service (Amazon S3) URI shown in the file details. Alternatively, you can reference it with this code:

import pandas as pd
from sagemaker_studio import Project

# Initialize the project
proj = Project()

# Get the S3 root path
s3_root = proj.s3.root

df = pd.read_csv(s3_root + '/ShippingLogs.csv')
df.head()

The dataset contains columns including Carrier, ActualShippingDays, ExpectedShippingDays, ShippingOrigin, ShippingPriority, and OnTimeDelivery. Add a second cell to analyze shipping performance for a single carrier:

import matplotlib.pyplot as plt

carrier_data = df[df['Carrier'] == 'GlobalFreight']
# Flag late deliveries
carrier_data['is_late'] = carrier_data['ActualShippingDays'] > carrier_data['ExpectedShippingDays']
late_pct = carrier_data['is_late'].mean() * 100
# Visualize actual vs expected shipping days
plt.figure(figsize=(12, 4))
plt.hist(carrier_data['ActualShippingDays'] - carrier_data['ExpectedShippingDays'], bins=20, edgecolor='black')
plt.axvline(x=0, color='red', linestyle='--', label='On time')
plt.title(f'Shipping Delay Distribution - GlobalFreight ({late_pct:.1f}% late)')
plt.xlabel('Days Over Expected')
plt.ylabel('Number of Shipments')
plt.legend()
plt.show()

With the notebook working interactively, you’re ready to automate it.

Running a notebook asynchronously

To trigger an asynchronous run, open your notebook. In the notebook header, choose the menu on the Run all button, and then choose Run in background.

Notebook header with the Run all menu expanded, showing the Run in background option

This captures a snapshot of the notebook in its current state and starts a run on a separate dedicated compute. You can continue working on other tasks or close the browser entirely. Your interactive session isn’t affected.

You will see a notification at the bottom of your screen confirming that the run started. To check the status of your run, choose View Run in the notification. This opens a view showing every background and scheduled run with its status, duration, and a link to view the full output.

Run history view showing background and scheduled runs with status, duration, and output links

You can choose to view the run details at any point to view results as cells run. The run details include three tabs:

  • Output: The notebook in read-only mode with cell results rendered, including dataframe outputs, visualizations, and print statements.
  • Parameters: The parameter values used for this run.
  • Logs: Run logs for debugging.

Run details view showing the Output, Parameters, and Logs tabs with rendered cell output

You can also access past runs by selecting the View Runs option in the notebook header.

Notebook header with the View Runs option highlighted

Stopping an in-progress run

If you need to cancel a run, open the run, and choose Stop. The run terminates, and its status updates to reflect the cancellation.

Run detail view with the Stop button selected to terminate an in-progress run

What to know about background runs

Compute: Each background run uses its own dedicated compute, separate from your interactive session. Your interactive work isn’t interrupted.

Packages: The packages that you install through the notebook’s package manager will be available in your background runs. When you use !pip install in code cells, the asynchronous run installs those packages as well.

Local files: Background runs can’t access files stored locally in your notebook environment. Reference data from your project’s shared storage (Amazon S3) or connected data sources instead.

Startup time: Expect a few minutes of startup time while compute is provisioned and your environment is prepared.

Creating a recurring schedule

Now that you’ve confirmed asynchronous runs work correctly, you can automate the notebook on a schedule. Choose the schedule icon in the notebook header to open the schedule creation form.

Schedule creation form opened from the notebook header schedule icon

Configure the following settings:

  • Schedule name: Enter a descriptive name, such as Daily Shipping Report.
  • Schedule type: Choose Recurring for repeated runs or One-time for a single future run.
  • Frequency: Define how often the notebook runs using a rate (for example, every one day) or a cron expression. Set the time zone and the start and end dates for the schedule. For example, set the schedule to run every day at 7:00 AM UTC starting tomorrow.
  • Flexible time window (optional): The number of minutes after the scheduled start time within which the run can be invoked. For example, with a 5-minute window, the notebook runs within 5 minutes of the start time.
  • Advanced settings:
    • Compute Instance: Keep the current settings or override with a different instance type for the asynchronous run to use.
    • Timeout: Set a maximum run duration to help prevent notebooks from running indefinitely. If left blank, it defaults to 60 minutes.

Choose Create.

Configured schedule form with name, recurring type, daily frequency, and advanced settings populated

The schedule appears in the Schedules tab of the activity panel. SageMaker Unified Studio creates an Amazon EventBridge Scheduler schedule for each schedule you configure.

Schedules tab in the activity panel listing the newly created Daily Shipping Report schedule

Viewing schedule run history

To view past runs for a schedule, choose the schedule name in the Schedules activity panel. This opens the schedule details view, where you can see the list of runs triggered by that schedule, the duration of each run, and a link to open the notebook output for an individual run.

Schedule details view showing the list of past runs with status, duration, and output links

Editing and deleting schedules

To modify a schedule, choose Edit next to it in the Schedules panel. You can change the frequency, instance type, timeout, and other configuration fields. To pause or resume a schedule, choose Pause or Resume from the same menu. To remove a schedule, choose Delete from that menu. Deleting a schedule stops future runs but preserves historical run outputs in Amazon S3 for auditing purposes.

Schedules panel with the Edit, Pause, Resume, and Delete options for a schedule

Parameterizing notebooks

With parameters, you can reuse a single notebook across different inputs without duplicating code. For example, you can run the same shipping performance report for each carrier by passing a different carrier name to each run.

Defining parameters

Open the Parameters activity panel and choose Add. Set the parameter name to carrier and the default value to GlobalFreight.

Parameters activity panel with the carrier parameter and GlobalFreight default value configured

Using parameters in code

In your notebook, replace the second cell with the following code. This retrieves the carrier parameter value using the SageMaker Unified Studio Python SDK instead of the hardcoded value:

import sagemaker_studio
import matplotlib.pyplot as plt

carrier = sagemaker_studio.nbutils.parameters.get("carrier")

carrier_data = df[df['Carrier'] == carrier].copy()
carrier_data['is_late'] = carrier_data['ActualShippingDays'] > carrier_data['ExpectedShippingDays']
late_pct = carrier_data['is_late'].mean() * 100

plt.figure(figsize=(12, 4))
plt.hist(carrier_data['ActualShippingDays'] - carrier_data['ExpectedShippingDays'], bins=20, edgecolor='black')
plt.axvline(x=0, color='red', linestyle='--', label='On time')
plt.title(f'Shipping Delay Distribution - {carrier} ({late_pct:.1f}% late)')
plt.xlabel('Days Over Expected')
plt.ylabel('Number of Shipments')
plt.legend()
plt.show()

Creating schedules with different parameter values

Now create three schedules for the same notebook, each targeting a different carrier:

  • “daily-shipping-gf” with carrier = GlobalFreight.
  • “daily-shipping-mc” with carrier = MicroCarrier.
  • “daily-shipping-shipper” with carrier = Shipper.

When you view a historical run, a separate Parameters tab in the run output displays the parameter values that were active for that run.

You can also override parameter values when triggering an on-demand background run. Choose the menu on the Run all button, then choose Run with settings. You can keep the defaults or provide custom values for that run.

Orchestrating with Workflows

To combine notebooks into a multi-step pipeline, such as running a data calculation notebook before the shipping log notebook, you can use the Notebook Operator in the Workflows tool to orchestrate them.

To do this, choose the Add to workflows button under the options menu of the notebook header.

Notebook header options menu with the Add to workflows button highlighted

This takes you to the Workflows tool, adding a new Notebook Operator task with prefilled properties from your notebook. When configuring the Operator task:

  • Select the target notebook from the notebook menu.
  • Use the Parameters widget to pass notebook parameters into the run of the notebook.
  • Specify optional arguments such as the compute instance and timeout configuration for the run.

Workflows canvas with a Notebook Operator task configured with notebook, parameters, and compute settings

Workflows also supports polling for the status of a notebook run for a particular notebook using Notebook Sensor. In Workflows, you can add a new Sensor task by hovering on the edge of the existing Operator task, where a plus (+) button is displayed.

Workflows canvas showing the plus button on the edge of an Operator task for adding a Sensor

You can then search for and add the Notebook Sensor to the canvas.

Task picker dialog with Notebook Sensor selected for adding to the workflow canvas

When configuring the Sensor task, specify the notebook run ID within the text field. The Operator’s form field contains Jinja templating to retrieve the notebook run. If the Sensor is used within the same workflow as the Operator, this template can be copied to use within a Sensor to poll the notebook run. Select the target notebook from the notebook menu.

Notebook Sensor configuration panel with the notebook run ID field populated using Jinja templating

Within Workflows, you can configure notebook runs to emit outputs and use those outputs as inputs for subsequent notebook runs.

Building off of the previous shipping log notebook example, we will pass the carrier parameter from an upstream notebook’s output. Your shipping-logs-analysis notebook should be already set up.

Because the notebook depends on the carrier parameter, you can specify it in the Parameters panel.

Parameters panel for the shipping-logs-analysis Operator with the carrier parameter dependency configured

Now, define a second notebook, calculate-best-carrier, which performs a calculation to determine our best carrier to use for shipping:

import pandas as pd
from sagemaker_studio import Project

# Initialize the project
proj = Project()

# Get the S3 root path
s3_root = proj.s3.root

df = pd.read_csv(s3_root + '/ShippingLogs.csv')
df.head()

carrier_stats = df.groupby('Carrier').agg(
    total=('OrderID', 'count'),
    late=('OnTimeDelivery', lambda x: (x == 'Late').sum())
).reset_index()
carrier_stats['late_pct'] = carrier_stats['late'] / carrier_stats['total'] * 100

best = carrier_stats.sort_values('late_pct', ascending=True).iloc[0]
best_carrier = best['Carrier']

print("Late % by carrier:")
print(carrier_stats.to_string(index=False))
print(f"\nBest carrier: {best_carrier} ({best['late_pct']:.1f}% late)")

To configure the calculate-best-carrier notebook’s outputs, you can choose the Variables panel. A new selector is available at the bottom of this panel which allows you to select variables to mark as outputs.

Variables panel with the selector at the bottom for marking notebook variables as outputs

We want this notebook to emit the best_carrier variable.

Variables panel showing best_carrier marked as an output variable for the calculate-best-carrier notebook

Now, use the Add to workflows button as previously demonstrated to quickly add this notebook within a workflow. Chain a second Notebook Operator that points to our shipping-logs-analysis notebook. Because we specified a parameter dependency on carrier for this notebook, it’s available as an option in the Parameters widget menu.

Parameters widget menu of a Notebook Operator showing carrier as a configurable parameter dependency

When they’re chained, the notebook tasks detect the outputs set in upstream notebook runs. These outputs can be selected as keys within the Parameters widget of the Operator to pass into the run. This can be done recursively for an arbitrary number of Operator tasks. We can select the emitted best_carrier output from the calculate-best-carrier notebook.

Parameters widget displaying best_carrier as a selectable upstream output to pass into the next Operator

You can now choose the Save button on the top left of the visual canvas and the Run button to start the workflow. When the workflow is completed, the specified notebook outputs are available in the Task Output panel and the notebook run result can be viewed in the Notebooks tool.

Task Output panel showing the emitted notebook outputs after a successful workflow run

Notebook run result rendered in the Notebooks tool after the chained workflow completes

In a similar manner, the Notebook Sensor will also emit the notebook outputs from a particular notebook’s run which can be used within other tasks. This is useful when you want to retrieve outputs from a notebook run in another workflow.

Debugging a failed run with AI assistance

When viewing your past runs, you notice that a run from earlier today has a Failed status. Choose the failed run to open the notebook output in read-only mode.

In this example, suppose you incorrectly referred to column name ActualShippingDays as DeliveryDays. The run would fail with a KeyError: 'DeliveryDays' in the cell that computes late deliveries.

At the top of the failed run output, choose Troubleshoot with AI. Choosing the Troubleshoot with AI button lands you in the notebook with the Agent chat panel open.

Failed run output with the Troubleshoot with AI button highlighted at the top of the page

The data agent analyzes the cell outputs, identifies the cell that errored, explains the root cause, and suggests a fix. In this case, it identifies that the column DeliveryDays doesn’t exist in the dataframe and suggests updating the code reference. You can review the change, then verify the fix by choosing Run in background from the Run all menu to trigger a test run before the next scheduled run.

Note: You can also use the Data Agent to create schedules and start notebook runs using natural language, without having to navigate.

Cleaning up

To avoid incurring future charges, delete the resources that you created in this walkthrough:

  • Delete any schedules that you created from the Schedules panel in your notebook.
  • Delete test notebooks if you don’t need them.
  • Navigate to the Workflows page and delete any workflows that you created during this walkthrough.
  • Your project’s Amazon S3 storage retains historical run outputs until you manually remove them.

Conclusion

In this post, we showed how to run notebooks in the background in Amazon SageMaker Unified Studio using background runs, schedules, parameterization, workflow orchestration, and AI-assisted debugging. Using a shipping logistics dataset, we demonstrated how a single notebook can be parameterized to generate performance reports for different carriers on independent schedules, all without duplicating code or managing extensive infrastructure.

To get started, open a notebook in your SageMaker Unified Studio project, choose the menu on the Run all button in the notebook header, and choose Run in background. For more advanced use cases, explore workflows in Amazon SageMaker Unified Studio to build multi-step data pipelines, or review the Amazon SageMaker Unified Studio User Guide for additional configuration options.

Learn more:

If you have feedback or questions, reach out on AWS re:Post for Amazon SageMaker Unified Studio.


About the authors

Shivani Mehendarge

Shivani Mehendarge

Shivani is a Software Development Engineer at Amazon Web Services, where she builds scalable infrastructure that helps data teams run and automate their workloads in Amazon SageMaker Unified Studio. She is passionate about solving complex distributed systems challenges and building reliable cloud services.

Regan Perk

Regan Perk

Regan is a Senior Software Development Engineer on the Amazon SageMaker Unified Studio team. She designs, implements, and maintains features that enable customers to manage schedules and workflows in SageMaker Unified Studio.

Qazi Ashikin

Qazi Ashikin

Qazi is a Software Development Engineer at Amazon Web Services, where he works on developing features that allow customers to orchestrate workflows and schedules in SageMaker Unified Studio. He also works on AWS Glue Studio, where he builds agentic systems and maintains services that enable data analytics.

Building highly available Oracle databases with Amazon FSx for NetApp ONTAP

Post Syndicated from Vignyanand Penumatcha original https://aws.amazon.com/blogs/architecture/building-highly-available-oracle-databases-with-amazon-fsx-for-netapp-ontap/

Oracle databases power mission-critical enterprise applications, making their continuous availability essential for business operations. Traditional Oracle high availability (HA) solutions require complex clustering software, expensive shared storage arrays, and specialized database administration teams. These conventional approaches often introduce single points of failure while demanding significant operational overhead.

Modern cloud architectures offer a transformative approach that combines Amazon FSx for NetApp ONTAP (FSxN) with Amazon EC2 Auto Scaling groups, automated AMI creation, AWS Lambda-driven orchestration, and AWS Systems Manager Parameter Store (SSM Parameter). This solution removes traditional Oracle HA complexities while delivering enterprise-grade availability, automated recovery, and makes sure new instances launch with the latest Oracle configuration.

This post shows how to build a highly available Oracle database architecture using FSxN shared storage, Auto Scaling groups with dynamic AMI updates, and serverless orchestration to help reduce recovery times with current configurations.

Solution overview

The solution uses multiple AWS services working together to create a comprehensive high availability architecture. FSxN Multi-AZ provides persistent shared storage spanning availability zones for Oracle database files, software, and configurations, so that data remains accessible when EC2 instances are replaced. Auto Scaling groups deliver automated instance lifecycle management with the latest AMI configurations, so failed instances are quickly replaced with identical configurations that can immediately access the existing Oracle database files on FSxN. AWS Backup creates AMIs that capture the latest Oracle host configurations including patches and settings, preserving the complete server state for consistent deployments. AWS Lambda extracts the AMI ID from backup recovery points and updates the SSM Parameter, orchestrating the entire configuration management workflow. Systems Manager Parameter Store stores the current AMI ID for Auto Scaling group launch templates, so new instances always launch with the most recent configuration and can immediately connect to the Oracle database on shared storage.

The following diagram shows the complete architecture with all AWS services and their interactions:

AWS architecture diagram showing Oracle Database disaster recovery across two Availability Zones using FSx for ONTAP synchronous replication, AWS Backup automation with EventBridge and Lambda, and Auto Scaling group with SSM Parameter Store for AMI management.

Key benefits include:

  • Recovery Time Objective (RTO): Can help achieve 2–5 minutes with latest Oracle configuration
  • Recovery Point Objective (RPO): Near-zero through synchronous Multi-AZ replication
  • Configuration consistency: New instances launch with identical Oracle host setup
  • Automated AMI management: Scheduled AMI creation with Parameter Store updates

Walkthrough

This walkthrough demonstrates implementing Oracle HA using Amazon FSx for NetApp ONTAP shared storage, AWS Backup-driven AMI creation, Lambda orchestration, and Auto Scaling groups with Parameter Store integration for configuration consistency and automated failover.

Prerequisites

For this walkthrough, you should have the following prerequisites:

  • An AWS account with appropriate permissions for Amazon FSx, Auto Scaling, EC2, Lambda, and Systems Manager
  • A VPC with subnets in at least two Availability Zones
  • Oracle database software

Keep in mind that customers are responsible for their own Oracle licensing compliance.

  • An EC2 instance with Oracle database installed and configured
  • AWS Identity and Access Management (IAM) roles for AMI creation and cross-service communication
  • Basic knowledge of Oracle database administration and AWS automation

Assumptions

This post is a conceptual illustration of the architecture. Your specific implementation will vary based on your VPC layout, Oracle version, storage requirements, and organizational security policies.

We assume the reader is familiar with:

  • Creating and configuring Amazon FSx for NetApp ONTAP file systems through the AWS console
  • iSCSI concepts including initiators, targets, and multipath I/O
  • Oracle database startup and shutdown procedures
  • AWS Backup, Lambda, and Auto Scaling group fundamentals

For detailed step-by-step instructions on specific AWS services, refer to the additional resources section.

Step 1: Create an Amazon FSx for NetApp ONTAP file system

FSxN Multi-AZ provides the persistent shared storage foundation for this architecture. Unlike Amazon Elastic Block Store (Amazon EBS) volumes, which are bound to a single AZ, FSxN Multi-AZ replicates data synchronously across two AZs with automatic failover. This means that when an EC2 instance is replaced (whether in the same AZ or a different one), the new instance can immediately access the existing Oracle database files without restoring from backup.

To create the file system, navigate to the Amazon FSx console and select Amazon FSx for NetApp ONTAP as the file system type.

The critical configuration choice is selecting Multi-AZ deployment, which places an active file server in one AZ and a standby in another.

Amazon FSx console showing oracle-fsxn-multi-az file system configuration with ONTAP Multi-AZ 1 deployment, 1024 GiB SSD storage, 512 MB/s throughput, spanning us-east-1a preferred and us-east-1b standby subnets.

FSxN console showing Multi-AZ deployment type selection with preferred and standby subnets in separate availability zones.

After the file system is created, you need to set up a Storage Virtual Machine (SVM), which acts as a logical storage container providing data access to your Oracle instances. The SVM creation is done from the FSx console under your file system’s details.With the SVM in place, the next step is configuring iSCSI access. FSxN exposes iSCSI endpoints—these are IP addresses (one per AZ) that your EC2 instances use to connect to the storage over the iSCSI protocol. You can find these endpoint addresses in the FSx console under your SVM’s Endpoints tab.

Amazon FSx Storage Virtual Machine configuration page showing oracle-svm with Created lifecycle state, NFS, iSCSI, and management endpoints for Oracle Database storage connectivity.

SVM Endpoints tab showing iSCSI endpoint IP addresses for each availability zone. These addresses are used in the EC2 instance’s iSCSI discovery configuration.

The iSCSI setup involves creating iGroups (which define which EC2 instances can access the storage) and LUNs (logical storage units mapped to those groups) through the NetApp ONTAP CLI. On the EC2 side, you configure the iSCSI initiator to discover and connect to the FSxN endpoints, then mount the resulting block devices. Using multipath I/O with both endpoints makes sure that Oracle data remains accessible even during an AZ failover. For detailed iSCSI configuration steps, see mounting iSCSI LUNs on Linux clients.

A dedicated security group is required for FSxN access. At minimum, the security group must allow inbound traffic on ports 111 (NFS portmapper), 635 (NFS mountd), 2049 (NFS), 3260 (iSCSI), 4045–4046 (NFS lock), 443 (HTTPS for management), and 22 (SSH for ONTAP CLI). Restrict the source to only your Oracle EC2 instances’ security group.

Step 2: Set up AWS Backup for EC2 instance protection

AWS Backup captures the complete state of your Oracle EC2 instance. The key design choice here is using tag-based resource selection rather than specifying instance IDs directly. Because Auto Scaling groups replace instances (and generate new instance IDs), tag-based selection makes sure that any new instance with the correct tags are automatically included in the backup plan.Configure a backup plan with a frequency appropriate for your environment and set the resource assignment to select EC2 instances matching your application tag (for example, ‘Application: Oracle’).

AWS Backup console showing blog-test backup plan with hourly backup rule targeting Oracle EC2 instances identified by the Application:oracle-db tag.

AWS Backup resource assignment configured with tag-based selection. Any EC2 instances tagged with the application tag are automatically included in the backup plan.

Step 3: Configure Lambda for AMI management

When AWS Backup completes an EC2 backup, it creates an AMI as the recovery point. An Amazon EventBridge rule detects this completion event and triggers a Lambda function. The function extracts the AMI ID from the backup recovery point, updates the SSM Parameter Store parameter with the new AMI ID, and cleans up older AMIs to control storage costs.

AWS Lambda function configuration for oracle-backup-handler showing Python 3.11 runtime, EventBridge trigger, and description indicating it processes AWS Backup completion events and updates AMI in SSM.

Lambda function overview showing the EventBridge trigger, Python 3.11 runtime, and function description indicating its role in processing backup completions and updating AMI references in SSM.

This event-driven approach means the latest AMI is available without manual intervention. The Lambda function needs IAM permissions for EC2 (to manage AMIs), SSM (to update the parameter), and Backup (to read recovery point metadata).

Amazon EventBridge rule oracle-backup-completion configured to trigger the oracle-backup-handler Lambda function when AWS Backup completes an EC2 backup job, with event pattern filtering for COMPLETED state.

EventBridge rule configured to match AWS Backup job completion events for EC2 resources, with the Lambda function as the target.

Step 4: Configure the Systems Manager Parameter Store

The SSM Parameter Store holds the current AMI ID that the Auto Scaling group’s launch template references. The parameter is created with the aws:ec2:image data type, which enables the launch template’s resolve:ssm: functionality, a feature that allows the launch template to dynamically resolve the AMI ID at instance launch time without requiring a template version update.

AWS Systems Manager Parameter Store showing /oracle/ec2/ami-id parameter with AMI value ami-0a705a7d5523c555, version 857, last modified by the oracle-backup-lambda-role on April 25, 2026.

SSM Parameter Store showing the /oracle/ec2/ami-id parameter with aws:ec2:image data type. The “Last modified user” confirms the Lambda function is automatically updating this parameter after each backup cycle.

When Lambda updates this parameter after each backup cycle, the next instance launched by the Auto Scaling group will automatically use the latest AMI. This removes the operational burden of manually updating launch template versions.

Step 5: Set up an Auto Scaling Group with dynamic AMI

The launch template references the SSM parameter using the resolve:ssm: prefix for the AMI ID field. This is the mechanism that ties the entire automation pipeline together. The mechanism backups trigger AMI creation, AMI IDs flow into Parameter Store, and the launch template resolves the latest AMI at launch time.

EC2 Launch Template oracle-db-launch-template version 75 showing AMI ID resolved from SSM parameter resolve:ssm:/oracle/ec2/ami-id with r7i.large instance type for Oracle Database deployment.

Launch template AMI configuration showing the ‘resolve:ssm:’ prefix, which dynamically retrieves the latest AMI ID from Parameter Store at instance launch time.

The Auto Scaling group is configured with minimum, maximum, and desired capacity all set to 1. This is not traditional auto-scaling, it’s a self-healing pattern. The sole purpose is to detect when the Oracle instance becomes unhealthy and automatically launch a replacement. The health check grace period should be set to at least 300 seconds (5 minutes) to allow Oracle sufficient time to start before health checks begin evaluating the new instance.

The launch template also includes a User Data script that runs on each new instance. This script configures the iSCSI initiator, discovers and connects to the FSxN endpoints, mounts the Oracle data volumes, and starts the Oracle database through a systemd service. This automation makes sure that a replacement instance is fully operational without manual intervention.

EC2 Auto Scaling group oracle-db-asg configuration showing desired capacity of 1, scaling limits 1-1, r7i.large instance type, oracle-db-launch-template with Latest version, spanning two availability zone subnets.

Auto Scaling group configured with min=max=desired=1 across two availability zones, providing self-healing capability.

Test the complete workflow

To validate the architecture, simulate an instance failure by terminating the current Oracle EC2 instance.

The expected sequence is:

  1. The Auto Scaling group detects the instance is unhealthy (within approximately 30 seconds)
  2. A new instance launches from the latest AMI resolved from Parameter Store (approximately 2 minutes)
  3. The User Data script connects to FSxN using iSCSI and starts Oracle (approximately 2–3 minutes)
  4. The Oracle database is available and accepting connections (total elapsed: approximately 5 minutes)

Auto Scaling group Activity History showing the self-healing sequence — the unhealthy instance is terminated, and a replacement is launched automatically within seconds.

The new instance automatically inherits the application tags from the Auto Scaling group, which means AWS Backup includes it in the next backup cycle without manual configuration.

Cleaning up

To avoid incurring future charges, delete the resources:

  • Delete Lambda functions and EventBridge rules
  • Remove Parameters from Systems Manager Parameter Store
  • Delete AWS Backup plans and backup vault
  • Deregister created AMIs
  • Terminate Auto Scaling group instances
  • Delete the Amazon FSx for NetApp ONTAP file system

Conclusion

This architecture facilitates Oracle high availability with configuration consistency by combining FSxN persistent shared storage with automated AMI management and AWS Backup protection. The Lambda-driven AMI management from backup recovery points and Parameter Store integration helps make sure that replacement instances launched by Auto Scaling groups always use the latest Oracle host configuration and can immediately connect to the existing Oracle database files stored on FSxN. Replacements occur only when health checks fail. Organizations can target high availability while maintaining configuration consistency across instance replacements. The automated AMI management alleviates configuration drift and makes sure that disaster recovery scenarios restore Oracle instances with identical host-level configurations that can immediately access the persistent Oracle database on shared storage. Healthy instances continue running unchanged, with replacements occurring only, when necessary, because of health check failures.Next steps include implementing cross-Region AMI replication, adding AMI validation testing, and developing custom health checks that verify both Oracle database and host configuration consistency.

Additional resources

Identify unused AWS KMS keys and prevent accidental key deletions

Post Syndicated from Andrea Rossi original https://aws.amazon.com/blogs/security/identify-unused-aws-kms-keys-and-prevent-accidental-key-deletions/

As you scale your use of Amazon Web Services (AWS), managing KMS keys becomes increasingly important. Whether you manage a handful of keys or thousands across multiple AWS accounts and AWS Regions, there’s often a need to audit key usage to help you meet compliance requirements, evaluate your risk posture, and optimize key management costs. However, determining which keys are actively in use and which have been sitting idle can be a time consuming and complex task.

To help with this, AWS Key Management Service (AWS KMS) has launched the GetKeyLastUsage API, a new feature that you can use to quickly determine when each key was last used for a cryptographic operation, significantly enhancing your audit capabilities and key lifecycle management. For more information, see Determine past usage of a KMS key.

Before this launch, the primary way to audit key usage was through AWS CloudTrail logs. CloudTrail captures every cryptographic operation by default, so the data is available. The difficulty is turning that data into actionable insight. You need to identify which keys to examine, query the right logs, and repeat that process frequently enough to maintain an accurate view. For the most recent 90 days, CloudTrail event history makes this manageable. Beyond that, you need to create a dedicated trail to deliver logs to Amazon Simple Storage Service (Amazon S3) for long-term retention, then query those logs using tools such as Amazon Athena to determine when a key was last used.

Determine when a key was last used

AWS KMS now provides a direct way to see when a key was last used for cryptographic operations. You can also see this information using the AWS Management Console for AWS KMS and the AWS Command Line Interface (AWS CLI).

The GetKeyLastUsage API returns the date and time of the most recent cryptographic operation performed with a KMS key, without requiring you to search through CloudTrail logs. The API returns the date and time of the last key operation, the type of operation performed, CloudTrail event ID, and KMS request ID. You can access this information for all customer-managed keys and AWS managed keys irrespective of key spec, key origin, key store, or key usage type.

In addition, you can restrict a key from being disabled or scheduled for deletion if it was recently used, by incorporating this usage information as a condition within the KMS key policy. See the Preventing accidental key deletion with policy controls section for implementation details.

About the tracking period

One of the important concepts you must understand before relying on the last usage information reported on a KMS key is the tracking period. The tracking period is the date from which AWS KMS began tracking cryptographic activity for the key. Tracking began on April 23, 2026, for most AWS Regions. Understanding the tracking period is critical because it determines whether the absence of usage information means a key has never been used or only hasn’t been used since tracking started.

For example, if you have a key created on January 1, 2026, and you check its usage, any cryptographic operations that occurred between January 1 and April 22 wouldn’t be captured in the usage information. Thus, you can’t conclude that it’s never been used, because it might have been used in the months before tracking began.

Getting started

There’s nothing to enable or additional configuration required to view usage information on last cryptographic operation performed on your KMS keys.

To view KMS key usage:

  1. Go to the AWS KMS console and choose Customer-managed keys in the navigation pane and select a key. Look for Last used on the general configuration.
    Figure 1: KMS key general configuration page

    Figure 1: KMS key general configuration page

  2. Choose the link under Last used to see additional details such as Timestamp, Operation, and the CloudTrail event ID.
    Figure 2: View last used details including timestamp, operation, and event ID

    Figure 2: View last used details including timestamp, operation, and event ID

  3. The Last used column is also shown when you attempt to schedule key deletion, so that you can make informed decisions.
    Figure 3: Scheduled key deletion warning

    Figure 3: Scheduled key deletion warning

API reference

See the following examples for ideas on how to use the GetKeyLastUsage API to better understand KMS key usage.

Use case 1: Cost optimization through unused key cleanup

If you manage thousands of AWS KMS keys distributed across multiple AWS accounts, you might have keys that have remained unused since creation or keys that are no longer needed. By cleaning up these keys, you can reduce operational costs and minimize your security footprint. However, without visibility into which keys are actively performing cryptographic operations, it can be difficult to distinguish between keys protecting critical workloads and those that can be safely decommissioned.

Note that there are some precautions that you should take before scheduling key deletion. While the last usage information can help identify unused keys, it shouldn’t be the only factor in deciding whether to delete or disable a key. The last usage information tells you when a key was last used, not whether it will be needed in the future. A key might be unused for months but still required to decrypt files, for compliance scenarios or disaster recovery as shown in figure 4.

When you identify a potentially unused key, first disable it using DisableKey and monitor your applications and services for any encryption or decryption failures.

Figure 4: A use case where GetKeyLastUsage doesn’t accurately reflect whether a KMS key is still required

Figure 4: A use case where GetKeyLastUsage doesn’t accurately reflect whether a KMS key is still required

As an example, Amazon EBS volumes only interact with KMS keys during specific lifecycle events like volume creation, attachment, and detachment. After a volume is attached to an Amazon Elastic Compute Cloud (Amazon EC2) instance, the plaintext data encryption key is cached in the Nitro Card hardware, and all subsequent read/write operations use this cached key without any further AWS KMS API calls. This means a production volume running continuously for months or years will show no KMS activity during that entire period. However, the volume remains completely dependent on that KMS key for any future operations like instance restarts, volume reattachments, or disaster recovery scenarios. If someone deletes the KMS key, the encrypted data key stored with the volume can never be decrypted again, making the volume’s data permanently and irreversibly inaccessible. Before deleting any KMS key, you must verify it has no associated EBS volumes or snapshots, regardless of how long ago the last KMS API call occurred.

AWS provides a mechanism where you can create a CloudWatch alarm that notifies you if a key pending deletion is being accessed, giving you an opportunity to cancel the deletion before data becomes inaccessible.

Solution with GetKeyLastUsage API

Here’s a sample script that scans all customer-managed keys in an account and retrieves each key’s last usage date through the GetKeyLastUsage API. It accepts two optional inputs: a threshold in days and an AWS Region. The script filters and displays only keys that haven’t been used within the specified period, presenting results in a table with the key name, AWS account ID, AWS Region, and last usage date. This can help you identify unused encryption keys.

The following is an example to scan all keys that haven’t been used in the last 180 days in the us-east-1 Region:

./script.sh 180 us-east-1
#!/bin/bash
DAYS=${1:-90}
REGION=${2:-$(aws configure get region)}
CUTOFF=$(date -v-${DAYS}d +%s 2>/dev/null || date -d "-${DAYS} days" +%s)
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
printf "Showing keys not used in the last %s days (Region: %s)\n\n" "$DAYS" "$REGION"
printf "%-50s %-15s %-20s %-15s\n" "Key Name" "Account ID" "Region" "Last Usage Date"
printf "%.0s-" {1..100}
printf "\n"
for key_id in $(aws kms list-keys --region $REGION --query 'Keys[*].KeyId' --output text); do
key_manager=$(aws kms describe-key --region $REGION --key-id $key_id --query 'KeyMetadata.KeyManager' --output text)
if [ "$key_manager" = "CUSTOMER" ]; then
last_usage=$(aws kms get-key-last-usage --region $REGION --key-id $key_id)
timestamp=$(echo $last_usage | jq -r '.KeyLastUsage.TimeStamp // empty')
if [ -z "$timestamp" ]; then
last_epoch=0
else
last_epoch=$(date -jf "%Y-%m-%dT%H:%M:%S" "$(echo $timestamp | cut -d. -f1)" +%s 2>/dev/null || date -d "$timestamp" +%s)
fi
if [ "$last_epoch" -lt "$CUTOFF" ]; then
key_alias=$(aws kms list-aliases --region $REGION --key-id $key_id --query 'Aliases[0].AliasName' --output text)
key_name=${key_alias:-$key_id}
[ "$key_name" = "None" ] && key_name=$key_id
if [ -z "$timestamp" ]; then
tracking_date=$(echo $last_usage | jq -r '.TrackingStartDate' | cut -d'T' -f1)
last_used="${tracking_date}*"
else
last_used=$(echo $timestamp | cut -d'T' -f1)
fi
printf "%-50s %-15s %-20s %-15s\n" "$key_name" "$ACCOUNT_ID" "$REGION" "$last_used"
fi
fi
done
printf "\n* = No operations performed since tracking started\n"

Use case 2: Preventing accidental key deletion with policy controls

Organizations frequently face the risk of accidental key deletions, which can have severe operational consequences. Despite precautions and safety measures, accidents can happen. A key might be deleted because someone believes it’s no longer in use, only to discover that critical applications or workloads depend on it. This results in data access failures, application downtime, and emergency recovery procedures. Without visibility into recent key usage, teams lack the information needed to make safe disable decisions or implement effective safeguards.

Solution with policy based controls

To prevent KMS keys from being accidentally Disabled or Deleted use the kms:TrailingDaysWithoutKeyUsage condition key in key policies to automatically block deletion or disabling of recently used keys:

  1. Open the AWS KMS console and choose Customer managed keys in the navigation pane.
  2. Select the key you want to protect.
  3. In the Key policy tab, choose Edit.
  4. In the policy editor, add the following statement:
{
  "Sid": "PreventDeletionOfRecentlyUsedKeys",
  "Effect": "Deny",
  "Principal": "*",
  "Action": [
    "kms:ScheduleKeyDeletion",
    "kms:DisableKey"
  ],
  "Resource": "*",
  "Condition": {
    "NumericLessThanEquals": {
      "kms:TrailingDaysWithoutKeyUsage": "365"
    }
  }
}
  1. Choose Save changes.

The policy prevents deletion or disabling a key if it was used within the past 365 days. You can adjust the threshold to match your organization’s requirements. For more information about the condition key, see kms:TrailingDaysWithoutKeyUsage.

Important considerations

When reviewing key usage for possible deletion, consider the following:

  • Key deletion is irreversible and makes encrypted data unrecoverable. AWS enforces a 7–30 day waiting period. During this time, monitor usage attempts and cancel the deletion if necessary. Delete a key only if you’re certain that no data has been encrypted or will be encrypted with it. Consider disabling the key first to test the impact of unavailable keys.
  • CloudTrail remains authoritative because it provides the full audit trail. GetKeyLastUsage quickly tells you when and what operations occurred, but CloudTrail shows you who made the request and with what parameters. Learn more about logging KMS API calls with CloudTrail.

Conclusion

The GetKeyLastUsage API enhances your KMS key management capabilities by providing immediate access to usage data that was previously only present in CloudTrail logs. Start by opening the AWS KMS console and checking the Last used field for any customer-managed keys and AWS managed keys to see this information in action. For broader key auditing, integrate the API into your existing automation scripts using the AWS CLI examples provided.

If you have feedback about this post, submit comments in the Comments section below.


Andrea Rossi

Andrea Rossi

Andrea is the Solutions Architect who always asks “but is it secure?” one more time. Based in Milan, Italy, he works with customers to architect cloud solutions where security is foundational, not an afterthought, from network-level hardening to integrating Generative AI workloads into compliant environments.

Poojil Tripathi

Poojil Tripathi

Poojil is a Solutions Architect based in Austin, TX, who would like to remind you that you should never click on links. He works with customers to design secure-by-design cloud solutions on AWS, specializing in encryption and healthy paranoia.

Automating contract intelligence with Doczy.ai™ on AWS

Post Syndicated from Sanket Nasre original https://aws.amazon.com/blogs/architecture/automating-contract-intelligence-with-doczy-ai-on-aws/

Extracting actionable insights from thousands of contracts and legal documents remains a challenge. For organizations, critical business information is locked in unstructured documents such as contracts, legal agreements, provider arrangements, and vendor invoices. Extracting and operationalizing this information has traditionally been a manual, error-prone, and resource-intensive process. This leads to missed savings opportunities, costly delays, and significant inefficiencies across the enterprise.

AArete, a global management and technology consulting firm specializing in healthcare, recognized this challenge and developed Doczy.ai™, an intelligent contract interpretation solution powered by generative AI on Amazon Web Services (AWS).

In this post, we show you how Doczy.ai™ uses generative AI on AWS to automate contract intelligence at scale, transforming unstructured documents into structured, actionable insights, so organizations can automate critical business processes and unlock the full value of their data.

The challenge: Data trapped in documents

For healthcare organizations, managing and interpreting contracts and documents represents a major operational bottleneck. Manual review processes require deploying teams to extract data from thousands of documents. This is an approach that is neither scalable nor sustainable, highly prone to error, and costly. Organizations relying on institutional knowledge face additional risks: critical information resides with a few key individuals, creating knowledge silos and succession planning challenges. Existing Contract Lifecycle Management (CLM) systems often prove inadequate for capturing the nuanced and complex terms unique to each agreement. These legacy systems can only configure predefined fields, missing the rich detail and contextual information that distinguishes contracts. The downstream impact is substantial: in healthcare, reimbursement terms must be manually translated into claims systems—a slow, error-prone process. Similarly, verifying vendor invoices against contract terms often requires manual effort, leading to payment processing delays and missed contractual savings opportunities. These inefficiencies ultimately leave significant value on the table.

This is where Doczy.ai™ provides significant value.

Doczy.ai™: An intelligent contract interpretation solution

Doczy.ai™ directly addresses these challenges using advanced AI and scalability on AWS. Developed by AArete, Doczy.ai™ pushes the boundaries of document intelligence. The solution automatically interprets complex documents and converts them into a structured, queryable information repository that allows organizations to unlock the full value of their data and drive smarter decisions.The evolution of Doczy.ai™ reflects rapid AI advancement. Prior to 2020, document processing required manual effort, with individuals processing approximately 100 documents per week. Between 2020–2023, the firm implemented rules-based contract processing, achieving approximately 55% accuracy. The breakthrough came in 2024 with an AI-based processing built on AWS achieved 99% accuracy—a dramatic improvement over the 55% accuracy of traditional rules-based systems.

Doczy.ai™ architecture

Doczy.ai™ is built on a comprehensive AWS architecture designed to handle the entire document processing lifecycle: from the moment a file enters the system to the moment it generates actionable business intelligence.

Doczy.ai is built on a comprehensive AWS architecture designed to handle the entire document processing lifecycle: from the moment a file enters the system to the moment it generates actionable business intelligence.

Architecture of Doczy.ai™

External users access the platform through a secure Next.js frontend, with Amazon Cognito managing authentication and authorization behind the scenes. After authentication, users upload documents directly to Amazon Simple Storage Service (Amazon S3), where durable, scalable object storage ensures nothing is lost and everything is accessible at scale. From there, the real intelligence begins.

An AWS Lambda function triggers Amazon Textract to extract text and metadata from documents in various formats. What sets Doczy.ai™ apart at this stage is its patented “smart chunking” algorithm, a proprietary approach that goes far beyond pulling words off a page. Rather than treating a document as a flat sequence of text, smart chunking preserves hierarchical structure and one-to-many relationships within documents. It uses a combination of semantic and keyword search to decompose text into meaningful, context-aware chunks, applying dynamic parameters to maintain logical relationships throughout. Sequential identifiers and metadata-driven grouping organize these chunks into field groups, detecting overlaps and removing duplications while keeping the document’s natural flow intact.

After chunking, the document enters the dual clustering engine of Doczy.ai™. This two-lens methodology analyzes every contract simultaneously from both a semantic and a structural perspective. On the semantic side, extracted text is converted into embeddings, numerical representations of meaning, and similar ideas are grouped together even when they’re expressed in different words. On the structural side, pattern-recognition algorithms identify clause types, formatting conventions, table layouts, and hierarchical organization, understanding. For example, that a three-nested-level exhibit carries fundamentally different implications than a straightforward attached schedule.These two analyses don’t operate in isolation. Projection algorithms compare the semantic and structural clusters side by side, synthesizing them into a unified, enriched document model that captures both meaning and context. It’s this convergence that drives the 99% accuracy rate of Doczy.ai™. The system doesn’t just read the words, it understands the contract. Advanced large language models (LLMs) then generate structured output grounded in this dual-clustered intelligence.Before output is finalized, the system determines each document’s file class and generates prompts tailored to the extracted text, cluster classification, and domain context. Through few-shot and multi-shot prompting, the platform continuously edits the prompt on domain-specific examples and based on real outputs, creating a feedback loop that compounds accuracy improvements over time.

The resulting structured data flows into Snowflake, forming a centralized repository that powers intelligent dashboards with actionable insights and visualizations. Throughout the entire pipeline, Amazon CloudWatch monitors performance in real time and proactively surfaces issues before they escalate, while AWS Secrets Manager safeguards sensitive information, ensuring that security is not an afterthought, but a foundational layer woven into every stage of the system.

The transformative impact of Doczy.ai™

The results of this AI-powered approach are transformative and measurable. By automating contract interpretation and document processing, Doczy.ai™ has demonstrated significant impact at scale for multiple organizations across healthcare and financial services. The scale of operations over the last 22 months demonstrates the maturity and production readiness of Doczy.ai™. This solution has processed 2.5 million contract documents (50 million pages) with 137 million API calls to Amazon Bedrock and 442 billion tokens—a level of automation and accuracy previously unattainable through manual or traditional document processing approaches. Over this same period, Doczy.ai™ has helped clients achieve approximately 330 million dollars in cumulative direct and indirect savings.The 99% accuracy rate represents significant improvement over the approximately 55% accuracy of rules-based systems and far exceeds manual processing, which is typically affected by fatigue and human error. The 97% reduction in manual processing time translates directly to cost savings and enables organizations to reallocate human resources to higher-value activities that require judgment and strategic thinking.

A use case in action: Business process automation for health plans

For health plans, Doczy.ai™ provides a powerful solution to automate and improve contract management across the entire lifecycle. It ingests existing contracts in both paper and digital formats, integrates with contract management systems such as Coupa and Icertis, and processes new contracts and amendments as they’re executed. It then creates a centralized metadata repository that feeds directly into downstream systems, enabling end-to-end business process automation.This automation unlocks critical capabilities: Organizations can continuously analyze and improve contract terms, identifying opportunities to improve financial performance and operational efficiency. The architecture feeds accurate, up-to-date contract data directly into claims systems, automating the configuration process that previously required manual translation of reimbursement terms and removing manual data entry, configuration errors, and delays. Additionally, the platform helps maintain claim payment accuracy by assessing payments against contract terms, identifying discrepancies, and flagging potential overpayments or underpayments before they occur.By automating manual processes, health plans can adapt quickly to new contract terms and regulatory requirements. The intelligent dashboards and actionable insights provided by Doczy.ai™ enable decision-makers to understand contract performance, identify trends, and take proactive action to optimize financial outcomes.

Getting started with Doczy.ai™

Organizations interested in using Doczy.ai™ to transform document processing and contract management can engage with AArete to discuss their specific use cases and requirements. AArete offers the platform as a Software as a Service (SaaS) solution, enabling rapid deployment without significant infrastructure investment. AArete’s team of experts will configure this solution for your specific document types, domain terminology, and business processes, supporting maximum value from day one.

Conclusion

The challenge of unlocking data from unstructured documents is a major hurdle for many businesses, particularly in healthcare and financial services where contracts and agreements govern critical operational and financial relationships. By embracing intelligent document intelligence on AWS, organizations can solve this long-standing operational challenge and unlock a new frontier of strategic advantage, turning their data into their most valuable asset.

Built on a sophisticated architecture that orchestrates Amazon Cognito, Amazon S3, AWS Lambda, Amazon Textract, Amazon Elastic Container Service (Amazon ECS), Amazon Bedrock, Amazon CloudWatch, and AWS Secrets Manager, Doczy.ai™ demonstrates how modern cloud services can solve complex document-heavy business problems. Its advanced hybrid smart chunking, dual clustering, and prompt optimization techniques form the core of a patented contract intelligence engine.

Doczy.ai™ delivers tangible impact, processing up to 250,000 contract documents per week with 99% accuracy, reducing manual processing time by 97%, and helping clients unlock roughly 330 million dollars in cumulative savings over 22 months. By embracing this intelligent document processing, organizations can turn contracts into a strategic data asset, improving efficiency, accuracy, and profitability while freeing teams to focus on higher-value work.

To learn more about how AArete and Doczy.ai™ can help your organization transform document processing and unlock the value of your unstructured data, visit the AArete website.


About the authors

Multi-Region event-driven failover architecture with Amazon EventBridge and Route 53

Post Syndicated from Napoleone Capasso original https://aws.amazon.com/blogs/compute/multi-region-event-driven-failover-architecture-with-amazon-eventbridge-and-route-53/

Multi-Region Event-Driven Failover Architecture with Amazon EventBridge and Route 53

Event-driven architectures enable applications to respond to events in real-time, providing scalability and loose coupling between components. However, ensuring high availability across multiple AWS regions requires careful design of failover mechanisms. This post demonstrates how to build a resilient multi-region event-driven architecture using Amazon EventBridge, Amazon API Gateway, and Amazon Route 53 health-based failover.

Overview

Organizations building event-driven applications need to achieve high availability and disaster recovery capabilities. This architecture provides automatic failover between AWS regions while maintaining regional independence for event processing. The solution uses Amazon Route 53 health checks to monitor regional Amazon API Gateway endpoints and automatically routes traffic to healthy regions without manual intervention.

The architecture delivers several key benefits. Regional independence reduces latency by processing events in the same region where they originate. Amazon DynamoDB global tables provide automatic data replication across regions, ensuring data availability during regional failures. The solution provides robust failover capabilities while maintaining architectural simplicity.

Organizations with strict availability requirements can find this solution particularly valuable. All event processing remains within AWS regions, and failover occurs automatically based on health check results. The architecture supports both planned maintenance windows and unplanned regional outages, providing flexibility for operational needs.

Solution overview

The solution implements an active-passive multi-region architecture where events flow through Amazon API Gateway to regional Amazon EventBridge buses. Amazon Route 53 health checks monitor the primary region and automatically route traffic to the secondary region during failures. Each region processes events independently, while Amazon DynamoDB Global Tables replicate data across regions.

The following diagram provides an overview of the solution:

The above diagram depicts the multi-region architecture running across two AWS regions. The Route 53 DNS service serves as the main entry point for the application, with health checks monitoring both regions. Each region contains an identical stack with Amazon API Gateway, Amazon EventBridge, Amazon SQS, and AWS Lambda. The Amazon DynamoDB Global Table replicates data between regions automatically.

Solution deployment

To deploy this solution, follow the instructions in the GitHub repository and clone the repository. The solution deploys in two AWS regions. Ensure valid SSL certificates exist in AWS Certificate Manager (ACM) in both regions for the custom domain.

Prerequisites

For this walkthrough, the following resources are needed:

  • AWS Account: An AWS account with permissions to create and manage Amazon API Gateway, Amazon EventBridge, Amazon SQS, AWS Lambda, Amazon DynamoDB, Amazon Route 53, AWS IAM, and AWS CloudFormation resources
  • AWS Serverless Application Model (SAM): The AWS SAM CLI installed, as the templates use the SAM transform for Lambda and API Gateway resource definitions
  • Domain Name: A registered domain with a Route 53 hosted zone- SSL Certificates: ACM certificates for the custom domain in both deployment regions
  • AWS CLI: The AWS CLI installed and configured with credentials for the target AWS account
  • Region Selection: Two AWS regions for deployment

Walkthrough

The AWS CloudFormation templates from the sample GitHub repository create a secure, multi-region architecture that provides automatic failover for event-driven applications. The templates provision regional API Gateway endpoints, EventBridge buses, SQS queues, Lambda functions, and an Amazon DynamoDB Global Table. The solution establishes health monitoring through Route 53 health checks and configures DNS failover routing. The templates use AWS Serverless Application Model (SAM) transform to simplify Lambda and API Gateway resource definitions.

Step 1: Deploy the primary stack

The primary stack creates the foundational resources in the primary region. This includes the Amazon EventBridge bus, Amazon API Gateway with custom domain, health check, AWS Lambda function, Amazon SQS queue, and Amazon DynamoDB Global Table. The stack creates an EventBridge bus that receives events from API Gateway:

EventBus: 
Type: AWS::Events::EventBus 
Properties: 
Name: !Ref EventBusName

The API Gateway uses AWS service integration to forward events directly to EventBridge:

x-amazon-apigateway-integration: 
type: "aws" 
uri: !Sub "arn:aws:apigateway:${AWS::Region}:events:path//" 
credentials: !GetAtt ApiGatewayEventBridgeRole.Arn 
httpMethod: "POST"

The health check monitors the API Gateway endpoint to determine regional availability:

DomainHealthCheck: 
Type: AWS::Route53::HealthCheck 
Properties: 
HealthCheckConfig: 
Type: HTTPS 
ResourcePath: /Prod/health FullyQualified
DomainName: !Sub ${Api}.execute-api.${AWS::Region}.amazonaws.com 
Port: 443 
RequestInterval: 30 
FailureThreshold: 3

The Route 53 DNS record configures failover routing with the PRIMARY designation:

ApiDnsRecord:
Type: AWS::Route53::RecordSet
Properties:
HostedZoneId: !Ref HostedZoneId
Name: !Ref CustomDomainName
Type: A
SetIdentifier: primary-region
Failover: PRIMARY
HealthCheckId: !Ref DomainHealthCheck

The DynamoDB Global Table creates replicas in both regions:

DataTable: 
Type: AWS::DynamoDB::GlobalTable 
Properties: 
BillingMode: PAY_PER_REQUEST 
Replicas: 
- Region: !Ref AWS::Region 
- Region: !Ref SecondaryRegion

Note the `DataTableName` output value for use in the secondary stack deployment. The `CustomDomainURL` output provides the endpoint to invoke the solution.

Step 2: Deploy the secondary stack

The secondary stack creates identical resources in the secondary region , except for the Amazon DynamoDB table which references the existing Global Table. The secondary stack creates its own Amazon EventBridge bus, Amazon API Gateway, health check, AWS Lambda function, and Amazon SQS queue. The Route 53 DNS record uses the SECONDARY designation

Step 3: Event processing flow

Events flow through the processing pipeline in each region. API Gateway receives events and forwards them to EventBridge using the PutEvents API. EventBridge evaluates event rules and routes matching events to SQS queues. Lambda functions poll the SQS queues and process events in batches. AWS Lambda writes processed data to the DynamoDB Global Table, which replicates across regions.

The Lambda function processes events from the queue and writes to DynamoDB:

def handler(event, context): 
for record in event.get('Records', []): 
body = json.loads(record['body']) 
detail = body.get('detail', {}) 
event_id = body.get('id', '') 
item = { 'id': event_id, 'detail': detail, 'timestamp': datetime.utcnow().isoformat() } 
table.put_item(Item=item)

Testing

Fetch the custom domain URL and test it by sending an event:

curl -X POST https://api.example.com \-H "Content-Type: application/json" \ -d '{ "Detail": { "IsHelloWorldExample": "true" }, "DetailType": "POSTED", "Source": "demo.event" }' -v

The response includes an `X-Region` header indicating which region processed the request. Under normal conditions, this shows the primary region.

To test failover:

  1. Remove the base path mapping for the primary region:
aws apigateway delete-base-path-mapping \ --domain-name api.example.com \ --base-path '(none)' \ --region {primary-region}
  1. Delete the primary API Gateway stage:

aws apigateway delete-stage \ --rest-api-id <primary-api-id> \ --stage-name Prod \ --region {primary-region}

  1. Wait 2-3 minutes for the health check to fail. The Route 53 health check performs checks every 30 seconds with a failure threshold of 3, requiring 90 seconds to detect the failure.
  2. Send another request to the API endpoint:
curl -X POST https://api.example.com \-H "Content-Type: application/json" \ -d '{ "Detail": { "IsHelloWorldExample": "true" }, "DetailType": "POSTED", "Source": "demo.event" }' -v
  1. Verify the failover: The `X-Region` header now shows the secondary region, confirming successful failover.

Verify event processing in the secondary region:

  1. Check the Lambda logs for successful processing:

aws logs tail /aws/lambda/<secondary-lambda-name> --region {secondary region}

You should see log entries similar to:

Processing message: 
{"version":"0",
"id":"abc12345-...",
"source":"demo.event",
"detail-type":"POSTED",...} 
Event Source: demo.event
Detail Type: POSTED
Successfully wrote item to DynamoDB: abc12345-... 
Successfully read item from DynamoDB: 
{'id': 'abc12345-...', 
'source': 'demo.event', 
'detailType': 'POSTED', 
'detail': 
{'data': {'IsHelloWorldExample': 'true'}, 
...}, 
'timestamp': '2025-01-15T18:30:00.000000', 
'processed': True}
  1. Verify the data in Amazon DynamoDB:

aws dynamodb scan \ --table-name <table-name> \ --region {secondary region}```

The scan results should include items with the event details:

{ "Items": 
[ { "id": {"S": "abc12345-..."}, 
"source": {"S": "demo.event"}, 
"detailType": {"S": "POSTED"},
"detail": 
{"M": {"data": 
{"M": 
{"IsHelloWorldExample": 
{"S": "true"}}}}}, 
"timestamp": {"S": "2025-01-15T18:30:00.000000"},
"processed": {"BOOL": true} } ], 
"Count": 1 }
  1. Restore the primary region – recreate the stage:

aws apigateway create-stage \ --rest-api-id <primary-api-id> \ --stage-name Prod \ --deployment-id <deployment-id> \ --region {primary region}

  1. Restore the primary region – recreate the base path mapping:

aws apigateway create-base-path-mapping \ --domain-name api.example.com \ --rest-api-id <primary-api-id> \ --stage Prod \ --region {primary region}

You can find the “deployment-id” by running: aws apigateway get-deployments \ --rest-api-id <primary-api-id> \ --region {primary region}

After 2-3 minutes, the health check passes and Route 53 routes traffic back to the primary region.

Cleanup

To remove the solution and avoid ongoing charges, delete the CloudFormation stacks in the correct order. Delete the secondary stack first, then the primary stack. This order is important because the Amazon DynamoDB Global Table is owned by the primary stack. Warning: Deleting these stacks permanently removes all resources including the Amazon DynamoDB global table and any event data stored in it. Back up any data you need before proceeding. This action cannot be undone. The following resources incur costs while deployed:

  • Amazon API Gateway (REST API)
  • Amazon Route 53 health checks and DNS records
  • Amazon DynamoDB global table (with cross-region replication)
  • AWS Lambda function invocations and duration
  • Amazon SQS queue operations
  • Amazon CloudWatch Logs storage

Delete the secondary stack:

aws cloudformation delete-stack --stack-name secondary-stack --region {secondary region}

Wait for the secondary stack deletion to complete:

aws cloudformation wait stack-delete-complete --stack-name secondary-stack --region {secondary region}

Delete the primary stack:

aws cloudformation delete-stack --stack-name primary-stack --region {primary region}

Wait for the primary stack deletion to complete:

aws cloudformation wait stack-delete-complete --stack-name primary-stack --region {primary region}

This removes all resources including the Amazon EventBridge buses, Amazon API Gateways, AWS Lambda functions, Amazon SQS queues, Amazon DynamoDB Global Table, Amazon Route 53 health checks, DNS records and IAM roles.

Conclusion

This post demonstrates how to establish a resilient multi-region architecture for event-driven applications using Amazon EventBridge, Amazon API Gateway, and Amazon Route 53. The solution uses Route 53 health-based failover, a powerful capability that automatically routes traffic to healthy regions based on health check results. This architecture significantly enhances application availability by providing automatic failover during regional outages while maintaining regional independence for event processing.

Building a scalable user search layer on top of Amazon Cognito

Post Syndicated from Philip Chen original https://aws.amazon.com/blogs/architecture/building-a-scalable-user-search-layer-on-top-of-amazon-cognito/

Imagine a teammate who needs to find a user across thousands of accounts with only a partial email address, a last name, and a known access level. How quickly can your team respond? If your use case involves straightforward searches on standard Amazon Cognito attributes, the built-in ListUsers API is likely all you need. But for advanced scenarios involving custom attributes, fuzzy matching, complex filtering, and sub-second response times, a dedicated search layer is the right investment.

Amazon Cognito provides robust user authentication and management capabilities for modern applications. As applications scale, development teams typically implement advanced search functionality to find users by partial email match, segment group membership, or audit across multiple custom attributes.

In this post, we show how to build a comprehensive scalable user search layer on top of Amazon Cognito using AWS Lambda, Amazon DynamoDB, and Amazon OpenSearch Service.

Solution overview

This solution extends Amazon Cognito with advanced search capabilities using AWS Lambda, Amazon DynamoDB, and Amazon OpenSearch Serverless.

Key capabilities:

  • Multiple search types: Exact match, prefix match, and fuzzy search
  • Complex filtering: Query across email, phone, groups, and registration date simultaneously
  • High performance: Sub-second response times at any scale
  • Automatic synchronization: Real-time updates as users authenticate or update profiles
  • API-driven: RESTful API with pagination support

The architecture uses Cognito Lambda triggers to capture user data during authentication, stores it in DynamoDB, and indexes it in OpenSearch Serverless through DynamoDB Streams. The following architecture diagram illustrates how these components work together.

Figure 1: Solution architecture for Searchable Cognito Users

Walkthrough

The solution architecture demonstrates two flows: Ingestion flow and Search flow.

Ingestion flow

The ingestion flow captures and indexes user data through two paths: Cognito Lambda triggers and AWS CloudTrail. Together, these paths maintain synchronization between the search index and Cognito without requiring manual intervention or scheduled batch jobs.

1. Cognito Lambda triggers

This path captures user data during authentication events using a Cognito trigger Lambda function that handles two trigger types: Post-confirmation and Pre-token generation. The post-confirmation trigger creates the initial user record on sign-up, while the pre-token generation trigger tracks login activity and app client information on each subsequent authentication. The pre-token generation trigger also provides access to the user’s group membership in the event payload, which is indexed as a searchable field. The flow operates through the following steps:

  1. Client initiates sign-up or login — User submits authentication request to Amazon Cognito.
  2. Post-confirmation trigger — On sign-up, Cognito invokes the Cognito trigger Lambda which creates the initial user record in the DynamoDB user table with profile attributes (email, name, groups).
  3. Pre-token generation trigger — On each login, Cognito invokes the Cognito trigger Lambda which updates the user’s login timestamp and app client information in the DynamoDB user table.
  4. Stream processing — DynamoDB Streams detects the new or updated record and triggers the OSS ingest Lambda.
  5. Index updated — OSS ingest Lambda processes the stream event and indexes the user data in OpenSearch Serverless.

Note: The Cognito Lambda triggers are deployed in a VPC. Cognito enforces a 5-second timeout on trigger functions. If you’re extending these triggers with additional functionality or already using post-confirmation or pre-token generation triggers, ensure the combined execution time stays well within this limit. Consider provisioned concurrency if cold starts are a concern.

Figure 2: User Data Ingestion via Cognito Lambda Triggers

2. CloudTrail

This path captures admin-initiated user changes that occur outside the authentication flow, such as creating users using the Cognito console or CLI. These actions don’t trigger Cognito Lambda triggers, so CloudTrail and EventBridge bridge the gap. The flow operates through the following steps:

  1. Admin action performed — User performs an admin action in Amazon Cognito (for example, create user, update attributes, add to group, disable user).
  2. API call logged — AWS CloudTrail captures the Cognito admin API call.
  3. EventBridge rule matched — An Amazon EventBridge rule matches the Cognito admin event.
  4. CloudTrail event Lambda invoked — EventBridge invokes the CloudTrail event consumption Lambda, which reads the current user state from Cognito and upserts the profile in the DynamoDB user table.
  5. Stream change event — DynamoDB Streams emits the change event.
  6. Invoke OSS Lambda — The stream event triggers the OSS ingest Lambda.
  7. Index user data — OSS ingest Lambda indexes the updated user data in OpenSearch Serverless.

Figure 3: User Data Ingestion via CloudTrail

Figure 4: Data model for indexed user attributes in Amazon DynamoDB

Search flow

With the search flow, authorized users can query the indexed user directory:

  1. Query submission — Authenticated user submits search query through the UI.
  2. Request validation — API Gateway receives the request with the Cognito JWT token and validates it using the Cognito authorizer.
  3. Search execution — Upon successful validation, the search Lambda function is invoked with the search parameters.
  4. OpenSearch query — Lambda assumes a read-only role for OpenSearch Service access and executes the query against the OpenSearch Serverless index.
  5. Results returned — Lambda formats and returns the query results to the frontend, where the UI displays them in a paginated format.

Figure 5: Search Flow Sequence Diagram

Figure 6: Demo UI user search integration on multiple properties

Figure 7: Demo UI user search integration on auto-suggest

Try it yourself

Ready to see this solution in action? The repository includes everything you need to deploy a complete working implementation in your own AWS environment.

The source code for this solution is available on GitHub at: https://github.com/aws-samples/sample-user-search-layer-for-cognito.

The repository includes everything you need: AWS CDK infrastructure code, Lambda function implementations, a React frontend, and documentation. You can have a fully functional searchable user directory running in your account in under 20 minutes. When you’re finished testing, clean up all resources to avoid ongoing charges.

Conclusion

In this post, you learned how to extend Amazon Cognito with advanced search capabilities. By combining OpenSearch Serverless, DynamoDB Streams, and Lambda functions, you can build a scalable, event-driven architecture that automatically maintains a searchable user directory with sub-second query performance.

This pattern unlocks powerful use cases: support teams can quickly locate users across thousands of accounts, administrators can segment users by group membership for targeted communications, and compliance teams can audit user attributes with complex filtering.

To dive deeper into the AWS services powering this solution:

About the authors

Simplifying policy management with URL and Domain Category filtering on AWS Network Firewall

Post Syndicated from Lawton Pittenger original https://aws.amazon.com/blogs/security/simplifying-policy-management-with-url-and-domain-category-filtering-on-aws-network-firewall/

Network administrators face a persistent challenge: maintaining domain blocklists and allowlists that keep pace with the internet. New websites and services emerge daily, and keeping these lists current requires constant manual updates that leave gaps in coverage. This challenge intensifies when managing access to rapidly evolving categories like AI services, where new tools launch on a regular basis.

AWS Network Firewall is a managed, stateful network firewall and intrusion detection and prevention service for fine-grained control of your virtual private cloud (VPC) network traffic. With URL and domain category filtering, security teams can use predefined categories to control access instead of managing individual domains. AWS-managed URL and domain categories stay current automatically as new domains are registered, removing the need for manual list maintenance.

This feature is especially useful for organizations navigating AI governance. Instead of manually tracking every new AI service, you can control access to the entire Artificial Intelligence and Machine Learning category while creating exceptions for approved services. The same approach works for social media, streaming sites, gambling, and dozens of other categories, all with built-in audit trails for compliance reporting.

In this post, we walk through URL and domain category filtering configurations for AWS Network Firewall, from basic rules to exception handling and monitoring strategies that give you visibility into how your workloads interact with external services.

Streamlined policy management with predefined categories

With URL and domain category filtering, you control website access using predefined categories instead of individually specifying sites in a domain list rule group. You can select from AWS-managed categories such as Social Networking, Gambling, or Artificial Intelligence and Machine Learning to implement and maintain filtering policies. AWS keeps these categories current automatically, so you don’t need to update firewall policies when new domains are registered.

Network Firewall offers two category filtering options. Domain category filters by domain name using the TLS Server Name Indication (SNI) field, with no decryption required. URL category filters by the full URL path, which requires TLS inspection for HTTPS traffic. To keep things straightforward, this post focuses on domain category filtering. To set up URL category filtering with TLS inspection, see Creating a TLS inspection configuration in Network Firewall.

Prerequisites

To follow the steps in this post, start by making sure that you have the following prerequisites in place:

  1. An existing Network Firewall deployment: This walkthrough assumes you have an existing Network Firewall deployment to filter egress traffic flows from your Amazon Virtual Private Cloud (Amazon VPC) in place. If you aren’t already using Network Firewall, see Getting started with AWS Network Firewall to set up your firewall before proceeding.
  2. The HOME_NET variable set correctly at the firewall policy level: The rules in this post use the $HOME_NET variable to scope traffic to your internal network. In the AWS Management Console for Amazon VPC, select your firewall policy under the Firewall policies tab, select the Details tab, and check the policy variables section under HOME_NET variable override values. We recommend setting this to all RFC 1918 private IP address ranges: 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. When you set $HOME_NET at the policy level, all rule groups associated with that policy inherit the value automatically. Network Firewall automatically maps $EXTERNAL_NET to the inverse of $HOME_NET, so configuring HOME_NET correctly also configures $EXTERNAL_NET.
Figure 1: Firewall policy details tab showing the HOME_NET variable override values set to RFC 1918 private IP address ranges

Figure 1: Firewall policy details tab showing the HOME_NET variable override values set to RFC 1918 private IP address ranges

Create a category rule using the console rule builder

To get started quickly, you can create a domain category rule using the console’s built-in rule builder. In this example, we create a single alert rule for the Artificial Intelligence and Machine Learning category.

  1. Open the AWS Management Console, search for and open the Amazon VPC console.
  2. In the left navigation, scroll to Network Firewall and select Rule groups.
  3. Choose Create rule group.
  4. For Rule group type, select Stateful rule group.
  5. For Rule group format, select Standard stateful rules.
  6. For Rule evaluation order, select Strict order. Choose Next.
    Figure 2: Create Network Firewall rule group page showing Stateful rule group type, Standard stateful rules format, and Strict order evaluation selected

    Figure 2: Create Network Firewall rule group page showing Stateful rule group type, Standard stateful rules format, and Strict order evaluation selected

  7. Enter Domain-Category-Rules for the Name, Domain Category Rules for the Description, and 50 for the Capacity. Choose Next.
  8. In the rule group editor, select the Category Matching radio button.
  9. Under Category Matching, select Match all selected categories.
  10. Under AWS category type, select Domain Category from the dropdown.
  11. Under Categories, select Artificial Intelligence and Machine Learning.
  12. For Protocol, select TLS.
  13. For Source, select Custom, then enter $HOME_NET in the dialog box.
  14. Set the Destination IP to Any.
  15. For Action, select Alert.
  16. Choose Add rule to add this rule to the rule group. Choose Next.
    Figure 3: Completed category matching rule showing TLS protocol, $HOME_NET source, Any destination, and Alert action added to the rule group

    Figure 3: Completed category matching rule showing TLS protocol, $HOME_NET source, Any destination, and Alert action added to the rule group

  17. Under Customer managed key, leave the default setting (Customize encryption settings should remain unchecked).
  18. Under Add tags – optional, leave the default setting of no tags.
  19. Choose Next, then Create rule group.

This rule generates an alert log entry each time a connection matches a domain in the Artificial Intelligence and Machine Learning category. It doesn’t block traffic. To block traffic, change the action to Drop or Reject in step 15.

Creating the same rule using Suricata compatible rule strings

The console rule builder is a quick way to get started, but we recommend using Suricata compatible rule strings for production deployments. Suricata rules give you full control over rule options, make rules straightforward to copy, edit, share, and back up, and support the majority of the Suricata engine. For more information, see Limitations and caveats for stateful rules in AWS Network Firewall.

The following walkthrough creates the same alert rule you built with the console rule builder, this time using a Suricata rule string.

In the Amazon VPC console, navigate to Network Firewall, then select Network Firewall rule groups.

  1. Choose Create rule group.
  2. For Rule group type, select Stateful rule group.
  3. For Rule group format, select Suricata compatible rule string.
  4. For Rule evaluation order, select Strict order. Choose Next.
    Figure 4: Create Network Firewall rule group page showing Stateful rule group type, Suricata compatible rule string format, and Strict order evaluation selected

    Figure 4: Create Network Firewall rule group page showing Stateful rule group type, Suricata compatible rule string format, and Strict order evaluation selected

  5. Enter Suricata-Domain-Category-Rules for the Name, Suricata Domain Category Rules for the Description, and 50 for the Capacity. Choose Next.
  6. Leave the Rule variables section empty. The $HOME_NET variable is inherited from the firewall policy, as configured in the prerequisites.
  7. Leave IP set references empty.
  8. Paste the following rule into the Suricata compatible rule string editor:
    alert tls $HOME_NET any -> $EXTERNAL_NET any (msg:"Artificial Intelligence and Machine Learning Category"; aws_domain_category:Artificial Intelligence and Machine Learning; sid:1000001;)

  9. Choose Next.
    Figure 5: Suricata compatible rule string editor with the domain category alert rule pasted in and the rule variables section left empty

    Figure 5: Suricata compatible rule string editor with the domain category alert rule pasted in and the rule variables section left empty

  10. Under Customer managed key, leave the default setting (Customize encryption settings should remain unchecked).
  11. Under Add tags – optional, leave the default setting of no tags. Choose Next.
  12. Choose Create rule group.
  13. After creating the rule group, return to your firewall policy and add it under Stateful rule groups. We recommend associating new rule groups in a development or test environment first to validate behavior before deploying to production.

The following table explains each component of this rule:

alert Action: generate an alert log entry when the rule matches. Other actions include pass, drop, and reject.
tls Protocol: inspect TLS traffic, matching against the SNI field in the TLS Client Hello.
$HOME_NET any -> $EXTERNAL_NET any Source and destination: match traffic from any internal IP address (HOME_NET) and port to any external IP address (EXTERNAL_NET) and port. The HOME_NET variable defines your internal network ranges, and the EXTERNAL_NET variable is automatically set to the inverse.
msg:”Artificial Intelligence and Machine Learning Category” The message written to the alert log when this rule is triggered.
aws_domain_category:Artificial Intelligence and Machine Learning The AWS-managed domain category to match against. The firewall looks up the destination domain in the category database and matches if the domain belongs to this category.
sid:1000001 A unique signature ID for this rule. Each rule in a rule group must have a unique SID.

Managing exceptions for approved services

You can manage exceptions to keep business-critical websites accessible. For example, say you need to allow access to OpenAI while blocking all other AI and ML traffic. To do this, return to the Suricata-Domain-Category-Rules rule group you created earlier and replace the basic alert rule with the following ruleset. Select the Suricata-Domain-Category-Rules rule group, under the Rules section, choose Edit.

Figure 6: Selecting Suricata-Domain-Category-Rules rule group to edit with new rules

Figure 6: Selecting Suricata-Domain-Category-Rules rule group to edit with new rules

Paste in the following rules and choose Save rule group.

# Allow OpenAI (TLS)
pass tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; dotprefix; content:".openai.com"; nocase; endswith; flow:to_server; alert; msg:"Allow OpenAI over TLS"; sid:1000001;)

# Allow OpenAI (HTTP)
pass http $HOME_NET any -> $EXTERNAL_NET any (http.host; dotprefix; content:".openai.com"; nocase; endswith; flow:to_server; alert; msg:"Allow OpenAI over HTTP"; sid:1000002;)

# Block all other AI/ML category traffic (TLS)
reject tls $HOME_NET any -> $EXTERNAL_NET any (msg:"Block non-approved AI/ML sites over TLS"; aws_domain_category:Artificial Intelligence and Machine Learning; flow:to_server; alert; sid:1000003;)

# Block all other AI/ML category traffic (HTTP)
reject http $HOME_NET any -> $EXTERNAL_NET any (msg:"Block non-approved AI/ML sites over HTTP"; aws_url_category:Artificial Intelligence and Machine Learning; flow:to_server; alert; sid:1000004;)

Figure 7: Suricata compatible rule string editor with the exception-based ruleset containing pass rules for OpenAI and reject rules for the AI/ML category

Figure 7: Suricata compatible rule string editor with the exception-based ruleset containing pass rules for OpenAI and reject rules for the AI/ML category

With strict order evaluation, the firewall evaluates rules in the order you define them. The pass rules for OpenAI appear first, so matching traffic is allowed before the broader category block rules run.

To verify the rules are working as expected, test from a host that routes traffic through your network firewall. These commands suppress the response body and check the exit code of the curl request. If curl completes a TCP connection, it prints CONNECTION ALLOWED. If the firewall resets the connection, curl exits with a non-zero code and prints CONNECTION BLOCKED.

A request to openai.com should succeed because it matches the pass rule:

curl -s -o /dev/null https://openai.com && echo "CONNECTION ALLOWED" || echo "CONNECTION BLOCKED"

Result: CONNECTION ALLOWED

A request to chat.mistral.ai should be rejected because it matches the broader AI/ML category block rule:

curl -s -o /dev/null https://chat.mistral.ai && echo "CONNECTION ALLOWED" || echo "CONNECTION BLOCKED"

Result: CONNECTION BLOCKED

How to monitor category usage

When you add a domain category rule to your firewall policy, Network Firewall performs a category lookup for every connection that matches the rule’s protocol and IP specifications. The rules in this post match on $HOME_NET any -> $EXTERNAL_NET any, which means the firewall looks up the category for all outbound traffic originating from your internal network. This is why it’s important to have the $HOME_NET variable configured correctly at the firewall policy level. With this configuration, a single category rule is enough for category metadata to appear in your firewall logs across all matching connections, not just connections that match the specific category in your rule.

Each log entry includes an aws_category field containing a JSON array of all categories the destination domain belongs to. A single domain can map to multiple categories. For example, a request to chat.mistral.ai produces a log entry with “aws_category": "[\"Social Networking\",\"Artificial Intelligence and Machine Learning\"]” because that domain belongs to both categories.

You can access firewall logs through Amazon CloudWatch, Amazon Simple Storage Service (Amazon S3), and Amazon Data Firehose. These logs show which categorized websites your workloads access, helping you track usage patterns and enforce acceptable use policies.

The following sample log entry shows what a blocked request to chat.mistral.ai looks like using the exception-based rules from the previous section. The alert.signature field contains the rule’s msg value, and the aws_category field lists all categories the destination domain belongs to:

{ 

     "firewall_name": "egress-and-east-west-firewall", 

     "availability_zone": "us-east-1a", 

     "event_timestamp": "1775599146", 

     "event": { 

          "aws_category": "[\"Social Networking\",\"Artificial Intelligence and Machine Learning\"]", 

          "tx_id": 0, 

          "app_proto": "tls", 

          "src_ip": "10.1.1.100", 

          "src_port": 58664, 

          "event_type": "alert", 

          "alert": { 

                    "severity": 3, 

                    "signature_id": 1000003, 

                    "rev": 1, "signature": 

                    "Block non-approved AI/ML sites over TLS", 

                    "action": "blocked", 

                    "category": "" 

          }, 

          "flow_id": 763153567844057, 

          "dest_ip": "172.66.2.203", 

          "proto": "TCP", 

          "verdict": { 

                    "action": "drop", 

                    "reject-target": "to_client", 

                    "reject": [ 

                         "tcp-reset" 

                    ] 

          }, 

          "tls": { 

               "sni": "chat.mistral.ai", 

               "version": "UNDETERMINED" 

          }, 

          "dest_port": 443, 

          "pkt_src": "geneve encapsulation", 

          "timestamp": "2026-04-07T21:59:06.906761+0000", 

          "direction": "to_server" 

     } 

} 

The aws_category field shows the domain belongs to both the “Social Networking” and “Artificial Intelligence and Machine Learning” categories. The verdict field confirms the connection was dropped with a TCP reset sent to the client.

Traffic that matches a pass rule with the alert keyword also generates a log entry with the aws_category field populated. For example, a connection to chat.openai.com that matches the OpenAI exception rule from the earlier section produces a log entry with alert.action set to “allowed” and the same category metadata. This means your queries capture both blocked and allowed traffic.

Querying logs with CloudWatch Logs Insights

If you send your firewall logs to Amazon CloudWatch Logs, you can use CloudWatch Logs Insights to analyze category traffic patterns. A single connection can generate multiple log entries (for example, a reject rule log and a default action log for the same flow), so the following queries deduplicate by flow_id to count each connection only once. Because a single domain can belong to multiple categories, results are grouped by category combination. For example, traffic to a domain categorized as both “Social Networking” and “Artificial Intelligence and Machine Learning” appears as a single combined entry.

To get started, navigate to the CloudWatch console. In the left navigation pane under Logs, select Logs Insights. Under Query scope, leave Log group name selected, then select your AWS Network Firewall alert logs log group. For the time window, we recommend starting with the default of 1 hour to keep the queries light. Enter each of the following queries into the editor and choose Run query to review the results. Note that CloudWatch Logs Insights queries incur charges based on the amount of data scanned. See Amazon CloudWatch pricing for details.

Most accessed categories

This query shows which category combinations your workloads connect to most frequently:

fields @timestamp, event.aws_category, event.flow_id
| filter ispresent(event.aws_category) and event.aws_category != "[]"
| stats latest(event.aws_category) as categories by event.flow_id
| stats count(*) as connections by categories
| sort connections desc
| limit 20

Figure 8: CloudWatch Logs Insights query results showing the most frequently accessed category combinations sorted by connection count

Figure 8: CloudWatch Logs Insights query results showing the most frequently accessed category combinations sorted by connection count

Least accessed categories

This query reverses the sort order to surface category combinations with the fewest connections, helping you identify categories that might not be relevant to your environment or that warrant further investigation:

fields @timestamp, event.aws_category, event.flow_id
| filter ispresent(event.aws_category) and event.aws_category != "[]"
| stats latest(event.aws_category) as categories by event.flow_id
| stats count(*) as connections by categories
| sort connections asc
| limit 20

Figure 9: CloudWatch Logs Insights query results showing the least frequently accessed category combinations sorted by connection count ascending

Figure 9: CloudWatch Logs Insights query results showing the least frequently accessed category combinations sorted by connection count ascending

Most accessed categories, allowed traffic only

The event.verdict.action field indicates the actual outcome of each connection:drop for blocked traffic and alert for allowed traffic. This query shows which category combinations have the most allowed connections:

fields @timestamp, event.aws_category, event.flow_id, event.verdict.action
| filter ispresent(event.aws_category) and event.aws_category != "[]"
| stats latest(event.aws_category) as categories, latest(event.verdict.action) as verdict by event.flow_id
| filter verdict = "alert"
| stats count(*) as connections by categories
| sort connections desc
| limit 20

Figure 10: CloudWatch Logs Insights query results showing the most accessed category combinations filtered to allowed traffic only

Figure 10: CloudWatch Logs Insights query results showing the most accessed category combinations filtered to allowed traffic only

Most accessed categories, blocked traffic only

The same query filtered to blocked connections. Change the verdict filter to drop:

fields @timestamp, event.aws_category, event.flow_id, event.verdict.action
| filter ispresent(event.aws_category) and event.aws_category != "[]"
| stats latest(event.aws_category) as categories, latest(event.verdict.action) as verdict by event.flow_id
| filter verdict = "drop"
| stats count(*) as connections by categories
| sort connections desc
| limit 20

Figure 11: CloudWatch Logs Insights query results showing the most accessed category combinations filtered to blocked traffic only

Figure 11: CloudWatch Logs Insights query results showing the most accessed category combinations filtered to blocked traffic only

Drill down into a specific category

This query uses a like filter to find all traffic where the aws_category field contains a specific category, regardless of what other categories the domain also belongs to. In this example, the query returns all domains your workloads have connected to that map to the Artificial Intelligence and Machine Learning category, broken down by domain and verdict. Replace the category name in the like filter to investigate any category.

fields @timestamp, event.tls.sni, event.aws_category, event.verdict.action, event.flow_id
| filter ispresent(event.aws_category) and event.aws_category like /Artificial Intelligence and Machine Learning/
| stats latest(event.tls.sni) as sni, latest(event.verdict.action) as verdict by event.flow_id
| stats count(*) as connections by sni, verdict
| sort connections desc
| limit 20

Figure 12: CloudWatch Logs Insights query results showing a drill down into the Artificial Intelligence and Machine Learning category with connections broken down by domain and verdict

Figure 12: CloudWatch Logs Insights query results showing a drill down into the Artificial Intelligence and Machine Learning category with connections broken down by domain and verdict

Bandwidth consumption by category

This query shows which category combinations consume the most egress bandwidth. It correlates flow logs (which contain byte counts) with alert logs (which contain category data) using the shared flow_id field. To run this query, select both your alert log group and your flow log group in CloudWatch Logs Insights.

fields @timestamp
| filter ispresent(event.netflow.bytes) or ispresent(event.aws_category)
| stats sum(event.netflow.bytes) as flowBytes, latest(event.aws_category) as categories by event.flow_id
| filter ispresent(categories) and categories != "[]"
| stats sum(flowBytes) as totalBytes by categories
| sort totalBytes desc
| limit 20

Figure 13: CloudWatch Logs Insights query results showing bandwidth consumption by category combination sorted by total bytes descending

Figure 13: CloudWatch Logs Insights query results showing bandwidth consumption by category combination sorted by total bytes descending

These queries help you identify which categories your workloads access by volume, surface blocked and allowed traffic patterns, and pinpoint where the bulk of your egress bandwidth is going.

Conclusion

In this post, you walked through how to set up URL and domain category filtering on AWS Network Firewall, from creating your first category rule using both the console rule builder and Suricata compatible rule strings, to managing exceptions for approved services and monitoring category traffic patterns with CloudWatch Logs Insights. With AWS-managed categories that stay current automatically, you can control access to broad classes of websites without maintaining individual domain lists, and the built-in aws_category log field gives you the visibility to track how your workloads interact with external services.

This feature is available in all AWS commercial regions where AWS Network Firewall is supported.

To learn more, visit the AWS Network Firewall product page and the feature documentation.

Lawton Pittenger

Lawton Pittenger

Lawton is a Worldwide Security Specialist Solutions Architect at AWS, based in New York City. He specializes in helping customers design and implement effective network security controls. At AWS, he works with customers at scale and collaborates closely with service teams to drive continuous improvement in security services based on customer needs and feedback. Outside of work, his interests include skateboarding, snowboarding, and spending time in nature.

Sofia Aluma

Sofia Aluma-Santos

Sofía is a Sr. Security Specialist leading Network Security Go-To-Market and strategy. She helps customers build scalable, secure, resilient networks.

Eric Fortenbery

Eric Fortenbery

Eric is an AWS Solutions Architect based in Atlanta, GA who helps EdTech customers architect secure, scalable platforms.

Mostafa Elkhouly

Mostafa Elkhouly

With over a decade of experience in networking technologies and security, I’m your go-to tech enthusiast! When I’m not jet-setting or tinkering with the latest gadgets, I thrive on empowering customers to harness the full potential of AWS services.

How Zynga scaled multi-warehouse data governance with Amazon Redshift federated permissions

Post Syndicated from Johan Eklund, Matthew Wongkee, Noelia Tardón original https://aws.amazon.com/blogs/big-data/how-zynga-scaled-multi-warehouse-data-governance-with-amazon-redshift-federated-permissions/

Zynga, a global leader in interactive entertainment operates a portfolio of mobile game studios including Socialpoint, the creators of Dragon City and Monster Legends. Zynga’s analytics platform processes telemetry and revenue data across studios using Amazon Redshift as its central data warehouse.

As Zynga expanded its analytics architecture to include individual studios with their own compute environments, the team faced a challenge: how to maintain centralized data governance while granting studios independent query capacity. Their existing approach to permission management introduced lag and required custom infrastructure if scaled to multiple warehouses.

In this post, we walk through how Zynga adopted Amazon Redshift federated permissions and AWS IAM Identity Center to enforce consistent, tiered data access across provisioned and serverless Amazon Redshift environments without building custom synchronization pipelines.

The challenge

Zynga needed to onboard Socialpoint’s current Amazon Redshift workloads and make Zynga’s central data available to them. Zynga’s existing production cluster would house the Socialpoint raw data, but the compute would come from another warehouse set up as a consumer. At the same time, Zynga’s data access control policies would need to be enforced across all warehouses. Zynga uses a tiered access control policy which would need to be synced across all consumers with no permission lag or manual grant synchronization.

During the migration, Socialpoint’s specific extract, transform, and load (ETL) processes would be included in Zynga’s central ETLs and their data ingestion pipeline would be replaced by Zynga’s latest generation of data ingestion infrastructure. Because the migration process happens in stages, Amazon Redshift sizing would also gradually need to increase.

The team evaluated two alternatives before arriving at a solution:

  • AWS Lake Formation couldn’t manage local and cross-cluster permissions using the same interface, and required AWS Access and Identity Management (IAM) or IAM Identity Center authentication for all users including service accounts.
  • Manual grants on consumer clusters introduced a delay between when permissions were updated on the producer and when they took effect on the consumer. This approach would also require an external job that synced permissions and would be unlikely to scale well beyond 2–3 consumers.

Solution overview

Zynga implemented a solution using three AWS services working together:

  1. Amazon Redshift federated permissions enabled cross-cluster queries without explicit data shares. Permissions granted on the producer cluster propagate immediately to consumer workgroups through AWS Glue Data Catalog registration.
  2. AWS IAM Identity Center provides unified authentication through federation with Okta. When users sign in, their Okta group memberships are provisioned through a System for Cross-domain Identity Management (SCIM) and automatically map to Amazon Redshift roles, removing the need for external synchronization jobs.
  3. Amazon Redshift Serverless provides the compute layer for Socialpoint, scaling to zero when idle and avoiding the need to pre-size a provisioned cluster during the migration period.

The architecture uses a dual-grant approach where every permission is granted to both an IAM Identity Center group (for users) and a federated IAM role (for service accounts). This gives both authentication paths the same access.

How it works

Authentication with IAM Identity Center

Zynga’s existing Okta directory syncs to IAM Identity Center, which is connected to the Amazon Redshift Serverless workgroup. When a user authenticates, Amazon Redshift automatically creates a user mapped to their email address and assigns them to roles based on their Okta group membership.

For example, an analyst in the Gamma Tier group signs in and is automatically assigned the AWSIDC:role.sso.gamma role in Amazon Redshift. No manual role assignment or synchronization job is required.

Service accounts, used for programmatic access, authenticate differently. Either using their IAM role and calling the get-credentials API, or by using the new federated permissions feature. Each service account assumes a federated IAM role, which creates a corresponding federated user in Amazon Redshift (for example, IAMR:role_iam_gamma).

The identity layer shows two authentication paths. Interactive users authenticate through Okta, which syncs group memberships to AWS IAM Identity Center. On login, IAM Identity Center automatically maps users to tiered Amazon Redshift roles (AWSIDC:role.sso.gamma, beta, alpha). Service accounts take a separate path, assuming a cluster IAM role that creates a corresponding federated user (IAMR:role_iam_gamma) in Amazon Redshift.

Figure 1 – Identity layer

The dual-grant approach

To ensure that both users and service accounts can access the same data, every read permission is granted to both the IAM Identity Center group and the federated IAM role in a single statement:

GRANT SELECT ON schema.table TO
	'IAMR:role_iam_gamma',
	ROLE 'AWSIDC:role.sso.gamma';

Transitioning the producer cluster

The shared provisioned cluster already had active users with local grants. To avoid disruption, Zynga implemented a tri-grant approach on the producer during the transition period. Existing stored procedures were modified to grant permissions to three targets: the legacy local role, the IAM Identity Center group, and the federated IAM role.

GRANT SELECT ON schema.table TO
	ROLE role_rs_gamma,
	ROLE 'AWSIDC:role.sso.gamma','IAMR:role_iam_gamma';

This approach maintains backward compatibility for existing users on the producer while enabling immediate access from the new serverless workgroup. The long-term plan includes migration of all producer users to IAM Identity Center and retire the legacy local grants.

Stored procedures for consistent governance

Rather than requiring users to construct dual-grant statements manually, Zynga created stored procedures that encapsulate the grant logic:

  • grant_read accepts a table name and access tier, then issues the appropriate dual-grant for both the IAM Identity Center group and the federated IAM role.
  • grant_write grants data definition language (DDL) and data manipulation language (DML) permissions to the appropriate team-based role.

This provides a consistent interface for permission management regardless of which cluster or workgroup that the user is on.

Architecture diagram in the AWS Cloud showing a federated Amazon Redshift setup. On the left, a Zynga provisioned Amazon Redshift cluster (producer) holds Zynga central data — game telemetry and revenue data — and stored procedures grant_read(table, tier) and grant_write(table, team). On the right, a Socialpoint Amazon Redshift Serverless workgroup (consumer) holds Socialpoint ETL data and the same two stored procedures. Bidirectional arrows between the two environments label the top connection as "Federated queries (cross-cluster reads)" between the data stores, and the bottom connection as "Federated permissions" between the stored procedures.

Figure 2 – Data and compute layer

Results

The migration delivered measurable improvements:

  • Immediate permission propagation – Grants on the producer cluster took effect on the consumer workgroup instantly, replacing a process that previously required manual intervention and introduced lag.
  • Zero additional infrastructure cost – Federated permissions, federated queries, and IAM Identity Center added no incremental costs to the architecture.
  • Removed custom synchronization – The team removed the need for Lake Formation configurations, external AWS Lambda functions, and Airflow workflows for permission management.
  • Scalable pattern – The same architecture can be extended to additional studio workgroups without duplicating permission management logic.

Lessons learned

Zynga adopted federated permissions shortly after the feature launched in US West Oregon (us-west-2) in January 2026. The team shared several observations from their early adoption:

  • Run a proof of concept first. The team validated the full permission model in a test environment before deploying to production, including testing that existing data shares to other clusters were not disrupted.
  • Plan for the dual-grant requirement. Because IAM Identity Center users and federated IAM roles are distinct identity types, every read permission requires two grants. Encapsulating this in stored procedures prevents errors and reduces cognitive overhead.
  • Start with serverless for new workloads. With Amazon Redshift Serverless, the team can avoid sizing decisions during the migration period. If usage patterns later justify it, they can migrate to a provisioned cluster from a serverless snapshot with minimal downtime.
  • Engage with AWS. As an early adopter of a new feature, Zynga maintained regular contact with the Amazon Redshift team through their AWS Technical Account Manager to report issues and request enhancements.

Conclusion

Zynga’s adoption of Amazon Redshift federated permissions demonstrates how organizations with multi-cluster Amazon Redshift architectures can enforce centralized data governance without building custom synchronization infrastructure. By combining federated permissions with IAM Identity Center and Amazon Redshift Serverless, the team established a pattern that scales to additional studios while maintaining consistent access controls and reducing operational overhead.

To learn more about the services used in this post, see the following resources:


About the authors

Johan Eklund

Johan Eklund is a Principal Software Engineer on the Data Engineering team at Zynga, focused on building and operating large-scale data platforms on AWS. Day-to-day, his work spans Redshift architecture, ETL orchestration, platform operations, and security and access control. Outside of work, Johan enjoys exploring art and music through creative coding and DIY hardware synthesizers.

Matthew Wongkee

Matthew Wongkee is a Principal Software Engineer at Zynga, where he builds scalable and high-performance data platforms and infrastructure, including Amazon Redshift. He focuses on enabling teams to leverage their data through self-service capabilities, including data discovery, query optimization, curated datasets, and data quality. In his spare time, he practices short track speed skating.

Noelia Tardón

Noelia Tardón is a Principal Software Engineer at Zynga. Based in Barcelona, she works on large-scale cloud data platforms with a focus on performance optimization and cost efficiency. She designs and builds data lake and data warehouse solutions while ensuring compliance with data protection regulations. Outside of work, she enjoys spending time with her family and running.

Sandeep Adwankar

Sandeep is a Senior Technical Product Manager at AWS. Based in the California Bay Area, he works with customers around the globe to translate business and technical requirements into products that enable customers to improve how they manage, secure, and access data.

Satesh Sonti

Satesh is a Principal Analytics Specialist Solutions Architect based out of Atlanta, specializing in building enterprise data platforms, data warehousing, and analytics solutions. He has over 19 years of experience in building data assets and leading complex data platform programs for banking and insurance clients across the globe.

Photo of AUthor - Steve Phillips

Steve Phillips

Steve is a Principal Technical Account Manager and Analytics specialist at AWS in the North America region. Steve currently focuses on data warehouse architectural design, data lakes, data ingestion pipelines, and cloud distributed architectures.

Automate data discovery and centralized management with AWS Glue Data Catalog

Post Syndicated from Ramakrishna Natarajan original https://aws.amazon.com/blogs/big-data/automate-data-discovery-and-centralized-management-with-aws-glue-data-catalog/

Managing sensitive data across sprawling data environments is hard. In this post, we show you how to tackle data discovery, classification, and governance across your databases, data warehouses, and object storage to regain visibility and control over your data landscape. As you build new features, products, and services, your data naturally spreads across multiple systems to meet immediate application and business needs. Different teams spin up their own data stores, and before long, you’re dealing with a complex web of repositories—often with limited visibility into what exists where. This data sprawl becomes most challenging when you must understand and protect your sensitive data. Security teams often struggle to maintain accurate inventories of data categorization and classification. Stakeholders demand comprehensive insights into data classification and processing activities, usually on tight deadlines, and keeping up-to-date data inventories becomes increasingly daunting as your data grows. Without automation, you’re left with manual processes that stretch over weeks, leave room for human error, and create unnecessary business risk.

The need for automation

In a typical manual scenario, creating a new database triggers a chain of time-consuming events. The governance team reviews the new data source, documents its contents, and scans for sensitive data. The security team assesses its configuration and access controls. Days or weeks pass before you fully understand this new asset’s sensitivity.

With automation, creating a new database triggers immediate action. The system detects the new source, catalogs its structure, identifies sensitive data, and updates a central inventory within minutes, supporting proper governance from the moment you create it. Here’s how it works on AWS: When you create an Amazon Simple Storage Service (Amazon S3) bucket for customer orders, you add tags such as Business Function, Data Owner, and Purpose. After the bucket is in use, the system detects it, creates catalog entries, analyzes data patterns, identifies sensitive information, and updates governance records without additional input from you. This gives your organization real-time visibility. Security teams instantly see which repositories contain sensitive information. Governance teams generate up-to-date inventory reports on demand, and data teams immediately understand sensitivity levels, helping them use data responsibly.

Solution overview

The solution uses key AWS services across three layers that work together for comprehensive data visibility and categorization.

Detection Layer: Continuously monitors your AWS environment for new resource creation. When you provision an Amazon S3 bucket, Amazon Relational Database Service (Amazon RDS) database, or Amazon DynamoDB table, Amazon EventBridge rules capture this activity and initiates the governance workflow, so no data source goes unnoticed.

Architecture: EventBridge triggers Lambda and SQS to create Glue crawlers and ETL jobs for new S3 data sources
Figure 1 Automated data source discovery (S3 example) workflow using EventBridge Rules and Lambda functions

Processing Layer: After a new source is detected, AWS Glue crawlers analyze its schema while specialized jobs scan for sensitive data patterns. The system also extracts metadata from resource tags, enriching your understanding of each repository’s purpose and ownership.

Architecture: Glue PII Detection jobs scan S3, DynamoDB, and Aurora; Lambda updates Glue Data Catalog
Figure 2 PII detection and processing workflow using AWS Glue jobs and DynamoDB staging

Management Layer: Maintains a central source of truth about your data assets. AWS Glue Data Catalog provides a unified view across your organization, tracking schema changes and sensitivity levels. This layer also manages the processing workflow state and generates insights for stakeholders.

Architecture: Lambda extracts S3 bucket metadata via EventBridge, stores in DynamoDB, updates Glue Data Catalog
Figure 3 Tag-based metadata capture and Data Catalog update workflow

Setting up the solution

This solution uses AWS Cloud Development Kit (AWS CDK) for deployment, organized into four stacks that build upon each other.PrerequisitesBefore deployment, verify that you have:

  • Access to an AWS account with permissions to create resources in Amazon S3, AWS Lambda, Amazon DynamoDB, AWS Glue, and Amazon EventBridge
  • Node.js (version 18 or later) and npm installed
  • Access to a terminal to run AWS CDK CLI commands
  • Basic familiarity with AWS Console navigation

Step 1: Infrastructure deployment

Deploy four stacks using AWS CDK. Each establishes components for data discovery, cataloging, and PII detection.

  1. BaseInfraStack: Deploys core infrastructure—Amazon Virtual Private Cloud (Amazon VPC), DynamoDB tables for state management, EventBridge rules for monitoring, and Lambda functions for orchestration.
  2. GlueAssetsStack: Sets up S3 buckets for AWS Glue ETL scripts and deploys PySpark code for PII detection.
  3. GlueJobCreationStack: Creates Data Catalog databases and deploys Lambda functions that automate the creation of AWS Glue crawlers and PII detection jobs for newly discovered data sources.
  4. ReportingStack: Deploys Lambda functions that process PII detection results and tag metadata, updating the Data Catalog accordingly.

To deploy these stacks, you will use the AWS CDK CLI, running the following commands:

# Clone and prepare repository
git clone https://github.com/aws-samples/automated-datastore-discovery-with-aws-glue.git
cd automated-datastore-discovery-with-aws-glue
npm install
npx cdk bootstrap

# Deploy infrastructure stacks sequentially
npx cdk deploy BaseInfraStack
npx cdk deploy GlueAssetsStack
npx cdk deploy GlueJobCreationStack
npx cdk deploy ReportingStack
CloudFormation Stacks console showing four stacks with CREATE_COMPLETE status including BaseInfraStack

Figure 4 CloudFormation console showing successful stack deployment

Step 2: Verify initial setup

In the AWS Management Console, open DynamoDB and find the glueJobTracker table. This table is a critical component of the framework:

  • Purpose: Central state management – tracks processing states and configurations for discovered data sources.
  • Current state: The table should be empty because no discovery processes have been triggered yet.
  • Structure: Tracks states such as Data Catalog entry creation and PII detection job setup for each data source.

By verifying this table, you confirm that the infrastructure is ready to begin tracking new data sources.

DynamoDB glueJobTracker table scan returning zero items, showing empty table before pipeline execution

Figure 5 Empty DynamoDB glueJobTracker table before execution

Solution in action

This solution runs automatically in production through EventBridge triggers and scheduled AWS Glue crawlers. The following walkthrough executes each step manually so you can observe the workflow.You follow the journey of a newly created S3 bucket containing sensitive data, seeing how the solution discovers, catalog, and processes it through each stage.

Step 3: Create a new S3 bucket

  1. Open the Amazon S3 console.
  2. Choose Create bucket.
  3. Enter a unique name for your bucket (for example, demo-customer-data-20250819).
  4. In the Tags section, add the following tags:
    1. Key: gdpr-scan, Value: true
    2. Key: Business Function, Value: Sales – US
    3. Key: Data Classification, Value: Confidential
  5. Keep other settings as default and choose Create bucket.
S3 bucket properties with versioning disabled and data classification tags

Figure 6 S3 console showing new bucket creation with tags

Step 4: Upload sample data

  1. In the S3 console, open your newly created bucket.
  2. Choose Upload.
  3. Create a new file named customer_orders.csv with the below content.
  4. Upload this file to a folder named orders/ in your bucket.
order_id,customer_id,email,ssn
ORD001,CUST1001,[email protected],***-**-****
ORD002,CUST1002,[email protected],***-**-****

S3 upload succeeded showing customer_orders.csv uploaded

Figure 7: S3 console showing uploaded CSV file in the orders folder

Step 5: Verify automated detection

  1. Open the DynamoDB console.
  2. Navigate to the glueJobTracker table.
  3. Choose the Items tab.
  4. You should see a new item with an s3_location matching your bucket name.
40918.pngDynamoDB glueJobTracker table scan returning 1 item with S3 data source entry

Figure 8 DynamoDB console showing detected bucket entry in glueJobTracker table

Step 6: Initiate catalog creation

  1. Open the AWS Lambda console.
  2. Find the function with a name containing s3GlueCatalogCreator.
  3. Choose the function name to open its details.
  4. Choose the Test tab.
  5. Create a new test event with an empty JSON object {}.
  6. Choose Test to invoke the function.
  7. Check the execution result for a successful response.
Lambda function execution succeeded with logs showing Glue table, crawler, and DynamoDB item creation

Figure 9 Lambda console showing successful function execution

Step 7: Run the AWS Glue crawler

  1. Navigate to the AWS Glue console.
  2. In the left sidebar, choose Crawlers.
  3. Find the crawler with a name related to your S3 bucket.
  4. Select the crawler and choose Run crawler.
  5. Wait for the crawler to complete (typically 3–5 minutes).
AWS Glue crawler running

Figure 10 Glue console showing crawler in “Running” state

Step 8: Verify schema discovery

  1. In the AWS Glue console, go to Databases in the left sidebar.
  2. Choose the s3_source_db database.
  3. You should see a new table corresponding to your uploaded data.
  4. Choose the table name to view its schema.
Glue Data Catalog table Version 3 showing 4-column CSV schema with no sensitive data annotations yet

Figure 11 Glue console showing detected table schema

Step 9: Execute PII detection

  1. Return to the Lambda console.
  2. Find and open the function with a name containing s3GlueCreator.
  3. Use the Test tab to invoke this function with an empty JSON object {}.
  4. After successful execution, go to the AWS Glue console.
  5. Navigate to Jobs in the left sidebar.
  6. Find the newly created PII detection job (it should contain your bucket name).
  7. Select the job and choose Run job.
  8. Monitor the job execution in the Glue console.
AWS Glue PII detection job running with 10 DPUs, G.1X worker type, Glue version 4.0, showing run details

Figure 12 Glue console showing PII detection job in “Running” state

Step 10: Review PII detection results

  1. Open the DynamoDB console.
  2. Navigate to the piiDetectionOutputTable.
  3. In the Items tab, you should see new entries related to your data.
  4. These entries will show detected PII types and confidence scores.
DynamoDB piiDetectionOutputTable scan showing 2 PII detection results identifying USA_SSN and EMAIL types

Figure 13 DynamoDB console showing PII detection results in piiDetectionOutputTable

Step 11: Verify Data Catalog updates

  1. Open the AWS Lambda console.
  2. Find the function with a name containing ReportingStack-PIIReportS3.
  3. Choose the function name to open its details.
  4. Choose the Test tab.
  5. Create a new test event with an empty JSON object {}.
  6. Choose Test to invoke the function.
  7. Check the execution result for a successful response.
  8. Return to the AWS Glue console.
  9. Go to Databases > s3_source_db > Your table.
  10. Review the schema. PII columns should now have comments indicating their classification.
Glue Data Catalog table Version 7 with schema showing sensitive data element comments for EMAIL and USA_SSN

Figure 14 Glue console showing updated table schema with PII classifications

Note: While we focus on S3 data sources in this walkthrough, the framework extends to other data stores, offering a unified approach for PII detection and compliance management, so organizations can automatically discover, catalog, and monitor sensitive data elements across your entire data ecosystem. For more information, see aws-samples/automated-datastore-discovery-with-aws-glue.

Best practices and operational excellence

As you implement this solution, consider these key practices for effective results:

  • Design your tagging strategy to capture essential business context about each data source. Implement automated tag enforcement through AWS Organizations for consistency across teams.
  • Monitor automated workflows regularly and configure retention policies for processed data to manage costs.
  • For enhanced security, configure VPC endpoints for services such as Amazon S3, DynamoDB, and other data sources. This keeps traffic within the AWS network, which is especially important when processing sensitive data. Verify that server-side encryption (SSE) is enabled on your data stores. This solution uses AWS Key Management Service (AWS KMS) keys for DynamoDB tables and SSE-S3 for S3 buckets by default, aligning with data-at-rest encryption best practices.
  • For teams with multiple AWS accounts, implement cross-account discovery and cataloging to maintain a comprehensive view of your data landscape.
Multi-account EventBridge buses aggregate events to central account with Lambda and Glue Data Catalog

Figure 15 Centralized Storage of Glue PII Detection Results in AWS Data Catalog

Clean up

To avoid ongoing charges and remove the resources created by this solution, follow these steps:

  1. Empty and delete the S3 buckets created for sample data and AWS Glue assets.
  2. Delete the AWS CloudFormation stacks in reverse order of creation:
    1. ReportingStack
    2. GlueJobCreationStack
    3. GlueAssetsStack
    4. BaseInfraStack
  3. Manually delete any remaining resources:
    1. DynamoDB tables (glueJobTracker, piiDetectionOutput, tagCaptureTable)
    2. AWS Glue databases and crawlers
    3. Lambda functions
    4. EventBridge rules
  4. Review your AWS account to ensure that all related resources have been removed.

Remember, deleting these resources will remove all data and configurations associated with this solution. Make sure that you have saved any important information before proceeding with the clean-up.

Conclusion

In this post, you learned how to build an automated data governance framework using AWS Glue Data Catalog. You set up detection, processing, and management layers that automatically discover, catalog, and classify your data sources.This approach improves how you manage sensitive data assets. Teams spend less time on manual discovery and categorization, freeing them to derive value from data. The system gives you current insights into your data landscape and automatically identifies sensitive data, creating a trusted source of truth that helps teams work efficiently while maintaining controls.You can extend this framework with custom sensitivity patterns for your industry. Its modular design supports continuous improvement and integrates with existing workflows. This turns data governance from a manual burden into an efficient process that scales with your organization.


About the authors

Ramakrishna Natarajan

Ramakrishna Natarajan

Ramakrishna is a Senior Partner Solutions Architect at Amazon Web Services. He is based out of London and helps AWS Partners find optimal solutions on AWS for their customers. He specialises in Telecommunications OSS/BSS and has a keen interest in evolving domains such as Data Analytics, AI/ML, Security and Modernisation. He enjoys playing squash, going on long hikes and learning new languages.

Well-architected best practices for software supply chain security

Post Syndicated from Trevor Schiavone original https://aws.amazon.com/blogs/security/well-architected-best-practices-for-software-supply-chain-security/

There have been multiple notable supply chain attacks using the npm Registry since September: Shai-Hulud, Chalk/Debug, one abusing tea.xyz tokens, and recently axios. Thanks to community efforts involving the Amazon Inspector team, the Open Source Security Foundation, and others, the affected packages were quickly flagged, which reduced the impact of these incidents.

Supply chain attacks like Shai-Hulud exploit vulnerabilities on two fronts: compromised maintainer accounts that publish malicious packages, and consumer environments that download and execute those packages. The Shai-Hulud attack, shown in Figure 1, succeeded because maintainer credentials were compromised through phishing, enabling threat actors to publish malicious versions of popular packages. Incidents like these highlight the need for strong security practices within the software supply chain, and effective defense requires addressing both sides. Package maintainers need protections that prevent account compromise and limit sprawl when credentials are stolen. Package consumers need layered defenses that detect malicious packages, prevent their deployment, and limit damage when compromise occurs.

In this post, we explore best practices for package consumers. These practices are aligned with the AWS Well-Architected Framework – Security Pillar and you can use them to reduce exposure to similar threats and limit their impact if they occur.

Figure 1: Architecture diagram showing Shai-Hulud attack flow

Figure 1: Architecture diagram showing Shai-Hulud attack flow

Use temporary credentials and grant least privilege

When Shai-Hulud executed in developer environments and continuous integration and delivery (CI/CD) pipelines, it scanned for secrets such as npm tokens, GitHub tokens, and AWS Identity and Access Management (IAM) access keys. Long-term credentials exposed in this way enabled threat actors to propagate the malware further and access cloud resources. Recent incidents have shown organizations discovering multiple leaked IAM credential pairs, with concerns about additional exposed credentials and potential compromise of CI/CD pipelines.

Removing long-lived credentials from your developer environments and CI/CD pipelines reduces the scope of exposure in the event a system is compromised. For developers working locally, the new AWS CLI login command (aws login) simplifies the process of acquiring short-lived CLI credentials and removes the need to store long-lived credentials in configuration files. AWS IAM Identity Center also provides a straightforward way to acquire temporary credentials that expire automatically. For CI/CD pipelines, OpenID Connect (OIDC) federation with GitHub Actions, GitLab CI, or other platforms provide temporary credentials for each job without storing long-lived tokens. IAM can also federate AWS Identities to external services, allowing your AWS workloads to securely access external services without using long-term credentials. Temporary credentials expire automatically, limiting the window of exposure if a pipeline is compromised.

If you’re interacting with a third-party service that doesn’t support temporary credentials, consider storing the credentials to centralized storage using AWS Secrets Manager or AWS Systems Manager Parameter Store. Limit access to these secrets, require the use of temporary credentials, and apply automatic rotation and audit logging to reduce the risk of exposure of these credentials.

To reduce risk from credential exposure:

In the event of a security incident where credentials might be exposed, immediately rotate all long-term credentials to limit the scope of impact. Use Amazon GuardDuty and AWS CloudTrail to detect abnormal IAM activity and identify which credentials might have been compromised.

Implement defense in depth

Even with temporary credentials and least privilege, a single compromised account can enable threat actors to publish malicious packages or access sensitive resources. Defense in depth creates multiple layers of protection that work together to prevent sprawl after initial compromise. While adding approval workflows to every operation would dramatically decrease deployment speed, implementing them strategically for sensitive workloads provides balanced security.

The key principle is to ensure that if one credential or account is compromised, additional controls prevent that compromise from spreading across your organization. This includes multi-factor authentication (MFA) for access combined with different IAM roles for sensitive workloads. For single-developer open source projects, MFA becomes even more critical because there’s no separation of duties through multiple maintainers.

For package maintainers working in team environments, requiring multiple approvers to release packages to production creates separation of duties. However, developers can still trigger merge requests that initiate deployment pipelines, so multi-party approval should be implemented within the pipeline itself for sensitive deployments. This ensures that even if a developer’s credentials are compromised and they trigger a deployment, the pipeline requires additional approval before releasing to production.

For package consumers, multi-approval workflows in deployment pipelines help ensure that if a malicious package passes initial scanning, human review can catch suspicious changes before production deployment. Artifact signing provides a complementary cryptographic layer that works alongside these process controls.

The Shai-Hulud attack succeeded because compromised maintainer credentials allowed threat actors to publish malicious packages directly to the public npm registry. For package consumers, the defense is ensuring that packages pulled from public registries cannot reach production without verification. Artifact signing is one concrete implementation of this layered approach. By cryptographically binding a package or container image to the identity that produced it, signing creates a verification layer that is independent of the credential used to trigger the build—meaning a compromised developer credential alone isn’t sufficient to introduce an unverified artifact into your deployment pipeline.

Artifact signing as part of defense in depth

AWS Signer provides cryptographic signing for packages, creating an additional verification layer within your defense in depth strategy. The signing authorization model separates concerns: developer credentials shouldn’t have signing permissions. Only CI/CD pipeline roles should have signing permissions through the signer:StartSigningJob API. Signer uses FIPS 140-3 Level 3 validated hardware security modules (HSMs) to store signing keys, providing strong cryptographic protection.

The container image signing workflow, shown in Figure 2, demonstrates how signing integrates seamlessly into existing processes:

Figure 2: Diagram showing Signer signing workflow

Figure 2: Diagram showing Signer signing workflow

The benefits of Signer compared to building custom signing infrastructure include:

  • Fully managed: No need to build custom signing infrastructure or manage certificate lifecycle
  • Automated: Amazon ECR managed signing happens automatically on image push without manual steps
  • Centralized governance: Single signing profile can be used across multiple accounts and pipelines
  • Native integration: Built-in integration with Notation and Kyverno for signature verification in Amazon EKS
  • FIPS 140-3 Level 3 compliance: Meets stringent regulatory requirements for cryptographic operations

Audit logging for all signing operations enables detection of unusual patterns such as signing from new IP addresses, unusual times, or rapid succession of signing jobs. For every credential in your system, consider who can access what, how they prove their identity, and what second layer prevents sprawl if that credential is compromised. Artifact signing, combined with centralized storage and Software Bills of Materials (SBOMs), provides layered protection against tampering and malicious packages.

Centralize dependency management

By centralizing package and dependency management, you can validate and approve dependencies before they’re used in applications and quickly audit dependencies in the event of a supply chain security incident. Recent incidents have shown organizations discovering compromised npm packages in their internal artifact repositories, requiring rapid assessment of which applications might be affected.

On AWS, you can use AWS CodeArtifact to host and manage your organization’s software packages. You can use the package group configuration to define an approved list of upstream sources and block access to all others—a direct control against typosquatting attacks, where malicious packages are published under names that closely resemble legitimate ones. Rather than relying on developers to identify suspicious package names at install time, package group configuration enforces the boundary at the repository level. Centralization also helps you pin versions of dependencies to prevent automatic updates from pulling in malicious versions and quickly remove compromised dependencies across your software portfolio when an incident occurs.

For container images, Amazon ECR provides centralized image storage with AWS Key Management Service (AWS KMS) encryption and lifecycle policies. Combine Amazon ECR with Amazon Inspector scanning to continuously validate image integrity.

For additional guidance see SEC11-BP05: Centralize services for packages and dependencies.

npm provenance attestation

For npm packages specifically, provenance attestations provide a complementary control on the consumer side. Available since npm 9.5, npm provenance links a published package to the specific source repository and CI/CD workflow that produced it, using Sigstore as the underlying signing infrastructure. When a package is installed, the npm CLI can verify that the published artifact matches the attested build provenance—providing confidence that the package wasn’t tampered with between build and publication. For organizations consuming open source npm packages, checking for provenance attestations before adding a new dependency is a low-friction signal of supply chain integrity. Package maintainers publishing to npm can enable provenance by running npm publish with the –provenance flag from a supported CI/CD environment such as GitHub Actions.

For additional guidance see SEC11-BP06: Deploy software programatically and, from the DevOps Lens of the AWS Well-Architected Framework, DL.CS.2: Sign code artifacts after each build

Scan dependencies throughout the software development lifecycle

AWS provides services to help you scan dependencies continuously, from development through deployment:

  • In development: Kiro can perform software composition analysis during code reviews to identify vulnerable third-party code.
  • In code repositories and pipelines: Amazon Inspector scans first-party code, third-party dependencies, and Infrastructure as Code for vulnerabilities.
  • For container images: Amazon Inspector provides continuous vulnerability scanning of Amazon ECR images. Amazon Inspector can also be integrated directly into CI/CD pipelines to scan images before they are pushed to Amazon ECR or deployed, helping you block compromised dependencies earlier in the release cycle.

Traditional vulnerability scanners focus on known CVEs—publicly disclosed vulnerabilities with assigned identifiers. Supply chain attacks like Shai-Hulud involve malicious packages that function as zero-days: they’re intentionally crafted by threat actors and actively exploited before a CVE is assigned. Traditional vulnerability scanners that rely on CVE databases won’t detect these packages until they’ve been formally identified and catalogued, which can take days or weeks.

Detecting these requires behavioral analysis at scale and community collaboration, not just static signature matching. The operational scale of AWS—with threat intelligence from sources like MadPot and incident response data across millions of customers—enables detection of suspicious package behavior across multiple environments simultaneously. When a newly published package exhibits credential-harvesting behavior in multiple customer accounts within hours of publication, that cross-account signal enables rapid identification. These findings are contributed to community-maintained databases like the OpenSSF Malicious Packages Repository (github.com/ossf/malicious-packages), which assigns a formal identifier (MAL-ID) and shares it across the security community. For the tea.xyz token farming campaign, the average time from submission to formal identification was approximately 30 minutes. AWS services like Amazon Inspector participate in this community loop, contributing findings and ingesting newly assigned MAL-IDs to surface threats in your environment.

A related threat model worth understanding is the sleeper package: a package that appears benign at publication and activates malicious behavior only after a delay or trigger condition. Static analysis alone is insufficient to catch these packages because the malicious payload isn’t present or active at install time. Amazon Inspector behavioral analysis is specifically designed to detect this class of threat, complementing static vulnerability scanning.

Software Bills of Materials (SBOMs) in SPDX or CycloneDX format enable you to quickly assess exposure during incidents. When responding to supply chain incidents, use SBOMs to identify which applications contain compromised packages, prioritize remediation, and assess blast radius. In the Shai-Hulud incident, the compromised packages (MAL-2025-46974 and CVE-2025-59144) were identified early, providing actionable findings that customers could remediate quickly. Organizations that had scanning enabled but experienced alert fatigue may have missed critical alerts, highlighting the importance of proper alert routing and prioritization.

Figure 3: Screenshot of Amazon Inspector console showing malicious package findings

Figure 3: Screenshot of Amazon Inspector console showing malicious package findings

For additional guidance see SEC11-BP02: Automate testing throughout the development and release lifecycle.

Configure logging and monitoring

Visibility into activity is essential to detect anomalous behavior early. Configure logging and centralize those logs for analysis:

  • Enable application and service logging
  • Centralize and monitor logs across accounts
  • Use GuardDuty to continuously monitor for malicious activity and anomalous API calls
  • Aggregate findings with AWS Security Hub and enforce configuration best practices with AWS Config

CloudTrail logging provides audit trails for credential access and API activity. When responding to supply chain incidents, review CloudTrail logs for specific events that indicate credential compromise or malicious activity: sts:AssumeRole calls from unexpected IP addresses or regions, secretsmanager:GetSecretValue or ssm:GetParameter calls from unfamiliar sources, ecr:PutImage from developer workstations bypassing CI/CD pipelines, lambda:UpdateFunctionCode outside normal deployment windows, iam:CreateAccessKey followed by immediate API activity, and codecommit:GitPush or codebuild:StartBuild from unusual IP addresses. When combined with Amazon EventBridge rules, you can trigger automated responses when Amazon Inspector detects malicious packages or when these unusual credential access patterns occur.

Organizations affected by recent supply chain attacks have used CloudTrail analysis to determine the scope of credential exposure and identify which resources might have been accessed by compromised credentials. This forensic capability is essential for understanding blast radius and ensuring complete remediation.

For additional guidance see the best practices in SEC04: Detection.

These components of a defense in depth strategy work together to prevent sprawl after initial compromise and help you to detect and respond to the event. Figure 4. shows how these fit together.

Figure 4: Architecture diagram showing defense architecture

Figure 4: Architecture diagram showing defense architecture

Additional best practices

The Security pillar of the Well-Architected Framework also provides organizational best practices that are applicable to improving security processes across all dimensions of your organization. Relevant best practices include SEC11-BP01: Train for application security; SEC11-BP08: Build a program that embeds security ownership in workload teams; and SEC10: Incident Response.

Conclusion

Recent incidents including Shai-Hulud, Chalk/Debug, and tea.xyz reflect ongoing efforts by threat actors to target the the package registries, CI/CD pipelines, and developer credentials for increased attack surface and propagation. A single compromised maintainer account or malicious package can propagate across thousands of consumer environments simultaneously. The controls described in this post are designed with that threat model in mind. Temporary credentials limit the value of stolen tokens, centralized dependency management and upstream blocking reduce the attack surface at the registry level, artifact signing ensures that even if a build pipeline is compromised, unsigned artifacts cannot reach production, and dependency scanning throughout your software lifecycle helps you identify compromised packages early, before they can impact you. Each layer narrows the window of opportunity for a threat actor.

Learn more

See the following blogs and workshops to dive deeper on this topic:

If you have feedback about this post, submit comments in the Comments section below.

Trevor Schiavone

Trevor Schiavone

Trevor is a Senior Solutions Architect with a background in application development and architecture. He brings a builder’s perspective to helping customers design secure, scalable, and innovative solutions on AWS. He is also an active part of the AWS security community with a focus on application security and identity.

Desiree Brunner

Desiree Brunner

Desiree is a Security Specialist Solutions Architect working with regulated customers as part of the AWS EMEA Security & Compliance team. She builds on her background in DevOps and platform engineering to support her customers in designing secure, compliant cloud environments. Passionate about mental health and knowledge sharing, she regularly speaks at AWS events and supports teams on their cloud security journey.

Governing infrastructure as code using pattern-based policy as code

Post Syndicated from Guptaji Teegela original https://aws.amazon.com/blogs/security/governing-infrastructure-as-code-using-pattern-based-policy-as-code/

Organizations often struggle to enforce security and compliance requirements consistently across their cloud infrastructure. In one environment, a workload might be deployed in an AWS Region that was never approved for that class of data. In another, a security group might allow broader access than intended. Required tags might be missing. Encryption might be assumed but not configured. These gaps create risk, increase review effort, and make audits harder than they need to be.

Many organizations already have standards that describe what good infrastructure looks like. The more difficult problem is making sure those expectations are checked the same way across repositories, environments, and teams before infrastructure is deployed. Manual review helps, but it doesn’t scale when delivery moves faster and more teams provision infrastructure directly.

Policy as code helps address this problem. It turns control intent into preventive checks that run in delivery workflow.

A pattern-based policy model makes those checks more straightforward to review, maintain, and explain. Teams can organize policy checks around recurring control patterns such as required metadata, allowed configuration, exposure restriction, protection enforcement, and privilege constraint, as shown in Figure 1. This structure simplifies policy coverage across security, governance, risk, and compliance (GRC), and engineering teams.

This post shows you how to use Open Policy Agent (OPA) in continuous integration and continuous delivery (CI/CD) pipelines to validate Amazon Web Services (AWS) infrastructure changes before deployment. You will learn how to structure policy checks around recurring control patterns, fit those checks into a gated delivery workflow, and retain validation artifacts that support both release decisions and later audit review.

The Compliance Engineering and Automation team from AWS Security Assurance Services (AWS SAS) frequently helps customers implement policy as code as part of broader control design and compliance automation efforts. This post focuses on the pre-deployment layer. Runtime monitoring and post-deployment controls still matter, but they are outside the scope of this article.

Figure 1: Pattern-based policy as code in a gated delivery workflow

Figure 1: Pattern-based policy as code in a gated delivery workflow

Organize policies around recurring patterns

Teams sometimes build rules one service at a time, which can make policy as code libraries difficult to review and extend as the library grows. Similar control requirements can be expressed differently across repositories, and teams lose a common way to discuss what the policies are enforcing.

A pattern-based approach organizes policies around recurring control intent rather than service-specific checks, as shown in Figure 2. This makes coverage more straightforward to review, explain, and evolve as infrastructure changes.

A practical set of patterns includes:

  • Required metadata – for tags and other fields used for ownership, support, cost allocation, and automation.
  • Allowed configuration – for approved Regions, accepted deployment boundaries, and other approved settings.
  • Exposure restriction – for configurations that make infrastructure more reachable than intended, such as public ingress or internet-facing resources in the wrong environment.
  • Protection enforcement – for baseline safeguards such as encryption, logging, or deletion protection.
  • Privilege constraint – for AWS Identity and Access Management (IAM) definitions and access patterns that need tighter validation.
Figure 2: Recurring control patterns used to organize policy as code checks

Figure 2: Recurring control patterns used to organize policy as code checks

Where OPA fits in a layered governance model

This post focuses on the preventive layer. You still need runtime controls, drift monitoring, remediation workflows, and compliance reporting. On AWS, AWS Organizations, AWS Control Tower, AWS Config, and AWS Security Hub remain important after resources exist.

OPA fits earlier in the process and validates that infrastructure changes align with expectations. OPA evaluates structured input (HashiCorp Terraform plan JSON) against policy logic. It doesn’t replace AWS governance services that provide organizational guardrails, continuous monitoring, and resource level enforcement after resources exist.

As shown in Figure 3:

  • OPA – Checks proposed changes before deployment
  • AWS Organizations and Control Tower – Establish organizational guardrails
  • AWS Config and Security Hub – Provide visibility and monitoring after resources exist
  • Service-level protections – Enforce settings at the resource boundary
Figure 3: OPA validates changes pre-deployment; AWS services enforce guardrails, monitoring, and controls post-deployment

Figure 3: OPA validates changes pre-deployment; AWS services enforce guardrails, monitoring, and controls post-deployment

How to implement policy validation in your CI/CD pipeline

Use the following steps to integrate OPA policy evaluation into your delivery workflow:

Submit a change through a pull request or merge request.

  1. Run early validation checks such as formatting, syntax validation, and dependency checks.
  2. Generate a Terraform plan and convert it to JSON format.
  3. Evaluate the plan (JSON format) against the shared OPA policy library.
  4. Publish the validation report as an artifact.
  5. Run additional automated quality checks as needed.
  6. Use the validation artifact during approval decisions for higher-risk environments.
  7. Deploy approved changes.
  8. Continue post-deployment monitoring through AWS-native governance services.

Quality gates provide automated pass or fail results based on defined criteria. Approval gates control whether a change moves into a protected environment. This separation matters—manual approval isn’t the first place where anyone notices missing tags, a disallowed AWS Region, or public ingress. Automated checks identify those issues earlier. OPA belongs in the automated gate layer. Its output also feeds the approval process.

Structure your policy library by control domain and intent

A pattern-based library structure, as shown in the following sample, keeps the policy model closer to how teams talk about controls.

  opa-policies/
  ├── patterns/
  │ ├── baseline/ # Foundational security
  │ ├── tagging/ # Required tags
  │ ├── networking/ # Network controls
  │ ├── logging/ # Logging enablement
  │ ├── encryption/ # Encryption at rest and transit
  │ └── iam/ # IAM best practices
  ├── shared/
  │ ├── helpers.rego
  │ └── messages.rego
  ├── tests/
  ├── fixtures/
  └── docs/

A compliance engineer might describe a requirement as mandatory metadata. A cloud engineer might describe the same requirement as a tagging standard. The pattern structure helps both teams talk about the same thing.

Example 1: Enforce secure transport for Amazon S3

This example demonstrates the protection enforcement pattern for Amazon Simple Storage Service (Amazon S3). The goal is to verify that S3 bucket access is protected in transit by requiring a bucket policy that denies requests when aws:SecureTransport is set to false.

The policy checks two things: whether an S3 bucket policy includes a deny statement that blocks non-encrypted requests, and whether an S3 bucket has any corresponding bucket policy at all. The rule evaluates both create and update actions in the Terraform plan JSON.

This example uses an explicit deny rather than an allow statement for secure transport. An explicit deny overrides allow statements that might exist elsewhere in the policy set, making it the stronger enforcement pattern.

package compliance.amazon_s3.ssl

import future.keywords.in
import future.keywords.contains
import future.keywords.if

# Deny: S3 bucket policy missing SecureTransport deny statement
deny contains msg if {
    resource := input.resource_changes[_]
    resource.type == "aws_s3_bucket_policy"
    is_create_or_update(resource.change.actions)

    policy_value := resource.change.after.policy
    policy := json.unmarshal(policy_value)

    not has_secure_transport_deny(policy)

    msg := sprintf(
        "[S3-OPA-1] Resource '%s' does not enforce SSL/TLS. Bucket policy must include a Deny statement with Condition Bool aws:SecureTransport set to \"false\".",
        [resource.address]
    )
}

# Deny: S3 bucket created without any corresponding bucket policy
deny contains msg if {
    resource := input.resource_changes[_]
    resource.type == "aws_s3_bucket"
    is_create_or_update(resource.change.actions)

    bucket_name := resource.change.after.bucket
    not has_bucket_policy(bucket_name)

    msg := sprintf(
        "[S3-OPA-1] Resource '%s' (bucket '%s') has no bucket policy. A bucket policy with a Deny statement for aws:SecureTransport \"false\" is required.",
        [resource.address, bucket_name]
    )
}

is_create_or_update(actions) if { actions[_] == "create" }
is_create_or_update(actions) if { actions[_] == "update" }

has_bucket_policy(bucket_name) if {
    bp := input.resource_changes[_]
    bp.type == "aws_s3_bucket_policy"
    is_create_or_update(bp.change.actions)
    bp.change.after.bucket == bucket_name
}

has_secure_transport_deny(policy) if {
    stmt := policy.Statement[_]
    stmt.Effect == "Deny"
    stmt.Condition.Bool["aws:SecureTransport"] == "false"
    stmt.Principal == "*"
    action := stmt.Action
    action == "s3:*"
}

When you adapt this example, decide whether you want to require one exact policy shape or support several equivalent forms of enforcement. A strict rule is more straightforward to reason about, but it might create false positives if teams already use alternate policy structures that achieve the same outcome.

Example 2: Restrict public ingress on sensitive ports

This example implements the exposure restriction pattern. The goal is to identify Amazon Virtual Private Cloud (Amazon VPC) security group configurations that allow public ingress on sensitive ports before those rules are deployed.

The policy evaluates both inline aws_security_group ingress rules and standalone aws_security_group_rule resources, because customer repositories often use both modeling styles.

This example checks directly for public ingress on sensitive ports rather than trying to infer whether later controls might reduce actual exposure. Security group rules are a direct expression of intended network reachability, making them the right place to enforce this pattern early.

package compliance.amazon_vpc.ingress

import future.keywords.in
import future.keywords.contains
import future.keywords.if

# Sensitive ports that must not be open to the internet
sensitive_ports := {22, 3389, 5432}

# Deny: aws_security_group with inline ingress open to 0.0.0.0/0 on sensitive ports
deny contains msg if {
    resource := input.resource_changes[_]
    resource.type == "aws_security_group"
    is_create_or_update(resource.change.actions)

    ingress := resource.change.after.ingress[_]
    ingress.cidr_blocks[_] == "0.0.0.0/0"

    port := sensitive_ports[_]
    ingress.from_port <= port
    ingress.to_port >= port

    msg := sprintf(
        "[VPC-OPA-1] Resource '%s' allows ingress from 0.0.0.0/0 on port %d. Restrict access to specific CIDR ranges.",
        [resource.address, port]
    )
}

# Deny: aws_security_group_rule with type "ingress" open to 0.0.0.0/0 on sensitive ports
deny contains msg if {
    resource := input.resource_changes[_]
    resource.type == "aws_security_group_rule"
    is_create_or_update(resource.change.actions)

    resource.change.after.type == "ingress"
    resource.change.after.cidr_blocks[_] == "0.0.0.0/0"

    port := sensitive_ports[_]
    resource.change.after.from_port <= port
    resource.change.after.to_port >= port

    msg := sprintf(
        "[VPC-OPA-1] Resource '%s' allows ingress from 0.0.0.0/0 on port %d. Restrict access to specific CIDR ranges.",
        [resource.address, port]
    )
}

is_create_or_update(actions) if { actions[_] == "create" }
is_create_or_update(actions) if { actions[_] == "update" }

When you adapt this example, review which ports to treat as sensitive, whether both IPv4 and IPv6 exposure need checking, and how to handle approved exceptions.

Example 3: Enforce least privilege trust policy for IAM roles

This example implements the privilege constraint pattern for IAM role trust policies. The goal is to identify trust relationships that allow overly broad principals to assume a role. The policy inspects the assume_role_policy document for aws_iam_role resources and looks for wildcard principals in three valid representations: Principal is "*", Principal.AWS is "*", and Principal.AWS is an array containing "*". A wildcard principal allows a broader set of callers than most environments intend to permit. By treating wildcard principals as the prohibited pattern, the rule enforces a safer default and returns a clear result that reviewers can understand quickly.

package compliance.amazon_iam.trust

import future.keywords.in
import future.keywords.contains
import future.keywords.if

# Deny: IAM role with wildcard principal in trust policy
deny contains msg if {
    resource := input.resource_changes[_]
    resource.type == "aws_iam_role"
    is_create_or_update(resource.change.actions)

    policy_value := resource.change.after.assume_role_policy
    policy := json.unmarshal(policy_value)

    stmt := policy.Statement[_]
    stmt.Effect == "Allow"
    has_wildcard_principal(stmt)

    msg := sprintf(
        "[IAM-OPA-2] Resource '%s' has a wildcard principal in its trust policy. Specify explicit account ARNs, service principals, or federated providers instead of \"*\".",
        [resource.address]
    )
}

# Principal is directly "*"
has_wildcard_principal(stmt) if {
    stmt.Principal == "*"
}

# Principal.AWS is "*"
has_wildcard_principal(stmt) if {
    stmt.Principal.AWS == "*"
}

# Principal.AWS is an array containing "*"
has_wildcard_principal(stmt) if {
    stmt.Principal.AWS[_] == "*"
}

is_create_or_update(actions) if { actions[_] == "create" }
is_create_or_update(actions) if { actions[_] == "update" }

When you adapt this example, decide what least privilege means for your IAM trust model. The key design choice is whether your policy checks for a single prohibited pattern or validates trust relationships against an approved set of trusted principals and conditions.

AWS Labs provides IAM Policy Autopilot, an open-source Model Context Protocol (MCP) server and command-line tool that helps generate baseline identity-based IAM policies from application code. That is adjacent to the pattern shown here —IAM Policy Autopilot helps with policy generation, while this example focuses on validating whether IAM role trust policies are scoped appropriately in infrastructure changes.

CI/CD implementation examples

The following examples show the same operating model in two common CI/CD systems. The syntax changes, but the sequence stays the same: validate, plan, evaluate policy, retain the artifact, and use the result during promotion and approval. These examples assume OPA is installed in your CI/CD environment, the opa-policies directory contains your policy library, and Terraform is configured with appropriate credentials.

GitLab CI

stages:
  - validate
  - plan
  - policy_check

variables:
  TF_IN_AUTOMATION: "true"

terraform_validate:
  stage: validate
  script:
    - terraform fmt -check
    - terraform init
    - terraform validate

terraform_plan:
  stage: plan
  script:
    - terraform plan -out=tfplan
    - terraform show -json tfplan > tfplan.json
  artifacts:
    paths:
      - tfplan.json

opa_policy_check:
  stage: policy_check
  script:
    - opa eval --format pretty --data opa-policies --input tfplan.json "data.terraform.deny"
    - opa eval --format json --data opa-policies --input tfplan.json "data.terraform.deny" > policy-report.json
  artifacts:
    paths:
      - policy-report.json

GitHub Actions

name: Terraform Policy Check
on:
  pull_request:

jobs:
  policy-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3

      - name: Terraform Format Check
        run: terraform fmt -check
      - name: Terraform Init
        run: terraform init
      - name: Terraform Validate
        run: terraform validate
      - name: Terraform Plan
        run: terraform plan -out=tfplan
      - name: Convert Plan to JSON
        run: terraform show -json tfplan > tfplan.json
      - name: Run OPA Policy Check
        run: |
          opa eval --format pretty --data opa-policies --input tfplan.json "data.terraform.deny"
          opa eval --format json --data opa-policies --input tfplan.json "data.terraform.deny" > policy-report.json
      - name: Upload Validation Artifact
        uses: actions/upload-artifact@v4
        with:
          name: policy-report
          path: policy-report.json

Retain validation artifacts for review and audit support

In mature delivery workflows, policy results don’t disappear into pipeline logs but are retained as validation artifacts. Those artifacts help reviewers decide whether a change is ready for approval, supports exception handling by showing which controls failed and why, and can stay with the change record for later audit discussions. At a minimum, the artifact identifies the change or pipeline run, the evaluated scope, the policy package or version, the checks that ran, and the pass or fail results.

Test the policy model like software

The first few rules are usually straightforward.The real work starts when the library grows and multiple teams depend on it. Testing includes:

  • Positive and negative test cases – Each policy has cases that show valid input and cases that show expected failures.
  • Regression coverage – Shared helpers need regression coverage.
  • Realistic fixtures – Terraform plan fixtures look like real changes rather than tiny made-up samples.
  • Impact analysis – When a rule changes, teams can tell quickly what else might be affected.

If developers stop trusting the results, they stop treating policy as a useful mechanism.

A phased approach to rolling out policy checks

You don’t need broad coverage on day one. A phased rollout works better than an all at once enforcement approach.

Phase 1: Assess and pilot

  • Start in advisory mode so teams can see results without being blocked.
  • Identify two or three high-confidence patterns such as required metadata, approved Regions, or public exposure restrictions.
  • Run OPA against existing pipelines and review the output for accuracy.

Phase 2: Begin enforcement

  • Enforce the small set of high-confidence patterns after the output is stable and the failures are useful.
  • Integrate validation artifacts into your approval workflow.
  • Establish ownership and exception handling processes for shared packages.

Phase 3: Operationalize and expand

  • Formalize versioning for shared policy packages.
  • Expand pattern coverage based on team feedback and organizational priorities.
  • Connect pre-deployment validation with post-deployment monitoring through AWS Config, AWS Security Hub, and AWS Organizations.

Conclusion

Policy as code helps narrow the distance between what an organization says it expects and what its delivery system checks. By implementing these OPA patterns in your CI/CD pipelines, you can build a preventive layer that evaluates infrastructure changes before deployment. With a pattern-based library, validation artifacts, and clear ownership, policy as code becomes a repeatable way to help translate control intent into day-to-day delivery, while AWS governance services continue to provide visibility and monitoring after resources exist.

To learn more about policy as code and AWS governance capabilities, see:

If you have feedback about this post, submit comments in the Comments section below.

Guptaji Teegela

Guptaji Teegela

Guptaji is a Cloud Infrastructure Architect with AWS Security Assurance Services, where he focuses on compliance automation and policy-as-code for regulated workloads. He brings over 15 years of hands-on experience across site reliability engineering, platform engineering, and cloud architecture, with deep expertise in both AWS and Azure environments. Backed by a broad portfolio of industry certifications spanning cloud and security domains, Guptaji is driven by a passion for helping customers design and deliver reliable, secure, and highly automated cloud platforms.

Paul Keastead

Paul Keastead

Paul is a Senior Security Engineer with AWS Global Professional Services Security, specializing in compliance automation, policy as code, and security engineering for regulated workloads. A CISSP-ISSEP, Lead CMMC Certified Assessor, and former FedRAMP Assessor, he builds automated validation pipelines that translate control requirements into preventive, testable checks in delivery workflows. He brings over a decade of experience in national security and public sector technology compliance.

A systematic approach to benchmarking SQL processing engines on AWS

Post Syndicated from Anubhav Awasthi original https://aws.amazon.com/blogs/big-data/a-systematic-approach-to-benchmarking-sql-processing-engines-on-aws/

Selecting the right SQL processing solution for large-scale data analytics is a critical decision for organizations. As data volumes grow exponentially, the technology landscape has evolved to offer diverse options for processing and analyzing this information efficiently. This post presents a systematic framework for evaluating and benchmarking SQL processing engines on AWS, using Apache JMeter to conduct practical performance testing at scale.

The AWS analytics ecosystem

AWS offers a rich portfolio of SQL processing solutions to meet various analytical needs:

  • Serverless query services – Amazon Athena is a serverless, interactive query service that uses standard SQL to analyze data in Amazon Simple Storage Service (Amazon S3), offering automatic scaling, parallel query execution, and pay-per-query pricing with no infrastructure management required
  • Data warehouse solutions – Amazon Redshift offers scalable, high-performance cloud data warehousing with serverless options, zero-ETL integrations, AI-powered query assistance, and seamless machine learning (ML) integration for modern analytics at scale
  • Managed open source engines – Amazon EMR supports Apache Spark SQL, Apache Trino (formerly PrestoSQL), and other distributed query frameworks
  • Self-managed options – You can deploy open source engines like Apache Spark, Apache Flink, and Trino on Amazon Elastic Kubernetes Service (Amazon EKS) for greater control
  • Partner solutions – You can access specialized big data analytics tools through AWS Marketplace

These options are further enhanced by modern open table formats such as Apache Iceberg, Delta Lake, and Apache Hudi, which bring crucial enterprise features like ACID (Atomicity, Consistency, Isolation, and Durability) transactions, schema evolution, and time travel capabilities to data lakes. These SQL processing solutions operate under the AWS Shared Responsibility Model. AWS manages the security of the underlying infrastructure and services, and customers are responsible for secure configuration, access management, and data protection within their testing environments. This division of responsibility remains important when evaluating and benchmarking different SQL engines. Proper security configuration and implementation by customers is essential for maintaining a secure analytics environment.

Evaluation challenges in SQL engine selection

The rich ecosystem of SQL processing options creates significant evaluation challenges. Each SQL engine employs unique architectural approaches and optimization strategies, making direct comparisons complex. Organizations embarking on this evaluation journey face several interconnected obstacles:

  • Creating environments that accurately reflect production scenarios
  • Developing test datasets that mirror real-world data characteristics and volumes
  • Replicating real-world query patterns and concurrency levels
  • Maintaining uniform testing conditions across different engine architectures
  • Controlling infrastructure expenses throughout the evaluation process

Performance considerations at petabyte scale

When evaluating solutions for petabyte-scale deployments, the complexity intensifies considerably. Several critical factors come into play:

  • Resource management – Distributed SQL engines require precise balancing of CPU, memory, and storage resources. Suboptimal resource allocation can lead to query failures and performance degradation, particularly as data volumes grow.
  • Data distribution patterns – How data is distributed across partitions or nodes significantly impacts query performance. Data skew can create processing bottlenecks, with some nodes handling disproportionate workloads while others remain underutilized.
  • Concurrency handling – High-concurrency environments demand sophisticated workload scheduling and resource isolation mechanisms. The ability to maintain consistent performance under varying concurrent loads becomes a critical differentiator between solutions.
  • Meaningful metrics – Performance evaluation at scale requires comprehensive metrics analysis:
    • Mean, median, and percentile response times (particularly p90 and p95)
    • Query throughput under varying concurrency levels
    • Scalability characteristics across diverse workload types
    • Resource utilization efficiency during peak loads

Limitations of traditional benchmarks

Although industry-standard benchmarks like TPC-DS and TPC-H provide valuable insights, our experience with multiple customer engagements has shown that tailored, workload-specific testing often reveals performance characteristics not captured by these standardized tests. This is especially true for complex, multi-tenant environments with diverse query patterns. Organizations that complement standard benchmarks with workload-specific testing typically experience shorter proof-of-concept cycles, optimized evaluation costs, and more efficient testing operations. This comprehensive approach helps reduce uncertainty in the final solution selection process.

Prerequisites

Before you dive into the evaluation process, make sure you have the following prerequisites:

  • An AWS account with appropriate permissions to create and manage Amazon Elastic Compute Cloud (Amazon EC2) instances and access the SQL engines you plan to benchmark.
  • Basic familiarity with AWS services, particularly Amazon EC2 and the SQL engines you intend to evaluate (such as Athena, Amazon Redshift, or Amazon EMR).
  • Experience with SQL and data analytics concepts.
  • Access to the SQL engines you choose to benchmark. This post assumes you’ve already set up the engines you want to test. For setup instructions, refer to the AWS documentation for each service.
  • A dataset suitable for your benchmarking needs. Dataset creation and loading are not covered in this post. Build petabyte-scale synthetic test data with Amazon EMR on EC2 provides prescriptive guidance to generate test datasets at scale. Make sure your test datasets are stored in S3 buckets with encryption enabled (using SSE-KMS or SSE-S3) and that all service connections use TLS for data in transit.

Benefits of Apache JMeter

As organizations scale their analytics workloads to petabyte levels, there is a growing need for a robust, structured approach to SQL query performance testing. Although many organizations develop custom testing frameworks or use various benchmarking tools, these approaches often lack standardization and can be difficult to replicate across different SQL engines. The complexity of modern data architectures, combined with the variety of available SQL processing solutions, demands a systematic evaluation methodology. Apache JMeter emerges as a powerful solution to address this challenge. Though traditionally known for web application testing, JMeter’s extensible architecture and robust feature set make it particularly well-suited for SQL performance testing at scale.JMeter offers several advantages for evaluating SQL engines:

  • Support for multiple protocols and connections
  • Ability to simulate complex concurrent workloads
  • Built-in performance metrics and reporting
  • Extensible architecture for custom testing scenarios
  • Integration capabilities with continuous integration and continuous delivery (CI/CD) pipelines

Through this proposed framework, which has been validated across multiple customer engagements at petabyte scale, we aim to help organizations make more informed decisions when selecting a SQL processing solution. Our experience working with customers to assess various AWS Analytics services and open source solutions has demonstrated that a systematic evaluation approach significantly reduces proof-of-concept cycles and optimizes resource investments. This framework has helped organizations effectively evaluate services like Athena, Amazon Redshift, and Amazon EMR, alongside open source solutions such as Trino on Amazon EKS, based on their specific workload profiles and performance requirements.With this methodology, organizations can accomplish the following:

  • Navigate the complex landscape of large-scale data processing technologies
  • Reduce proof-of-concept cycles from months to weeks
  • Minimize infrastructure costs during evaluation phases
  • Make data-driven decisions about technology selection
  • Better align technology choices with business requirements
  • Establish repeatable testing patterns for future evaluations

Testing methodology in practice

A successful SQL engine evaluation requires understanding and replicating real-world workload patterns. Our methodology, refined through numerous customer engagements, focuses on comprehensive testing across multiple dimensions while remaining adaptable to specific organizational needs.

Query pattern selection

We begin by selecting representative query patterns that mirror production workloads:

  • Aggregation queries that summarize large datasets using operations like SUM, AVG, and COUNT
  • Complex join operations that test the engine’s ability to combine data efficiently across multiple tables
  • String operations that evaluate text processing capabilities
  • Nested queries that assess the engine’s optimization capabilities for complex query structures

A carefully selected set of 8–10 queries typically provides sufficient coverage while keeping the evaluation manageable. These should reflect your actual workload characteristics and business requirements.

Data volume variations

Testing across different data volumes is important for understanding scalability characteristics. We structure our tests around varying data scan ranges:

  • Small-scale scans – Queries accessing 1–7 days of data (megabytes to gigabytes)
  • Large-scale scans – Queries spanning 14–30 days (terabytes to petabytes)

This approach evaluates both I/O efficiency with large datasets and metadata handling with smaller, frequent queries, helping understand how services like Amazon EMR, Amazon Redshift, or Athena optimize query execution across different access patterns.

Concurrency testing

Real-world analytics environments rarely process single queries in isolation. Our methodology incorporates the following features:

  • Progressive concurrency testing starting at lower levels (typically 16, 32, 64, and 128 parallel queries), though these numbers can be adjusted based on your test infrastructure capacity and specific requirements. We recommend starting with smaller concurrency levels and gradually scaling up to understand performance characteristics
  • Varied query complexity and frequency (referred to as query weights) to simulate realistic workload distributions. This means some queries are run more often or are more resource-intensive than others, mimicking real-world usage patterns.
  • Mixed query patterns running simultaneously to test resource management.
  • Consistent execution across different date ranges to evaluate scaling behavior.

This approach is particularly important when evaluating managed services like the workload management capabilities of Amazon Redshift or the resource allocation strategies of Amazon EMR.

Query weight distribution

Production environments typically see varying frequencies of different query types. Our framework incorporates weighted query distribution to simulate real-world scenarios more accurately. In a typical distribution, frequent lightweight queries might represent 60% of the workload, complex analytical queries might comprise 30%, and resource-intensive data processing operations might make up the remaining 10%.This weighted approach makes sure performance testing reflects actual usage patterns rather than artificial benchmarking scenarios. The exact distribution should mirror your organization’s specific workload patterns.

Sequential vs. concurrent testing

Our methodology implements two distinct testing phases:

  • Sequential testing – Establishes baseline performance metrics:
    • Runs each query type independently across different date ranges
    • Runs multiple iterations to provide consistency and identify variability
    • Helps understand individual query performance characteristics
  • Concurrent testing – Simulates real-world multi-user scenarios:
    • Implements weighted query distributions
    • Tests different concurrency levels to identify scaling limitations
    • Evaluates resource management capabilities of different engines

JMeter efficiently implements both testing phases while maintaining consistent test conditions across SQL engines. Its ability to handle various JDBC connections makes it particularly suitable for testing AWS analytics services.Through this structured approach, organizations can gather comprehensive performance data reflecting their specific use cases, enabling informed SQL engine selection decisions while maintaining core principles of systematic evaluation and realistic workload simulation.

Test plans

To evaluate SQL engines’ performance under varying workloads, we designed two test scenarios: sequential and concurrent execution plans. Each scenario was executed across different data volumes by adjusting the query date range filters to cover 1, 7, 14, and 30 days. These variations simulate typical analytical workloads with progressively increasing data sizes.For sequential runs, each test was treated as a distinct batch, grouping all queries (Query 1 to Query 9) under the same date range—each query will scan data for 1, 7, 14, and 30 days with appropriate date filtering in the query’s where predicate. We used JMeter to capture average query response times for each batch. This configuration was run three times, and the final metrics reflect the average response time across these iterations to ensure reliability and account for environmental variance.Although three iterations provide initial insights, if you observe significant variations in results (typically more than 10% deviation between runs), consider expanding to 10 or more iterations. This additional sampling helps establish statistical significance, identify true performance patterns, and distinguish outliers (beyond three standard deviations) from normal variations. Document any consistent anomalies, because they may indicate important performance or security considerations for your specific environment.The following table shows the sample test plans template for the sequential test plan run.

 

Dataset Time Range Run Query Weights
Query 1 Query 2 Query 3 Query 4 Query 5 Query 6 Query 7 Query 8 Query 9
1 day Run 1
Run 2
Run 3
Avg
7 days Run 1
Run 2
Run 3
Avg
14 days Run 1
Run 2
Run 3
Avg
30 days Run 1
Run 2
Run 3
Avg

For the concurrent test plan, we introduced a probabilistic weighted distribution to the queries (Query 1 to Query 9), simulating a more realistic production-like environment where query frequency varies based on business relevance and usage patterns. This added a layer of complexity to better reflect how the SQL engine would perform under real-world concurrent access patterns.The following table shows the sample test plans template for the concurrent test plan run.

Dataset Time Range Concurrent Runs Query Weights
Query 1 Query 2 Query 3 Query 4 Query 5 Query 6 Query 7 Query 8 Query 9
1 days 8 11% 11% 11% 11% 11% 11% 11% 11% 11%
16 10% 5% 24% 5% 5% 5% 24% 14% 10%
32 8% 3% 24% 5% 5% 5% 24% 16% 8%
64 7% 3% 24% 6% 4% 6% 26% 16% 9%
128 1% 4% 19% 8% 5% 7% 14% 20% 22%
*7 days 8 11% 11% 11% 11% 11% 11% 11% 11% 11%
16 10% 5% 24% 5% 5% 5% 24% 14% 10%
32 8% 3% 24% 5% 5% 5% 24% 16% 8%
64 7% 3% 24% 6% 4% 6% 26% 16% 9%
**128 1% 4% 19% 8% 5% 7% 14% 20% 22%
14 days 8 11% 11% 11% 11% 11% 11% 11% 11% 11%
16 10% 5% 24% 5% 5% 5% 24% 14% 10%
32 8% 3% 24% 5% 5% 5% 24% 16% 8%
64 7% 3% 24% 6% 4% 6% 26% 16% 9%
128 1% 4% 19% 8% 5% 7% 14% 20% 22%
30 days 8 11% 11% 11% 11% 11% 11% 11% 11% 11%
16 10% 5% 24% 5% 5% 5% 24% 14% 10%
32 8% 3% 24% 5% 5% 5% 24% 16% 8%
64 7% 3% 24% 6% 4% 6% 26% 16% 9%
128 1% 4% 19% 8% 5% 7% 14% 20% 22%

For example, for configuration of *7 days concurrent run with **128 concurrency, the proposed configuration distributes Query 1 to Query 9 with appropriate weighted submissions such that Query 9 is executed the greatest number of times in the overall 128 executions submitted across all 9 queries for this run.

JMeter setup

To begin, you must set up JMeter on a machine that can handle the desired test load. An EC2 instance is a flexible and cost-effective option. Choose an instance type with sufficient vCPUs to support your maximum planned concurrency. For example, a c6i.4xlarge or higher is typically suitable for moderate to high throughput testing scenarios. For the operating system, you might choose Amazon Linux, which is optimized for AWS. For production-grade testing environments, deploy the JMeter EC2 instance in a private subnet of a virtual private cloud (VPC) with appropriate security groups that allow only required connections. This network isolation helps maintain security while executing performance tests. Consider using Amazon Virtual Private Cloud (Amazon VPC) endpoints for secure access to AWS services.

After the instance is provisioned, install Java (Java 17 LTS or Java 21 LTS) and download the latest version of JMeter. Be sure to configure the system with appropriate JVM options to allocate sufficient heap memory for large-scale test executions. Refer to Getting Started to learn more.

# Install Java
sudo yum update -y # For Amazon Linux
sudo yum install java-17-amazon-corretto -y

# Download JMeter and place the appropriate jdbc driver for the engine of your selection under lib folder
wget https://downloads.apache.org//jmeter/binaries/apache-jmeter-5.6.3.tgz
tar -xvzf apache-jmeter-5.6.3.tgz
cd apache-jmeter-5.6.3/lib

# Launch JMeter in GUI mode (if using a GUI-capable setup) or use CLI for remote testing
./bin/jmeter

JMeter concepts

Before you create test plans in JMeter, it’s important to understand a few foundational concepts that influence how your test plan behaves—such as thread groups, user-defined variables, and JDBC connection. These components enable the simulation of real-world query loads, including concurrency and pacing.

Test plans

The test plan is the top-level container for a JMeter test. It defines the overall testing strategy, including the queries to execute, their parameters, and the concurrent user behavior. These plans are represented as jmx files that can then be used for CLI-based execution. JMeter supports both GUI and CLI modes. It is highly recommended that you use the JMeter GUI primarily for creating test plans as jmx, and use the CLI for large load tests. You can also run thread groups consecutively for sequential execution. The default behavior is to run all thread groups in parallel suited for concurrent execution. Refer to Building a Test Plan to learn more about options available with test plans.

User-defined variables

User-defined variables are global parameters that you can reuse throughout the test plan. They are helpful for defining database credentials, server URLs, or query parameters. For example:DB_URL=jdbc:trino://trino-cluster.example.com:8889?SSL=true #Enable SSL/TLS

You can configure authentication (user name and password) through your organization’s approved methods, such as AWS Secrets Manager (see Move hardcoded secrets to AWS Secrets Manager) AWS Identity and Access Management (IAM) roles, or other secure credential management systems.

Thread groups

A thread group represents a group of virtual users (threads) executing test actions. Each thread simulates a single user sending requests to the SQL engine. This can be used to simulate concurrent runs. For example, in the preceding template, Query 3 has 19% weightage across 128 runs. This means .19*128=25 total runs, so we set the thread group to 25.

JDBC connection configuration

JDBC connection configuration sets up the database connection for the test. It specifies the database URL, driver, and credentials required for executing SQL queries. Key fields to configure are database URL and JDBC driver class. The following table summarizes the different configuration settings.

 

SQL Engine JDBC Driver JDBC Driver Class
Trino on EMR trino-jdbc-<trino_version>-amzn-0.jar io.trino.jdbc.TrinoDriver
Athena Athena JDBC 3.x driver com.amazon.athena.jdbc.AthenaDriver
Amazon Redshift Amazon Redshift JDBC driver com.amazon.redshift.jdbc.Driver
Trino on EKS Trino JDBC driver io.trino.jdbc.TrinoDriver

JDBC requests

The JDBC request executes SQL queries against the database using the configuration defined in the JDBC connection configuration.

For example, following command runs the JMeter in CLI mode:

# Run benchmarks in CLI mode 
./jmeter -n -t <path_to>.jmx -l <local path for log>.log -e -o <local path for>/output/

The output folder will contain an HTML report with different statistics. The following screenshot illustrates 128 concurrent runs.

Monitoring and logging

For comprehensive visibility and audit requirements, enable AWS CloudTrail logging, VPC Flow Logs, and service-specific logs (like Amazon S3 access logs). These logs can be centralized in Amazon CloudWatch Logs for monitoring and analysis. This provides proper audit trails while evaluating different SQL engines and helps track access patterns and potential security events.

Post-test steps

After running your JMeter tests, proceed with the following steps:

  1. Review the HTML report’s key metrics, including response times, throughput, and error rates across different query types and concurrency levels.
  2. Run identical test plans across your candidate SQL engines for direct performance comparison.
  3. Refine your test plans based on initial findings, focusing on areas where performance differences are significant.
  4. Factor in the cost implications alongside performance metrics to make a balanced decision.

These steps can help you systematically evaluate and select the most suitable SQL engine for your analytics workloads.

Resources

In the preceding steps, we walked through a UI-based setup for JMeter along with test plans. We have created a few sample JMeter test plans for both sequential and concurrent runs along with sample test reports. You can modify the plans to fit your needs.

  1. JMeter sample report
  2. JMeter test plan for sequential run
  3. JMeter test plan for concurrent run

Clean up

After you complete your benchmarking process, clean up the resources to avoid unnecessary costs:

  1. Stop or delete the EC2 instances used for running JMeter.
  2. Depending on which SQL engines you used for testing, clean up active resources.
  3. Review your AWS Management Console to confirm no active resources remain.
  4. If you created test datasets in Amazon S3 or other storage services specifically for this benchmarking, consider deleting them if they’re no longer needed.
  5. Although JMeter test plans and results don’t incur AWS costs, organize or delete local files as needed for your record-keeping.

Summary

Selecting the right SQL processing solution for large-scale analytics demands a systematic, data-driven approach. Our JMeter framework can help organizations effectively evaluate different SQL engines by simulating real-world workload patterns across various query types, data volumes, and concurrency levels. This methodology reduces proof-of-concept cycles and provides insights beyond traditional benchmarks, helping you assess managed AWS services like Athena and Amazon Redshift and open source solutions on Amazon EKS.


About the authors

Anubhav Awasthi

Anubhav Awasthi

Anubhav is a Senior Big Data Specialist Solutions Architect at Amazon Web Services (AWS). He collaborates with customers to provide expert architectural guidance for implementing and optimizing analytics solutions using Amazon EMR, Amazon Athena, AWS Glue, and AWS Lake Formation.

Gagan Brahmi

Gagan Brahmi

Gagan is a Specialist Senior Solutions Architect at Amazon Web Services (AWS), focused on Data Analytics and AI/ML. With over 20 years in information technology, he partners with customers to solve complex AI/ML challenges by leveraging data and AI/ML platforms. Gagan helps customers architect scalable, high-performance solutions that utilize distributed data processing, real-time streaming technologies, and AI/ML services to drive business transformation through artificial intelligence and data-driven insights. When not designing cloud-native data and AI solutions, Gagan enjoys exploring new places with his family.

Jayaprakash Boreddy

Jayaprakash Boreddy

Jayaprakash is a Senior Solutions Architect at AWS. He works with ISV customers in designing and building highly scalable, flexible and resilient applications on AWS Cloud.

Sahil Thapar

Sahil Thapar

Sahil is a Principal Solutions Architect. He works with ISV customers to help them build highly available, scalable, and resilient applications on the AWS Cloud.

Build petabyte-scale synthetic test data with Amazon EMR on EC2

Post Syndicated from Anubhav Awasthi original https://aws.amazon.com/blogs/big-data/build-petabyte-scale-synthetic-test-data-with-amazon-emr-on-ec2/

As you scale your data systems, you face a challenge: how to test thoroughly without putting customer data at risk. Using production data for testing can expose sensitive customer information to unauthorized access or breaches. For customers in regulated industries like finance and healthcare, this risk isn’t only a concern. It’s unacceptable. A data breach during testing could compromise their privacy, damage their trust, and expose organizations to significant compliance penalties. Synthetic test data solves this problem by generating artificial datasets that replicate the structure and patterns of real data without containing any actual customer information. This approach means you can test performance, validate data pipelines, and develop new features while ensuring that customer data remains protected and compliance requirements are met.

As data volumes grow from terabytes to petabytes, the architecture for generating synthetic data must evolve to meet increasing demands for scale, performance, and data quality. In this post, we show how you can build a scalable synthetic data generation solution using Amazon EMR, Apache Spark, and the Faker library.

The challenge of synthetic data generation

Traditional benchmark datasets like TPC-DS provide standardized schemas and predetermined data volumes for consistent testing environments across different systems. However, they fall short in meeting real-world testing requirements. These benchmarks don’t capture industry-specific patterns or the complex relationships found in actual production data. Their rigid schemas and simplified distributions fail to reflect business requirements, and scaling them while maintaining data consistency proves difficult. Perhaps most critically, generating massive datasets with traditional approaches requires specialized architectures to avoid proportional increases in compute costs and time.

Requirements for production-grade synthetic data

Effective workload validation demands synthetic data that mirrors production distributions while maintaining referential integrity across related tables and entities. The generation process must scale horizontally to accommodate growing data volumes while delivering deterministic results. Given identical input parameters, the system should produce the same dataset across multiple runs, supporting consistent testing cycles and comparative analysis.

Beyond technical requirements, synthetic data addresses compliance needs by minimizing exposure of personally identifiable information (PII) and protected health information (PHI) in non-production environments. This approach satisfies GDPR, HIPAA, and CCPA requirements while supporting secure cross-border data transfer, regular stress testing without compromising sensitive information, and providing an audit-friendly alternative to data masking that preserves analytical properties.

Solution overview

Architecting a synthetic data generation system that scales from terabytes to petabytes requires balancing several competing demands: the system must scale horizontally while maintaining data quality, generate large volumes efficiently, manage compute and storage resources cost-effectively, and support various schemas and output formats.

Our architecture addresses these challenges through four core components. Apache Spark on Amazon EMR provides the distributed computing framework necessary for large-scale generation. The Faker library offers synthetic data generation functions that integrate with Spark. Amazon Simple Storage Service (Amazon S3) with Apache Iceberg serves as the storage layer. We chose Iceberg for its schema and partition evolution capabilities without data rewrites, atomic transactions for consistency, precise time travel features for reproducible testing, and optimized performance at extreme scale. Amazon EMR handles dynamic resource allocation and cluster management.

The following diagram illustrates the solution architecture.

Solution architecture showing Amazon EMR generating synthetic data with Apache Spark and Faker, storing results in Amazon S3 with Apache Iceberg

Synthetic data generation at scale with Amazon EMR

Amazon EMR emerges as a particularly powerful solution for this use case, offering several advantages that directly address our requirements. It facilitates scaling of compute resources through instance fleets and Spot Instances, which can reduce costs by up to 90% compared to On-Demand pricing. The service provides built-in performance optimization for Spark applications with real-time monitoring through Amazon CloudWatch integration.

The managed infrastructure reduces operational overhead by handling the underlying Spark ecosystem and cluster lifecycle, while still providing control over scaling policies, instance types, and configurations. Integration with Amazon S3, AWS Glue, and Amazon Athena facilitates end-to-end data generation and testing workflows. Support for multiple programming languages and notebooks provides flexibility in implementing generation logic tailored to specific testing scenarios.

The synthetic data generation process follows a systematic approach designed for efficiency and scalability, as illustrated in the following diagram.

Synthetic data generation workflow showing the systematic process from configuration through data generation to storage

Although synthetic data generation isn’t a sensitive workload, it’s important to maintain robust security throughout the data generation process. Amazon EMR provides security features that align with organizational compliance requirements.

For comprehensive security guidance specific to Amazon EMR deployments, refer to Security in Amazon EMR. The solution follows the AWS Shared Responsibility Model, where AWS manages the security of the cloud infrastructure, and customers maintain responsibility for data security, access management, and compliance controls in the cloud. Specifically for synthetic data generation workloads, AWS manages the security of the underlying Amazon EMR infrastructure, network, and service operations, and customers implement appropriate security controls for their data generation pipelines. Consider the following key areas:

  • Data protection – Enable encryption at rest and in transit using Amazon EMR security configurations, including Amazon S3 encryption and TLS certificates for inter-node communication.
  • Network security – Deploy Amazon EMR clusters in private subnets with security groups following least privilege, and enable the Amazon EMR block public access feature.
  • Access control – Implement AWS Identity and Access Management (IAM) roles with least privilege for Amazon EMR service roles, Amazon Elastic Compute Cloud (Amazon EC2) instance profiles, and runtime roles to isolate job access. Fine-grained table-level and column-level permissions can be controlled using AWS Lake Formation. Additional authentication options are available using Kerberos and LDAP.

Optimize Faker for petabyte-scale data generation

When generating synthetic data at petabyte scale, using Faker’s implementations can quickly lead to performance bottlenecks. To overcome these limitations, adopt a combination of different optimization approaches instead of the default setup. Some of the approaches we adopted in this scenario are discussed in this section.

Faker instance pooling

The following code creates multiple Faker instances to avoid contention when generating data in parallel:

NUM_FAKER_INSTANCES = 10
faker_pool = [Faker() for _ in range(NUM_FAKER_INSTANCES)]

Consistent seed management

The following code provides reproducible data generation across distributed executors:

for faker in faker_pool:
    faker.seed_instance(42)  # For reproducibility
    random.seed(42)

Random access to Faker pool

The following code distributes load across multiple Faker instances to reduce contention:

faker = faker_pool[random.randint(0, NUM_FAKER_INSTANCES-1)]

Broadcast variables for reference data

The following code efficiently distributes reference data to all executors:

tenant_ids_broadcast = spark.sparkContext.broadcast(tenant_ids)
protocols_bc = spark.sparkContext.broadcast(protocols)

Batch generation of synthetic data

The following code generates fake data in batches rather than one-by-one:

return spark.range(1, num_endpoints + 1)
    .withColumn("hostname", random_hostname_udf())

ThreadPoolExecutor for parallel processing

The following code uses Python’s threading for parallel operations within executors:

def parallel_write_with_sync(dataframe_configs, max_workers=3):
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        # Parallel processing

Optimize Amazon EMR and Spark

When processing massive datasets with Spark on Amazon EMR, carefully tuning configurations can substantially enhance performance beyond the standard settings. In this section, we discuss ways to optimize the execution environment, so you can efficiently handle petabyte-scale workloads with synthetic data generation. By strategically using Spark’s advanced features and configuring Amazon EMR for your specific use case, you can improve throughput, reduce processing time, and maximize resource utilization.

Arrow configuration

The following code enables Apache Arrow for efficient data transfer between Python and JVM. The default value is false.

.config("spark.sql.execution.arrow.pyspark.enabled", "true")

Enable this configuration when your PySpark application frequently converts data between Python and JVM, especially for large DataFrames or when using Pandas operations. Keep this setting disabled for pure Spark SQL workloads or when memory is constrained.

This optimization is most effective in the following scenarios:

  • When processing large-scale datasets that require frequent conversion between Python and JVM.
  • In a PySpark application where large DataFrame operations and Pandas integration are needed.
  • With data science workloads that combine Python UDFs with Spark SQL operations.

Consider the following trade-offs:

  • Arrow maintains in-memory columnar format, resulting in increased memory consumption.
  • Not all data types are fully supported in older versions of Spark.
  • It might introduce overhead for very small datasets where conversion costs outweigh the benefits.

Adaptive query execution

The following code allows Spark to dynamically optimize query execution plans. The default value is true in Spark 3.2 and later, and false in earlier versions.

.config("spark.sql.adaptive.enabled", "true")

This optimization is generally recommended to keep enabled for most workloads. Consider disabling only when you have highly optimized, predictable queries where the adaptive overhead isn’t beneficial, or when troubleshooting query performance issues.

This optimization is most effective in the following scenarios:

  • Complex join operations with unknown or skewed data distributions.
  • Multi-stage queries where initial plans might be suboptimal.
  • When processing data with changing characteristics over time.

Consider the following trade-offs:

  • You may experience additional overhead during the query planning phase.
  • You might occasionally choose suboptimal plans for certain edge cases.

Parallelism configuration

The following code sets appropriate parallelism for distributed data processing based on the volume of data you’re generating. The default value for spark.default.parallelism is the total number of cores on all executor nodes or 2, whichever larger. The default value for spark.sql.shuffle.partitions is 200.

.config("spark.default.parallelism", 1000)
.config("spark.sql.shuffle.partitions", 1000)

Adjust this configuration when the default of 200 shuffle partitions creates too many small tasks (increase data volume) or too few large tasks (decrease for smaller datasets). Generally, aim for partition sizes of 100–200 MB. Modify default.parallelism when your RDD operations need different parallelism than the CPU-based default.

This optimization is most effective in the following scenarios:

  • When generating consistent volumes of synthetic data across multiple runs.
  • When you have predictable resource requirements.
  • When you need to precisely control executor utilization.

Consider the following trade-offs:

  • Static configuration might not adapt well to varying data volumes.
  • Too many partitions can lead to task scheduling overhead.
  • Too few partitions might cause memory pressure on executors.

Memory management

The following code optimizes memory allocation for execution and storage. The default value for spark.memory.fraction is 0.6, and for spark.memory.storageFraction is 0.5.

.config("spark.memory.fraction", 0.8)
.config("spark.memory.storageFraction", 0.3)

Increase memory.fraction from 0.6 to 0.8 when your workload is memory-intensive and you’re not using the JVM heap for other purposes. Adjust storageFraction based on your caching vs. execution memory needs. Decrease to 0.3 if you do minimal caching but have complex computations, and increase to 0.7 or higher for cache-heavy workloads.

This optimization is most effective in the following scenarios:

  • Workloads that are memory-intensive and need fine-grained control.
  • Workloads that balance between execution memory and cached data.
  • During synthetic data generation that has many interdependent fields.

Consider the following trade-offs:

  • Incorrect memory configuration can lead to frequent spills to disk or out-of-memory (OOM) errors.
  • You might need to change the configuration to suit different workload characteristics.
  • The settings must be monitored and tuned for optimal performance.

Limited Python UDF usage

The following code uses Spark’s built-in functions where possible instead of Python user-defined functions (UDFs). No additional configuration is needed. This is a coding practice.

.withColumn("risk_score", F.round(F.rand() * 9 + 1, 2).cast(DecimalType(3, 2)))

We recommend using Spark functions over Python UDFs when the same functionality can be achieved. Use Python UDFs only when complex business logic can’t be expressed using Spark’s built-in functions, or when integrating with specialized Python libraries.

This optimization is most effective in the following scenarios:

  • Simple transformations that can be performed using Spark functions.
  • High-throughput workloads where serialization overhead needs to be minimized.

Consider the following trade-offs:

  • This approach is less flexible compared to customer Python-based transformations or functions.
  • You might need to use complex expressions to accomplish certain data patterns.
  • There is a potential learning curve to familiarize yourself with Spark functions.

DataFrame caching

The following code caches frequently used DataFrames to avoid regenerating data. The default behavior doesn’t use caching. DataFrames are recomputed on each action.

endpoints_df = generate_endpoints().cache()

Use this optimization to cache DataFrames that are accessed multiple times in your application. Monitor memory usage and use MEMORY_AND_DISK storage level for large DataFrames. Uncache DataFrames when they’re no longer needed to free memory.

This optimization is most effective in the following scenarios:

  • When reusing reference data across multiple operations (can result in performance gains).
  • For workloads where the same data is processed on multiple occasions.

Consider the following trade-offs:

  • Too much caching might lead to memory process.
  • Planning is required to manage cache in environments where memory is scarce.

Optimal partitioning

By default, Spark determines partitioning based on input data and previous operations. The following code makes sure data is properly distributed across executors:

.repartition(20)

Use repartition() when you need to increase partitions for better parallelism or support even data distribution. Use coalesce() when reducing partitions to avoid small files. Generally, target 100–200 MB per partition for optimal performance.

This optimization is most effective in the following scenarios:

  • When controlling data distribution and avoiding data skew is very important.
  • Before executing an expensive operation that will benefit from balanced data distribution.
  • When optimizing downstream consumption use cases.

Consider the following trade-offs:

  • This option is more expensive than coalesce(). For large datasets, repartition() can lead to large shuffle.
  • The approach requires trial and experimentation to determine the optimal partition count.
  • There is no “one-size-fits-all” setting. Different applications or operations might gain performance with different partitioning.

Partition-aware writing

By default, data is written without partitioning. The following code organizes data for efficient storage and retrieval:

{"df": network_events_df, "name": "network_events", "partition_cols": ["tenant_id"]}

Partition data when you have predictable query patterns that filter on specific columns. Choose partition columns that are frequently used in WHERE clauses and have reasonable cardinality (avoid too many small partitions or too few large ones).

This optimization offers the following benefits:

  • Allows for highly parallel write operation across multiple executors.
  • Organizes the data that is close to real-world production data.
  • Allows for partition pruning when querying the data.

Consider the following trade-offs:

  • Excess partitioning or too fine-grained partitioning might result in small files.
  • It might result in data skew because of hot partitions.
  • You might encounter storage and metadata overhead because of excessive partitions.

Best practices

Through our journey from terabytes to petabytes, we’ve identified several best practices:

  • Begin with a modest dataset and incrementally scale, allowing for identification of bottlenecks at each stage.
  • Implement robust data validation checks to confirm synthetic data maintains expected properties at scale.
  • Regularly review and adjust Amazon EMR configurations, using Spot Instances and right-sizing clusters.
  • Develop parameterized job scripts that can adjust data volume, complexity, and cluster resources dynamically.
  • Design your synthetic data schema and generation logic to quickly accommodate new fields or changing distributions over time.

Conclusion

Our journey from terabytes to petabytes of synthetic data generation demonstrates how Amazon EMR, combined with Spark and Faker, can effectively address large-scale testing needs. The architecture we explored in this post scales to meet demanding data generation requirements while maintaining data quality and cost-efficiency.

We showed how starting with a solid foundation at terabyte scale, then gradually expanding through Amazon EMR managed services and Spot Instances, helps organizations build robust synthetic data pipelines. The combination of efficient data generation techniques, proper validation, and continuous monitoring provides reliable results at scale.

To begin implementing your own synthetic data generation system, start small, test thoroughly, and scale incrementally. For implementation guidance, refer to Generate production-grade synthetic data at petabyte-scale using Apache Spark and Faker on Amazon EMR.


About the authors

Anubhav Awasthi

Anubhav Awasthi

Anubhav is a Senior Big Data Specialist Solutions Architect at Amazon Web Services (AWS). He collaborates with customers to provide expert architectural guidance for implementing and optimizing analytics solutions using Amazon EMR, Amazon Athena, AWS Glue, and AWS Lake Formation.

Gagan Brahmi

Gagan Brahmi

Gagan is a Specialist Senior Solutions Architect at Amazon Web Services (AWS), focused on Data Analytics and AI/ML. With over 20 years in information technology, he partners with customers to solve complex AI/ML challenges by leveraging data and AI/ML platforms. Gagan helps customers architect scalable, high-performance solutions that utilize distributed data processing, real-time streaming technologies, and AI/ML services to drive business transformation through artificial intelligence and data-driven insights. When not designing cloud-native data and AI solutions, Gagan enjoys exploring new places with his family.

Jayaprakash Boreddy

Jayaprakash Boreddy

Jayaprakash is a Senior Solutions Architect at AWS. He works with ISV customers in designing and building highly scalable, flexible and resilient applications on AWS Cloud.

Sahil Thapar

Sahil Thapar

Sahil is a Principal Solutions Architect. He works with ISV customers to help them build highly available, scalable, and resilient applications on the AWS Cloud.

Regional routing for AWS access portals: Implementing custom vanity domains for IAM Identity Center

Post Syndicated from Georgi Baghdasaryan original https://aws.amazon.com/blogs/security/regional-routing-for-aws-access-portals-implementing-custom-vanity-domains-for-iam-identity-center/

AWS IAM Identity Center provides a web-based access portal that gives your workforce a single place to view their AWS accounts and applications. With the recent launch of IAM Identity Center multi-Region replication, customers can replicate their IAM Identity Center instance across multiple AWS Regions to improve resilience and reduce latency for a globally distributed workforce. As a result, users have a dedicated access portal URL in each Region where Identity Center is replicated, and where administrators need a consistent way to manage these portals to ensure that each user reaches the right one.

This post walks you through building a custom vanity domain (for example, aws.mycompany.com) that serves as a single, memorable entry point for access to IAM Identity Center through the AWS Management Console. The solution uses latency-based routing to automatically redirect users to their nearest healthy access portal endpoint and provides a mechanism to trigger failovers when a Regional Identity Center instance, or the broader AWS Region, is impaired. Because this solution operates outside of Identity Center—at the DNS and load balancer layer—users are transparently redirected to the appropriate Regional access portal URL. Note that the vanity domain itself will not appear in the browser’s address bar.

This guide is structured in three progressive phases: a single-Region redirect, multi-Region latency routing, and automatic health-based failover. You can adopt each phase independently, depending on your organization’s needs.

Note: While this guide focuses on IAM Identity Center access portal endpoints, the same approach using Amazon Route 53 latency-based routing, Application Load Balancer (ALB) redirects, and Amazon Application Recovery Controller (ARC) Region switch can be applied to build a custom vanity domain and intelligent routing layer for any other HTTP endpoint type.

Background

IAM Identity Center supports multiple access portal URL formats that resolve to the same web portal. The following table summarizes the supported formats in the standard AWS (classic) partition, along with their capabilities:

Format IPv4 Dual-stack Multi-Region* Example
https://{directoryId}.awsapps.com/start Yes No No https://d-1234567890.awsapps.com/start
https://{alias}.awsapps.com/start Yes No No https://mycompany.awsapps.com/start
https://{idcInstanceId}.{region}.portal.amazonaws.com Yes No Yes https://ssoins-1234567890.us-west-2.portal.amazonaws.com
https://{idcInstanceId}.portal.{region}.app.aws ★ Yes Yes Yes https://ssoins-1234567890.portal.us-west-2.app.aws

* Each Regional URL resolves only to its own Region’s portal instance and doesn’t fail over to another Region. Multi-Region here means the URL format is available in every Region where IAM Identity Center is replicated. To route users across Regions dynamically, use the vanity domain approach described in this post.

Note: The ★ highlighted row (https://{idcInstanceId}.portal.{region}.app.aws) is the recommended URL format. It supports both dual-stack (IPv4 and IPv6) and IAM Identity Center multi-Region replication. The awsapps.com formats aren’t always available in newer Regions and don’t support multi-Region capabilities. In additional replicated Regions, the custom alias isn’t supported, and the awsapps.com parent domain isn’t available.

Working with multiple Regional endpoints

As you expand your IAM Identity Center footprint through multi-Region replication, each replicated Region provides a dedicated access portal URL—directing your users to the low-latency entry point closest to their location. A user connecting from Europe and one connecting from Asia Pacific each benefit from their respective Regional endpoint. To deliver the best experience, organizations need a consistent, centrally managed way to direct users to the correct Regional destination; there are a few common approaches you can use to achieve this.

Customers typically start with a single Regional endpoint, which is straightforward to configure, but users in distant Regions experience higher latency, and a Regional incident can affect all users regardless of location. Others maintain per-Region bookmarks or configuration, which gives each user population the right endpoint but requires ongoing IT coordination and clear communication to users.

Custom vanity domains give you full control over DNS routing, health checks, and failover of your access portal connections; all behind a single, brand-aligned domain name (for example, aws.mycompany.com) that users access. A vanity domain makes this start URL memorable and consistent for users, regardless of the underlying IAM Identity Center configuration – a single address to remember and share, compared to maintaining a separate bookmark for each Regional endpoint or managing a growing list of application tiles in your external identity provider. The rest of this guide walks you through how to deploy this solution step by step.

Solution overview

The solution builds a lightweight routing and redirect layer in front of the IAM Identity Center access portal Regional endpoints. The architecture has the following components:

  • AWS IAM Identity Center – Your existing Identity Center instance
  • Amazon Route 53 – Manages your vanity domain’s hosted zone, latency-based routing policy, and health checks
  • AWS Certificate Manager (ACM) – Issues and automatically renews TLS certificates for your vanity domain in each Region
  • Application Load Balancer (ALB) – Handles HTTP and HTTPS traffic, issuing 302 redirects to the appropriate Regional access portal endpoint
  • Amazon Application Recovery Controller (ARC) Region switch – Orchestrates Regional failovers by controlling Route 53 health check states, so traffic is automatically shifted away from an unhealthy Region

This guide is structured in three progressive phases. You can adopt each phase incrementally based on your needs:

  • Phase 1: Sets up the vanity domain with a redirect to a single Regional access portal endpoint. Suitable for organizations with a single-Region Identity Center deployment.
  • Phase 2: Extends Phase 1 across multiple Regions with latency-based routing, so users are automatically directed to the nearest Regional endpoint. Requires IAM Identity Center multi-Region replication.
  • Phase 3: Adds an ARC Region switch for managed Regional failover. Without Phase 3, a Regional impairment requires manual DNS updates to redirect traffic. ARC automates this with rehearsable, controlled failover plans.

Figure 1: Solution architecture for custom vanity domain routing with IAM Identity Center.

When a user navigates to aws.mycompany.com, the following happens:

  1. Route 53 evaluates the latency records and routes traffic to the ALB in the lowest-latency healthy Region.
  2. The ALB terminates TLS using an ACM-managed certificate and issues a 302 redirect to the corresponding Regional Identity Center access portal URL.
  3. The user’s browser follows the redirect and loads the access portal directly. Subsequent authentication traffic flows between the browser and AWS—the ALB isn’t in the path.

If you’ve implemented Phase 3, ARC controls Route 53 health check states for each Region. With this configuration, you can stop routing traffic to any Region considered unhealthy.

Prerequisites

Before you begin to build the solution, ensure you have the following in place:

  1. An existing top-level domain (TLD) (for example, mycompany.com).
  2. An AWS IAM Identity Center organization instance configured.
  3. For Phases 2 and 3, you need IAM Identity Center multi-Region replication configured with at least two Regions. See Setting up IAM Identity Center multi-Region replication for instructions.
  4. AWS Identity and Access Management (IAM) permissions on a dedicated networking or shared services account in your organization to manage Route 53, ACM, Amazon Elastic Compute Cloud (Amazon EC2), ALB (phase 1 and 2), and ARC (phase 3).

Phase 1: Redirect to a single predefined access portal endpoint

In this phase, you create the foundational infrastructure: a Route 53 hosted zone, an ACM-managed TLS certificate, and an internet-facing ALB that issues a 302 redirect to your Regional access portal URL. By the end, users who navigate to aws.mycompany.com will be seamlessly redirected to your Identity Center portal.

Create a Route 53 hosted zone for your vanity domain

The hosted zone holds the DNS records that control how aws.mycompany.com resolves. If your top-level domain (mycompany.com) is already registered in Route 53, you create a subdomain hosted zone. If it’s registered with another registrar, you create a public hosted zone and configure name server (NS) delegation manually.

  1. In the AWS Management Console, navigate to Route 53 and choose Hosted zones, then Create hosted zone.
  2. Enter your vanity domain in the Domain name field (for example, aws.mycompany.com).
  3. Select Public hosted zone as the type, then choose Create hosted zone.
  4. Note the four NS records that Route 53 creates for the new hosted zone. You will need these in the next step.

Figure 2: Route 53 hosted zone details

Delegate your subdomain from the parent domain

To make Route 53 authoritative for aws.mycompany.com, you must add an NS record in the parent zone (mycompany.com) pointing to the name servers of the new hosted zone.

  • If mycompany.com is hosted in Route 53: Open the mycompany.com hosted zone, choose Create record, set the record name to aws, the type to NS, and paste the four NS values from the previous step. Choose Create records.
  • If mycompany.com is hosted elsewhere: Sign in to your registrar’s DNS management console and add an NS record for aws.mycompany.com using the four name server values from the previous step.

Note: DNS propagation for NS delegation can take up to 48 hours, though it typically completes within a few minutes for Route 53-to-Route 53 delegation.

Figure 3: Create a NS record type to delegate your subdomain from the parent domain

Request an ACM certificate

Your ALB requires a TLS certificate for aws.mycompany.com to serve HTTPS traffic. ACM provides free public certificates with automatic renewal.

  1. Go to the Certificate Manager console in the primary Region of IAM Identity Center (for example, us-east-2) and choose Request a certificate.
  2. Select Request a public certificate and choose Next.
  3. Enter your domain name (for example, aws.mycompany.com). Choose Add another name to this certificate and enter your Regional sub-domain (for example, us-east-2.aws.mycompany.com).
  4. Leave other options as defaults (Disable export, DNS validation – recommended, and key algorithm – RSA 2048) and choose Request.
  5. In the certificate details page, choose Create records in Route 53. ACM will automatically add the validation CNAME records to your hosted zone. The certificate status changes to Issued within a few minutes.

Figure 4: Request an ACM certificate for your domain

Create a security group for Identity Center ALB

The security group needs to allow inbound HTTP and HTTPS traffic for both IPv4 and IPv6 from the public internet to make the load balancer reachable.

  1. Go to the Amazon EC2 console, navigate to Security Groups, and choose Create security group.
  2. Enter a Name (for example, identitycenter-global-domain-alb-sg-us-east-2) and Description. Add four rules by choosing Add Rule under Inbound Rules.
    1. Set Type to HTTP, and Source to Anywhere-IPv4 (0.0.0.0/0) and to Anywhere-IPv6 (::/0).
    2. Set Type to HTTPS, and Source to Anywhere-IPv4 (0.0.0.0/0) and to Anywhere-IPv6 (::/0).
  3. Choose Add Rule under Outbound Rules and set Type to All traffic and Source to Anywhere-IPv6 (::/0).
  4. Choose Create security group.

Figure 5: ALB security group rules

Create an ALB with an HTTP and HTTPS redirect rule

The ALB is the component that performs the actual redirect to your IAM Identity Center access portal URL. The ALB listener accepts HTTPS requests on port 443 and responds with a 302 redirect to the appropriate Regional Identity Center access portal endpoint.

  1. Go to the Amazon EC2 console, navigate to Load Balancers, and choose Create load balancer. Select Application Load Balancer.
  2. Enter a name for your ALB (for example, identitycenter-redirect-alb).
  3. Configure basic settings: Set the scheme to Internet-facing, IP address type to Dualstack (or IPv4 if IPv6 isn’t supported by your virtual private cloud (VPC)), and select at least two Availability Zones. Ensure that the load balancer is operating in a VPC and subnets that are internet-facing.
  4. Under Security Groups choose the Security Group created in the previous step.
  5. Configure an HTTP listener: Add a listener on port 80 (HTTP) with Redirect to URL option. Choose URL parts and set Protocol to HTTPS, Port to 443, and status code to 302 (Found).

    Figure 6: Add an HTTP listener during ALB creation

  6. Configure an HTTPS listener: Add a listener on port 443 (HTTPS) with No pre-routing action (default) and Redirect to URL options. Choose Full URL and set the URL to your Regional Identity Center access portal endpoint (For example, https://ssoins-1234567890.portal.<your-region>.app.aws, for this blog the region is us-east-1). Set status code to 302 (Found).

    Figure 7: Add an HTTPS listener

  7. Under Default SSL/TLS certificate, select the ACM certificate you created in Step 3.

    Note: Make sure to select 302 – Found as the Status code. Selecting 301 – Permanently moved will result in browser caching the redirect URL which will prevent failovers from working correctly until the cache expires.

Create Regional Route 53 records pointing to your ALB

Create a DNS record in your hosted zone that resolves <your-region>.aws.mycompany.com to your ALB.

  1. Open your Route 53 hosted zone for aws.mycompany.com and choose Create record.
  2. Set the record name to the AWS Region name (For example: us-east-2) and the record type to A.
  3. Toggle Alias and in the drop down menu Route traffic to, select the alias target to Alias to Application and Classic Load Balancer, select your Region (For example:us-east-2), and select your ALB from the dropdown list.
  4. Leave routing policy as Simple routing, and select the Region (For example:us-east-2) and choose Create records.
  5. Repeat steps 1 through 4 to create AAAA record types.

Figure 8: Route 53 record with simple routing policy

Add latency-based routing configurations

Finally, create a DNS record in your hosted zone that resolves aws.mycompany.com to your Regional Route 53 record.

  1. Open your Route 53 hosted zone for aws.mycompany.com and choose Create record.
  2. Keep the subdomain name for this record as empty, so aws.mycompany.com is the fully qualified record and set the record type to A.
  3. Enable alias: Set the Route traffic to Alias to another record in this hosted zone, and select the hosted zone you created earlier (us-east-2.aws.mycompany.com).
  4. Set Routing Policy to Latency and select the corresponding Region (us-east-2 in this example).
  5. Add a clear name for the Record ID, such as us-east-2--ipv4 as a differentiator and choose Create records.
  6. Repeat the steps 1 through 5 to create AAAA record types with us-east-2--ipv6 as the record ID.
Figure 9: Route 53 record with latency-based routing

Figure 9: Route 53 record with latency-based routing

Test the configuration by navigating to https://aws.mycompany.com in a browser. You should be redirected to your Identity Center access portal. You can also validate using:
curl -I https://aws.mycompany.com

Expected response:

HTTP/2 302

location: https://ssoins-1234567890.portal.<your-region>.app.aws

Tip: To deploy Phase 1 automatically, download the CloudFormation template from the Deploying with CloudFormation section below.

Phase 2: Automatically route to the nearest Regional access portal endpoint

Phase 2 extends the solution to support IAM Identity Center multi-Region replication by deploying an ALB in each replicated Region and configuring Route 53 latency-based routing. Users are automatically directed to the access portal in the Region that has the lowest network latency from their location, which matches the active-active behavior of the Identity Center access portal itself.

Request ACM certificates in each additional Region

Repeat the steps from Request an ACM Certificate for each additional Region (for example, us-west-2) where you’ve replicated IAM Identity Center.

Create a security group and an ALB in each additional Region

Repeat the steps from Create a security group for Identity Center ALB and Create an ALB with an HTTP and HTTPS redirect rule in each additional Region. In each ALB’s redirect rule, set the target URL to the access portal endpoint for that specific Region. For example:

  • us-east-2 ALB redirects to https://ssoins-1234567890.portal.us-east-2.app.aws
  • us-west-2 ALB redirects to https://ssoins-1234567890.portal.us.west-2.app.aws

Create Regional and latency Route 53 records for the additional Region

For each additional Region where you’ve deployed an ALB and replicated Identity Center, create Regional and latency A and AAAA records as outlined in Create Regional Route 53 records pointing to your ALB and Add latency-based routing configurations.

Tip: To deploy Phase 2 automatically, download the CloudFormation template from the following Deploying with CloudFormation section.

Phase 3: Regional failover using ARC Region switch

Phase 3 introduces Amazon Application Recovery Controller (ARC) Region switch, a fully managed capability that you can use to plan, practice, and orchestrate Regional failovers with confidence. ARC Region switch vends Route 53 health checks directly as part of a Region switch plan. You attach these generated health checks to your Route 53 latency records, and ARC controls their healthy or unhealthy state during plan execution. You can further extend the solution to include custom automation triggered by Amazon CloudWatch alarms or synthetic canaries to update routing control state.

We recommend creating your ARC Region switch plan in the primary Region of your IAM Identity Center for ease of discovery.

Create an active-active instance of ARC Region switch plan

Create an ARC Region switch plan that will orchestrate failovers between your IAM Identity Center Regions and auto-generate the Route 53 health checks you will reference in the next step.

  1. Open the Application Recovery Controller console and choose Region switch in the navigation pane. Select Create Region Switch Plan.
  2. Enter a Plan name (for example, idc-access-portal-failover) and an optional description. Choose Active/Active for Multi-Region recovery approach. Select the Regions where IAM Identity Center is replicated ,including the primary Region.
  3. In the Execution Permission section, enter the Amazon Resource Name (ARN) of the IAM role that ARC will use to update Route 53 health check states during plan execution. If you don’t have an existing role, choose Create a new role to have ARC create one automatically. See AWS Managed Policy: AmazonApplicationRecoveryControllerRegionSwitchPlanExecutionPolicy for information about required permissions.
  4. Choose Create Plan and proceed to Build workflows. Enter optional descriptions and choose Save and continue.

    Figure 10: Region switch plan

  5. Set the Workflow type to Activate and set the Region to the corresponding Region (us-east-2 or us-west-2). Within each workflow, choose Add step/Run in Sequence. Choose an execution block to Amazon Route 53 health check execution blog under Networking.
  6. Choose Add and edit. Enter a Step name (for example, Activate Route53 Record Set).
  7. Set the Hosted zone to the hosted zone ID for your aws.mycompany.com domain, and set the Record name to aws.mycompany.com.
  8. Expand Record set identifiers. Choose Add record set identifier and enter a unique identifier for the record set (for example, us-east-2--ipv4 and us-east2--ipv6) and select your Region. Add two record set identifiers (A and AAAA records) for each of your Regions.
  9. Choose Save step.
  10. Repeat steps 5 and 6 for Deactivate and choose Save the plan.

    Figure 11: Workflow builder

  11. Choose Save workflows.
  12. Select the newly created plan and choose the Monitoring tab. Note the IDs of the health checks created.

    Figure 12: IAM Identity Center access portal plan

Update Route 53 record sets to reference ARC-managed health checks

Associate the ARC-generated health check IDs with the latency-based A and AAAA records you created in Phase 1 and 2. Route 53 uses these health checks—which are now controlled by ARC—to determine which Regions are eligible for DNS resolution. Route 53 still uses latency to choose from the healthy Regions.

    1. Go to the Route 53 console and choose Hosted zones.
    2. Select the hosted zone for aws.mycompany.com.
    3. Find the latency-based A record for us-east-2 that you created in Phase 2, and choose Edit record.
    4. In the Health check section, enable Associate with a health check. In the Health check ID dropdown, select the ARC-generated health check for us-east-2 that you noted at the end of the preceding procedure. Note: Ignore the warning This health check ID doesn’t belong to this AWS account. Make sure you have copied it accurately to use it.
    5. Choose Save changes.
    6. Repeat steps 3, 4, and 5 for A and AAAA records for each of your IAM Identity Center Regions.

Figure 13: Update Route53 record sets

Validate the setup by performing a failover

Validate the end-to-end configuration by executing a controlled failover. Because latency-based routing will always resolve aws.mycompany.com to us-east-2 for users in the primary geography, deactivating us-east-2 is the most direct way to confirm that Route 53 correctly fails over to us-west-2.

    1. Before executing the failover, confirm that aws.mycompany.com is resolving to the us-east-2:
      curl -I https://aws.mycompany.com
      Expected: A record pointing to the us-east-2 access portal URL (for example, https://ssoins-1234567890.portal.us-east-2.app.aws:443/).
    2. Go to the Amazon Application Recovery Controller console. In the left navigation pane, choose Region switch.
    3. Select your Region switch plan (idc-access-portal-failover) to open the plan details page.
    4. Choose Execute recovery.
    5. On the Execute plan page, select us-east-2 as the Region to fail out of.
    6. Select the Deactivate action and choose Start execution. ARC sets the us-east-2 health check to unhealthy. Route 53 stops resolving aws.mycompany.com to the us-east-2 ALB and routes traffic to us-west-2 instead.
    7. After a few seconds, confirm the failover has taken effect:
      curl -I https://aws.mycompany.com
      Expected: 302 redirect to the us-west-2 IAM Identity Center access portal URL
    8. To fail back, choose Execute plan again. Select us-east-2, select the Activate action and choose Start execution. ARC marks the us-east-2 health check healthy and Route 53 resumes routing traffic to that Region.

Tip: To deploy Phase 3 automatically, download the CloudFormation template from the Deploying with CloudFormation section that follows.

Deploying with CloudFormation

As an alternative to the manual console steps described previously, we provide CloudFormation templates that you can download and deploy for each phase. Each template is self-contained and parameterized, so you only need to provide your environment-specific values (such as your vanity domain name, VPC, and subnet IDs). Download the templates from the following links:

To deploy a template, navigate to the AWS CloudFormation console, choose Create stack, select Upload a template file, and upload the downloaded YAML file. Follow the prompts to provide parameter values and create the stack. For Phase 2, deploy the template once in each additional Region.

Deploy all phases with a single scrip

As an alternative to deploying each CloudFormation template individually, you can use the provided deploy.sh bash script to deploy all three phases in sequence. The script automates stack creation across your primary and additional Region. To get started, download the deployment package, then unzip the file into a local directory:

wget https://aws-security-blog-content.s3.us-east-1.amazonaws.com/public/sample/3536-regional-routing-for-aws-access-portals/Vanity-domains-cfn.zip
unzip  Vanity-domains-cfn.zip
cd Vanity-domains-cfn

Before running the script, open the deploy.sh file and update the following required parameters with your environment-specific values:

  • TLD – Your top-level domain (for example, mycompany.com)
  • TLD_HOSTED_ZONE_ID – The Route 53 hosted zone ID for your top-level domain
  • IDC_SUBDOMAIN – The Identity Center subdomain name (for example, aws)
  • IDC_INSTANCE_ID – Your IAM Identity Center instance ID (for example, ssoins-1234567890)
  • PRIMARY_REGION – The primary Region for your Identity Center instance (for example, us-east-2)
  • ADDITIONAL_REGIONS – The additional Region for multi-Region replication (for example, us-west-2)

After updating the configuration, run the deployment script:

./deploy.sh

The script deploys Phase 1 (single-Region redirect), Phase 2 (multi-Region latency-based routing), and Phase 3 (ARC Region switch failover) in order. Monitor the terminal output for stack creation progress and any errors.

After completing the setup, you can integrate the vanity URL (for example, aws.mycompany.com) directly into your identity provider, such as Okta or Microsoft Entra ID, as a bookmark application or a chiclet URL. By configuring the vanity URL as the bookmark target, users who launch the application from their identity provider dashboard are always redirected to the nearest IAM Identity Center access portal endpoint through latency-based routing. If a Regional impairment occurs and a failover is necessary, administrators can execute an ARC Region switch to deactivate the impaired Region, and users will automatically be redirected to the active Identity Center endpoint without any change to the bookmark URL or end-user experience.

Conclusion

In this post, you learned how to build a custom vanity domain for an AWS IAM Identity Center access portal using Amazon Route 53, AWS Certificate Manager, Application Load Balancer, and an Amazon Application Recovery Controller (ARC) Region switch. The three-phase approach lets you start with a single-Region redirect, progressively add latency-based routing as your IAM Identity Center footprint grows with multi-Region replication, and then introduce an ARC Region switch to gain fully managed, rehearsable Regional failover.

For more information about IAM Identity Center multi-Region replication, see the IAM Identity Center User Guide. For more resilience patterns, visit the AWS Architecture Blog posts about Resilience. If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.

Resources


Georgi Baghdasaryan

Georgi Baghdasaryan

Georgi is a Principal Engineer at Amazon Web Services, where he builds identity systems that help organizations securely manage access and authentication at scale. His broader focus is on reliable, high-impact infrastructure that enables customers to operate confidently in the cloud. Outside of work, Georgi enjoys experimenting with new matcha latte recipes and going on long bike rides.

Sowjanya Rajavaram

Sowjanya Rajavaram

Sowjanya is a Sr Solutions Architect who specializes in Identity and Security in AWS. She works on helping customers of all sizes solve their identity and access management problems. She enjoys traveling and exploring new cultures and food.

Author

Laura Reith

Laura is an Identity Solutions Architect at AWS, where she thrives on helping customers overcome security and identity challenges. In her free time, she enjoys wreck diving and traveling around the world.

Automating post-quantum cryptography readiness using AWS Config

Post Syndicated from Pravin Nair original https://aws.amazon.com/blogs/security/automating-post-quantum-cryptography-readiness-using-aws-config/

Migrating your TLS endpoints to Post-quantum cryptography (PQC) starts with understanding your current TLS endpoint inventory and posture. This post introduces the PQC Readiness Scanner — an automated tool that inventories your Application Load Balancer (ALB), Network Load Balancer (NLB), and Amazon API Gateway endpoints and continuously monitors their TLS configurations for PQC readiness. The scanner classifies each endpoint into a three-tier framework that helps prioritize and plan PQC migration.

As quantum computing advances, you need to migrate to quantum-resistant cryptography to protect your data long-term. The PQC Readiness Scanner helps you identify which endpoints to migrate first and tracks your progress across accounts. For web traffic, PQC key exchange algorithms are negotiated only within TLS 1.3. This means quantum-resistant connections require endpoints that support TLS 1.3 and PQC key exchange.

Under the AWS Shared Responsibility Model, AWS secures the infrastructure and enables PQC support across its services. Customers are responsible for configuring their resources to use PQC-capable TLS policies. For AWS-terminated TLS connections—such as those on Application Load Balancer (ALB), Network Load Balancer (NLB), Amazon API Gateway, and Amazon CloudFront—customers choose the security policy (an AWS-managed configuration defining supported TLS protocol versions and cipher suites for a listener) that determines TLS version and cipher suite, key exchange, and authentication algorithm support.

The automated PQC Readiness Scanner for AWS-terminated TLS endpoints is built using AWS Config conformance packs. A conformance pack is a collection of AWS Config rules and remediation actions that can be deployed as a single entity in an account and a Region or across an organization in AWS Organizations.

Solution overview

The PQC Readiness Scanner deploys AWS Config rules using a conformance pack to evaluate the security policy on each endpoint. Based on the evaluation, each resource is classified into a three-tier readiness framework that prioritizes migration actions needed to achieve PQ-ready TLS.

The PQC Readiness Scanner performs two checks per resource:

  1. Does the endpoint use a PQ-ready security policy?
  2. Does the endpoint support legacy TLS 1.0 or 1.1?

Each check returns COMPLIANT or NON_COMPLIANT status with specific policy recommendations.

PQC requires endpoints to support TLS 1.3 and use PQC key exchange algorithms. The three-tier framework helps you interpret findings and prioritize fixes. The goal is to have TLS 1.3 with PQC key exchange enabled on the endpoints. However, achieving this requires maintaining backward compatibility with clients.

Tier

Readiness level

TLS protocols

PQC status

Migration priority

Tier 1

PQ-ready (strongest posture)

TLS 1.3 only with PQC key exchange

PQ-ready

None

Tier 2

PQ-ready (backward compatible)

TLS 1.2 and 1.3 with PQC key exchange

PQ-ready

Low

Tier 3

Not PQ-ready

No PQC key exchange

Not PQ-ready

High

How to prioritize your migrations

  • Tier 1 represents the strongest security using only TLS 1.3 with PQC key exchange. These resources already meet the target state.
  • Tier 2 represents a backward-compatible PQ-ready configuration. Endpoints support both TLS 1.2 and TLS 1.3, with PQC key exchange negotiated on TLS 1.3 connections. Migration priority is low because these resources already provide quantum-resistant protection for clients that support TLS 1.3, while maintaining TLS 1.2 compatibility for legacy clients. Migrate to Tier 1 when client-side analysis confirms that the connecting clients support TLS 1.3 with PQC key exchange.
  • Tier 3 covers resources that aren’t PQ-ready. This includes endpoints without TLS 1.3 support, endpoints with TLS 1.3 but without PQC key exchange policies. These resources require immediate attention.

Assessment scope

The scanner evaluates the following AWS edge services that terminate TLS connections on behalf of your applications.

  • Edge services:
    • Application Load Balancer (ALB), Network Load Balancer (NLB) listeners with HTTPS, TLS, and TCP SSL protocols are evaluated.
    • API Gateway REST APIs are evaluated for AWS Regional and private endpoints along with API Gateway HTTP APIs (v2) and WebSocket APIs (v2).
  • Excluded edge services:
    • CloudFront distributions are excluded from the PQC readiness scope because TLS 1.3 with hybrid post-quantum key exchange is automatically enabled across existing CloudFront TLS security policies for viewer-to-edge connections. No customer action is required for inbound (viewer-facing) PQC on CloudFront.
  • Recommended approach for Classic load balancer:
    • For Classic Load Balancers, AWS recommends migrating to ALB or NLB. Classic Load Balancers don’t support TLS 1.3 or PQC key exchange and can’t be made PQ-ready.

How the solution works

AWS Config enables continuous monitoring and evaluation. Conformance packs enable organization-wide deployment. AWS Lambda is a serverless compute service that runs code to perform security policy evaluation based on the AWS Config rules. AWS Serverless Application Model (AWS SAM) is an open source framework used for deploying the AWS Lambda functions.

Figure 1: PQC readiness solution architecture

Figure 1: PQC readiness solution architecture

The PQC Readiness Scanner conformance pack implements four custom AWS Config rules powered by two Lambda functions:

Rule

What it checks

Non-compliant result

ELB PQ-ready

Load balancer listeners use security policies that support TLS 1.3 with PQC key exchange algorithms

Policy doesn’t include PQC support, the resource is marked with a recommended upgrade policy

ELB legacy TLS

Load balancer listeners allow TLS 1.0 or 1.1 connections

Legacy protocols are configured, the resource is flagged.

API Gateway PQ-ready

API Gateway endpoints use security policies that support TLS 1.3 with PQC key exchange algorithms

Policy doesn’t include PQC support, the resource is marked with a recommended upgrade policy

API Gateway legacy TLS

API Gateway endpoints allow TLS 1.0 or 1.1

Legacy protocols are configured, the resource is flagged.

Prerequisites

Before deploying the solution, you need:

  • AWS Command Line Interface (AWS CLI) configured with appropriate permissions
    aws configure
    aws sts get-caller-identity  # Verify

  • Python 3.12 installed. The Lambda runtime requires this version.
    python3 --version  # Should show 3.12.x

  • AWS SAM CLI installed (Installation Guide)
    pip install aws-sam-cli
    
    # Verify
    sam --version

  • AWS Config enabled in your target AWS Region.
    • Configure it to record (This step is not needed if your accounts are recording all resources by default)
      • AWS::ElasticLoadBalancingV2::LoadBalancer
      • AWS::ApiGateway::RestApi
      • AWS::ApiGatewayV2::Api resource types.
    • Enable via AWS Config Console → Recorder → Recording Strategy → Select specific resource types (Follow the steps in manual setup for AWS Config recording strategy for specific resource types)

Steps to deploy the PQC Readiness Scanner

Deploy the PQC Readiness Config Scanner in three phases. Complete deployment commands and configuration details are available in the GitHub repository. The Lambda functions must be deployed first because the conformance pack references their ARNs as parameters. See the GitHub repository for details.

Deploy to single account:

  1. Clone and Build:
    git clone https://github.com/aws-samples/sample-PQC-Readiness-using-AWS-Config.git
    
    cd sample-PQC-Readiness-using-AWS-Config/installation
    
    sam build

  2. Deploy to One or More Regions:
    # Make script executable (first time only)
    chmod +x deploy-per-regions.sh
    
    # Deploy to a single region
    ./deploy-per-regions.sh us-east-1
    
    # Deploy to multiple regions
    ./deploy-per-regions.sh us-east-1 us-west-2 eu-west-1

    Type y and continue if you have enabled AWS Config recording for these resources or its by default recording all resources.

    Figure 2: Type y and continue if you have enabled AWS Config recording for these resources or its by default recording all resources.

  3. The script automatically:
    • Deploys Lambda functions via SAM
    • Deploys conformance pack (creates Config rules)
    • Verifies deployment success
    • Provides clear status messages

The deployment creates two Lambda functions that perform PQ-ready and legacy TLS checks. It provisions IAM roles with least-privilege permissions for ELB, ALB, NLB, and API Gateway describe operations. Lambda permissions allow AWS Config to invoke the functions.

Example screen-print of how a successful deployment looks like.

Figure 3: Example screen-print of what a successful deployment looks like.

Multi-account deployment (Organizations):

For organization-wide deployment across multiple AWS accounts, use CloudFormation StackSets to deploy Lambda functions to each account.

Important Constraint: AWS Config CUSTOM_LAMBDA rules require the Lambda function to exist in the same account as the Config rule. You cannot use a centralized Lambda in one account to evaluate resources in other accounts.

Prerequisite: Shared S3 Bucket

Before packaging, create an S3 bucket accessible by each target account in your organization. This bucket will host the Lambda deployment artifacts that CloudFormation StackSets pulls into each member account.

# Create the shared S3 bucket (run from management/central account)
aws s3 mb s3://<your-org-shared-bucket> --region us-east-1

Grant read access to the target accounts using one of the following options:

aws s3api put-bucket-policy \
  --bucket <your-org-shared-bucket> \
  --policy '{
    "Statement": [
      {
        "Sid": "BucketOwnerFullAccess",
        "Effect": "Allow",
        "Principal": {
          "AWS": "arn:aws:iam::<bucket-owner-account-id>:root"
        },
        "Action": "s3:*",
        "Resource": [
          "arn:aws:s3:::<your-org-shared-bucket>",
          "arn:aws:s3:::<your-org-shared-bucket>/*"
        ]
      },
      {
        "Sid": "CrossAccountReadAccess",
        "Effect": "Allow",
        "Principal": {
          "AWS": [
            "arn:aws:iam::<account-id-1>:root",
            "arn:aws:iam::<account-id-2>:root"
          ]
        },
        "Action": ["s3:GetObject", "s3:ListBucket"],
        "Resource": [
          "arn:aws:s3:::<your-org-shared-bucket>",
          "arn:aws:s3:::<your-org-shared-bucket>/*"
        ]
      }
    ]
  }'

Replace <account IDs> with the AWS account IDs where StackSets will deploy the Lambda functions.

Note: The bucket must be in the same region as the StackSet deployment regions. For multi-region deployments, create one bucket per region and run sam package separately for each.

Step 1: Build and Upload Lambda Packages to S3

Run the packaging script from the installation/ directory:

cd installation

# Make script executable (first time only)
chmod +x deploy-stacksets.sh

# Build, package, upload to S3, and generate resolved template
./deploy-stacksets.sh <your-org-shared-bucket>

This script automatically:

  • Builds Lambda functions using SAM
  • Creates ZIP packages
  • Uploads ZIPs to the shared S3 bucket
  • Generates packaged-template.yaml with S3 values baked in (no parameters needed at deploy time)
Sample script output of successful upload of the lambda packages to S3 bucket

Figure 4: Sample script output of successful upload of the lambda packages to S3 bucket

Step 2: Deploy Lambda Functions via StackSets

Run the following from the management account (or delegated admin account):

# Create StackSet (--region sets the StackSet "home region" where it is managed)
aws cloudformation create-stack-set \
  --stack-set-name pqc-readiness-lambda-functions \
  --template-body file://packaged-template.yaml \
  --capabilities CAPABILITY_IAM \
  --permission-model SERVICE_MANAGED \
  --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false \
  --region us-east-1

# Deploy stack instances to member accounts
# --regions = target regions where Lambda functions are deployed in member accounts
# --region  = must match the StackSet home region above
aws cloudformation create-stack-instances \
  --stack-set-name pqc-readiness-lambda-functions \
  --deployment-targets OrganizationalUnitIds=ou-xxxx-xxxxxxxx \
  --regions us-east-1 \
  --region us-east-1

Important — StackSet home region vs deployment regions:

  • --region (on each CLI command) = the StackSet home region where the StackSet resource lives. Subsequent operations (describe, update, delete) must specify this same region.
  • --regions (on create-stack-instances) = the deployment target region(s) where stack instances are created in member accounts.
  • These are independent values. Specify --region explicitly to avoid accidental deployment to your CLI’s default region.

Note: SERVICE_MANAGED StackSets must be created from the management or delegated admin account. The management account itself is excluded from stack instance deployments — use deploy-per-regions.sh separately if you need the scanner in the management account.

Step 3: Deploy Organization Conformance Pack

aws configservice put-organization-conformance-pack \
  --organization-conformance-pack-name pqc-legacy-tls-compliance \
  --template-body file://conformance-packs/pqc-legacy-tls-conformance-pack.yaml

This creates Config rules in each member account that reference their local Lambda functions.

    Migration guidance and prioritization

    The three-tier system provides PQC migration priorities:

    High priority – Tier 3 (not PQ-ready):

    • Target: Resources without PQC support. This includes endpoints not using PQ-ready security policies, endpoints that still allow TLS 1.0 or 1.1.
    • Action: Upgrade to a PQ-ready policy containing PQ in its name, such as those ending with -PQ-2025-09 (see Elastic Load Balancing security policies documentation for the full list).
    • Important: Before upgrading to a PQ-ready policy, audit your client TLS versions. PQ-ready policies require TLS 1.3 support; legacy clients that only support TLS 1.2 or earlier will fail to negotiate a connection. Start with a Tier 2 backward-compatible policy (which supports both TLS 1.2 and 1.3 with PQC), monitor connection logs for TLS negotiation failures, and only move to a Tier 1 TLS 1.3-only policy after confirming that your clients support TLS 1.3 with PQC key exchange.
    • Risk: Endpoints don’t support post-quantum cryptography for data in transit. Legacy TLS protocols are vulnerable to current cryptographic attacks.

    Low priority – Tier 2 (PQ-ready, backward compatible):

    • Target: Resources using TLS 1.3 + PQ-ready policies that also support TLS 1.2 for backward compatibility.
    • Action: Consider TLS 1.3-only policies when client compatibility analysis confirms connecting clients support TLS 1.3.
    • Risk: Minimal. These resources already support PQ-TLS with TLS 1.3 connections. TLS 1.2 and earlier fallback maintains backward compatibility, which might indicate some clients aren’t negotiating in PQ-TLS. Remediation is to monitor logs, identify the volume of these connections and clients and plan migration for these clients to use TLS 1.3 with PQ-TLS.

    No action – Tier 1 (PQ-ready, optimal):

    • Target: Resources using TLS 1.3 only with PQC key exchange: These resources meet the target state. No migration needed.

    Viewing the results

    In each member account, navigate to AWS Config Console in the deployed region.

    Conformance Pack View

    Go to AWS Config → Conformance packs and look for:

    OrgConformsPack-pqc-legacy-tls-compliance-

    Note: Organization conformance packs are prefixed with OrgConformsPack- and have a random suffix appended (e.g., OrgConformsPack-pqc-legacy-tls-compliance-gyv22je0).

    PQC Conformance Pack Compliance Score is the percentage of the number of compliant rule-resource

    Figure 5: PQC Conformance Pack Compliance Score is the percentage of the number of compliant rule-resource

    Click the conformance pack to see an overall compliance summary across all 4 rules.

    Individual Rules View

    Go to AWS Config → Rules and find 4 rules with prefix pqc-:

    • pqc-elb-pqc-compliance-conformance-pack-
    • pqc-elb-legacy-tls-conformance-pack-
    • pqc-apigateway-pqc-compliance-conformance-pack-
    • pqc-apigateway-legacy-tls-conformance-pack-

    Click any rule to view:

    • Compliant vs non-compliant resource counts
    • Detailed annotations for each resource
    • Resource ARNs and current security policy configurations
    Visibility into Config rules status inside the conformance pack

    Figure 6: Visibility into Config rules status inside the conformance pack

    Sample image of the config rule findings and annotation describing the migeration guidance based on 3-tier classification.

    Figure 7: Sample image of the config rule findings and annotation describing the migration guidance based on 3-tier classification.

    Conclusion

    After deploying the PQC Readiness Scanner, you gain visibility into TLS posture across AWS edge services, which reduces manual configuration reviews. The tier system provides specific upgrade recommendations so teams can understand next steps without cryptographic expertise. The scanner automatically detects configuration changes to help new deployments maintain readiness standards. Built-in AWS Config reporting supports audit requirements and demonstrates measurable progress toward PQC readiness.

    Deploy the PQC Readiness Scanner and review your results with PQC Readiness Scanner. Start migration with high priority Tier 3 resources and monitor progress across your accounts using AWS Config aggregators.

    Additional resources

    If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on AWS Config re:Post or contact AWS Support.

    Pravin Nair

    Pravin Nair

    Pravin is a Senior Security Solutions Architect specializing in data protection and privacy at AWS. He partners with customers to architect secure, scalable cloud solutions that address complex security challenges across encryption, infrastructure protection, and privacy engineering. His expertise spans encryption at rest and in transit, infrastructure security, privacy-based architectures, and emerging security domains including generative AI security and post-quantum cryptography.