Tag Archives: Technical How-to

Building medallion architecture with Iceberg materialized views in Amazon SageMaker

2026-09-02 Gaurav Sharma

Post Syndicated from Gaurav Sharma original https://aws.amazon.com/blogs/big-data/building-medallion-architecture-with-iceberg-materialized-views-in-amazon-sagemaker/

Building a Medallion Architecture today typically means that you must build three separate systems working in concert: extract, transform, and load (ETL) jobs to transform data between layers, an orchestrator (such as Apache Airflow or AWS Step Functions) to sequence those jobs in the correct order, and custom change-data-capture (CDC) logic to make sure that each job processes only new or modified records. Each component must be authored, tested, deployed, and maintained independently and when one breaks, the entire pipeline stalls.

In this post, we show how Apache Iceberg materialized views in Amazon SageMaker collapse transformation, orchestration, and incremental processing into a single SQL definition per layer. You declare what each layer should contain, and the system handles when and how it refreshes based on your refresh configuration. With this approach, you can build a Bronze → Silver → Gold pipeline with three SQL statements. This reduces the complexity of maintaining separate orchestration code, CDC logic, and job artifacts.

What is medallion architecture

The medallion architecture organizes data into three progressive layers:

Bronze layer – Captures raw data as-is from source systems, preserving the original format for auditability and replay.
Silver layer – Applies cleaning, deduplication, type casting, and business logic to produce validated, query-ready datasets.
Gold layer – Aggregates Silver data into business-level metrics, key performance indicators (KPIs), and dimensional models optimized for analytics and reporting.

Each layer builds on the previous one, creating clear lineage from raw ingestion to business insight.

Traditional versus declarative approach

The two approaches differ in how much infrastructure you build and maintain.

Traditional approach

You write an ETL job such as Apache Spark script for Bronze to Silver layer and another for Silver to Gold layer. You build a directed acyclic graph (DAG) in Apache Airflow or a Step Functions state machine to run them in order. You implement CDC logic like tracking high watermarks, comparing snapshots, or consuming change streams such that each job processes only new data.

Declarative approach with Iceberg materialized views

You write one CREATE MATERIALIZED VIEW statement per layer with a SCHEDULE REFRESH EVERY N HOURS clause. The AWS Glue managed Spark compute executes the refresh, but you don’t author, version, or deploy a job artifact. Iceberg’s row-level change tracking (position-delete and equality-delete files) identifies which rows changed since the last refresh and AWS Glue processes only those rows. The dependency chain is implicit in the SQL definitions. The only code you maintain is the SQL transformation logic itself.

Apache Iceberg and materialized views

Apache Iceberg is an open-source, high-performance table format designed for petabyte-scale analytic datasets in data lakes. It provides ACID transactions, time travel, schema evolution, and hidden partitioning.

With an Iceberg materialized view, you can define each layer of a medallion architecture as a SQL statement. Under the hood, AWS Glue uses Iceberg’s change-tracking metadata to identify which rows changed since the last refresh, then processes only those rows using managed Spark compute. You configure scheduling and incremental processing through SQL definitions, and the system executes atomic refreshes without requiring you to write pipeline code.

When refreshed, the Gold materialized view reads incrementally from the Silver materialized view, which in turn reads from the Bronze table. This creates a declarative dependency chain: each layer’s definition points to the layer below it, and the system resolves which data to reprocess at each refresh.

Service support for Iceberg materialized views

At time of publication, the following services support creating and refreshing Iceberg materialized views:

For the latest version requirements, see the AWS Glue materialized views documentation.

Technical architecture

The architecture uses Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), as the storage layer. Amazon S3 Tables is a managed Apache Iceberg offering that alleviates the administrative overhead of maintaining Iceberg tables. AWS Glue Data Catalog manages table metadata, and Amazon SageMaker Unified Studio provides the AI-powered notebook environment with AWS Glue 5.1 for authoring and executing materialized view definitions.

The diagram illustrates a three-tier data lakehouse pipeline built on Apache Iceberg. The Bronze layer contains raw trip data (trips_bronze table on S3 Tables with fields: trip_id, city, vehicle_type, fare, status) that you ingest through INSERT/Append operations.

An incremental REFRESH feeds the Silver layer, where a materialized view (mv_trips_silver) performs timestamp conversion, null filtering, and computes derived columns like revenue_per_mile and rating_category. It processes only new or changed rows.

The Silver layer then refreshes two Gold layer materialized views on a daily schedule: mv_city_daily_metrics (city, date, trips, drivers, revenue, tips) and mv_vehicle_performance (vehicle_type, city, trips, revenue, distance). The Gold layer serves downstream consumers including Amazon Athena, Amazon Quick Sight, Amazon Redshift, and first-party (1P) or third-party (3P) compute engines supporting the Iceberg REST API.

The pipeline flows as follows:

Diagram of the medallion pipeline: a Bronze table feeds a Silver materialized view that feeds two Gold materialized views consumed by analytics engines

Figure 1: The three-tier medallion pipeline from the Bronze table through Silver and Gold materialized views to analytics consumers

Prerequisites

Before starting, verify that you have the following:

An AWS account with permissions for Amazon SageMaker Unified Studio, AWS Glue, S3 Tables, and AWS Lake Formation.
An Amazon SageMaker Unified Studio domain.

Step 1: Initialize the environment

Open the AWS Management Console and navigate to Amazon SageMaker.

Figure 2: The Amazon SageMaker console landing page

Choose Get Started to set up Amazon SageMaker Unified Studio.

SageMaker Unified Studio Get Started setup page

Figure 3: The Get Started page for setting up SageMaker Unified Studio

Choose Open to launch Amazon SageMaker Unified Studio.

Button to open and launch SageMaker Unified Studio

Figure 4: The option to open and launch SageMaker Unified Studio

After you’re in SageMaker Unified Studio, choose Data in the left pane to create the S3 Tables bucket (a managed Apache Iceberg feature of Amazon S3) and a database. Choose Add, then choose Create S3 Tables Catalog, and provide a catalog and a database name. Finally, choose Create Catalog.

Figure 5: The Create S3 Tables Catalog dialog with catalog and database name fields

After the catalog creation is complete, in the left navigation pane, choose Notebooks.

Notebooks option in the SageMaker Unified Studio left navigation pane

Figure 6: The Notebooks option in the SageMaker Unified Studio navigation pane

Choose Create Notebook.

Figure 7: The Create Notebook button in SageMaker Unified Studio

Before using the notebook, select either Athena Spark or Glue Spark compute connection as the runtime engine for your notebook.

Runtime engine selection showing Athena Spark and Glue Spark compute connections

Figure 8: Selecting Athena Spark or Glue Spark as the notebook runtime engine

Use the following code samples in individual notebook cells. You can also provide transformation requirements in natural language, and the SageMaker Data Agent will generate SQL code for you.

SageMaker Data Agent generating SQL from a natural language prompt

Figure 9: The SageMaker Data Agent generating SQL from a natural language request

Add each code block in a new cell by choosing the SQL button:

SQL cell-type button in the notebook toolbar

Figure 10: The SQL button for adding a code block to a notebook cell

Choose Athena Spark or Glue Spark as your compute from the cell menu.

Compute connection selection in the notebook cell menu

Figure 11: The compute selection in the notebook cell menu

If you encounter errors after cell execution, use the data agent chatbot or the Fix with AI button to resolve them.

Fix with AI button and data agent chatbot for resolving cell errors

Figure 12: The Fix with AI button for resolving cell execution errors

Step 2: Ingest data into Bronze

Generate 300 realistic ride-sharing trips and insert them directly into the Bronze Iceberg table. This simulates a raw data ingestion layer. In production, you generally configure a streaming source or batch load based on your requirements.

Copy the following code into the first notebook cell (use a Python cell type).

import random
from datetime import datetime, timedelta

CITIES = {
    "San Francisco": {"lat_range": (37.70, 37.82), "lon_range": (-122.52, -122.38), "surge_prob": 0.3},
    "Austin": {"lat_range": (30.22, 30.40), "lon_range": (-97.80, -97.68), "surge_prob": 0.15},
    "Chicago": {"lat_range": (41.85, 41.95), "lon_range": (-87.70, -87.60), "surge_prob": 0.2},
    "Seattle": {"lat_range": (47.55, 47.68), "lon_range": (-122.40, -122.28), "surge_prob": 0.25},
}
VEHICLE_TYPES = ["UberX", "Comfort", "XL", "Black"]
PAYMENT_METHODS = ["credit_card", "debit_card", "apple_pay", "google_pay", "cash"]
STATUSES = ["completed"] * 4 + ["cancelled_rider", "cancelled_driver"]
BASE_FARES = {"UberX": 2.50, "Comfort": 3.50, "XL": 4.00, "Black": 7.00}
PER_MILE = {"UberX": 1.75, "Comfort": 2.25, "XL": 2.50, "Black": 3.75}
PER_MIN = {"UberX": 0.35, "Comfort": 0.45, "XL": 0.50, "Black": 0.65}

rows = []
for i in range(300):
    city_name = random.choice(list(CITIES.keys()))
    city = CITIES[city_name]
    vehicle = random.choice(VEHICLE_TYPES)
    duration = random.randint(5, 45)
    distance = round(random.uniform(1.0, 20.0), 1)
    surge = round(random.uniform(1.0, 2.5), 1) if random.random() < city["surge_prob"] else 1.0
    base = BASE_FARES[vehicle]
    fare = round((base + distance * PER_MILE[vehicle] + duration * PER_MIN[vehicle]) * surge, 2)
    tip = round(fare * random.choice([0, 0, 0.1, 0.15, 0.2, 0.25]), 2)
    status = random.choice(STATUSES)
    day = random.randint(0, 2)
    hour = random.choices(range(24),
        weights=[1,1,1,1,1,2,4,8,10,8,6,5,6,5,5,5,6,8,10,8,6,4,2,1])[0]
    trip_time = datetime(2025, 12, 1) + timedelta(days=day, hours=hour, minutes=random.randint(0, 59))

    rows.append((
        f"TRIP-{i+1:06d}",
        f"DRV-{random.randint(1000, 5000)}",
        f"RDR-{random.randint(10000, 99999)}",
        city_name, vehicle,
        round(random.uniform(*city["lat_range"]), 6),
        round(random.uniform(*city["lon_range"]), 6),
        round(random.uniform(*city["lat_range"]), 6),
        round(random.uniform(*city["lon_range"]), 6),
        trip_time.isoformat(),
        (trip_time + timedelta(minutes=duration)).isoformat(),
        duration, distance, surge, base, fare, tip, round(fare + tip, 2),
        random.choice(PAYMENT_METHODS),
        random.choice([None, 3, 4, 4, 5, 5, 5]) if status == "completed" else None,
        status,
    ))

schema = ("trip_id STRING, driver_id STRING, rider_id STRING, city STRING, "
    "vehicle_type STRING, pickup_lat DOUBLE, pickup_lon DOUBLE, "
    "dropoff_lat DOUBLE, dropoff_lon DOUBLE, trip_start_time STRING, "
    "trip_end_time STRING, duration_minutes INT, distance_miles DOUBLE, "
    "surge_multiplier DOUBLE, base_fare DOUBLE, trip_fare DOUBLE, "
    "tip_amount DOUBLE, total_amount DOUBLE, payment_method STRING, "
    "rating INT, status STRING")

df = spark.createDataFrame(rows, schema)
df.writeTo("{CATALOG_NAME}.{NAMESPACE_NAME}.trips_bronze").createOrReplace()

print(f"Created Table and Inserted {len(rows)} trips into Bronze layer")

Step 3: Explore Bronze

Run a preview on the bronze table. The output should look like the following screenshot:

Preview of raw Bronze table trip records with string timestamps and nullable fields

Figure 13: A preview of raw trip records in the Bronze table

You should see raw, unprocessed trip records with string timestamps and nullable fields. This is exactly what the Silver layer will clean up.

Now, verify the ingested data by querying the Bronze table for basic statistics.

SELECT COUNT(*) as total_trips, COUNT(DISTINCT city) as cities,
COUNT(DISTINCT vehicle_type) as vehicle_types,
MIN(trip_start_time) as earliest, MAX(trip_start_time) as latest
FROM ({CATALOG_NAME}.{NAMESPACE_NAME}.trips_bronze

The output should look like the following screenshot:

Query results showing total trips, distinct cities, and vehicle types in the Bronze table

Figure 14: Bronze table statistics showing total trips, distinct cities, and vehicle types

Step 4: Create the Silver materialized view

This SQL statement defines the Silver layer as a materialized view that cleans, transforms, and derives new columns from the Bronze table. Note that this is only a definition. The system processes the data at refresh time.

CREATE MATERIALIZED VIEW IF NOT EXISTS {CATALOG_NAME}.{DATABASE}.mv_trips_silver
COMMENT 'Silver layer: Cleaned trip data with proper types and derived columns'
SCHEDULE REFRESH EVERY 1 DAY
AS
SELECT
trip_id, driver_id, rider_id, city, vehicle_type,
pickup_lat, pickup_lon, dropoff_lat, dropoff_lon,
CAST(trip_start_time AS TIMESTAMP) as trip_start_timestamp,
CAST(trip_end_time AS TIMESTAMP) as trip_end_timestamp,
duration_minutes, distance_miles, surge_multiplier,
base_fare, trip_fare, tip_amount, total_amount,
payment_method, rating, status,
CASE WHEN distance_miles > 0 THEN total_amount / distance_miles ELSE 0 END as revenue_per_mile,
CASE WHEN rating >= 4 THEN 'High' WHEN rating >= 3 THEN 'Medium' ELSE 'Low' END as rating_category
FROM {CATALOG_NAME}.{DATABASE}.trips_bronze
WHERE trip_id IS NOT NULL AND driver_id IS NOT NULL AND rider_id IS NOT NULL
AND total_amount >= 0 AND distance_miles >= 0

print("Silver MV created: urbanride.mv_trips_silver")

Verify the Silver layer output:

SELECT trip_id, city, vehicle_type, total_amount,
ROUND(revenue_per_mile, 2) as rev_per_mile, rating_category
FROM {CATALOG_NAME}.{DATABASE}.mv_trips_silver LIMIT 5

Notice how the Silver layer now has proper timestamps, derived revenue_per_mile, and rating categories: clean, typed, and ready for you to aggregate.

The output should look like the following screenshot:

Silver materialized view results with typed timestamps, revenue_per_mile, and rating_category columns

Figure 15: Silver materialized view results with typed timestamps and derived columns

Step 5: Create Gold materialized views

Gold materialized views read incrementally from the Silver materialized view. This is a nested materialized view pattern: a materialized view built on top of another materialized view.

Gold 1: City daily metrics

With this materialized view, you can aggregate trip data by city and date with a scheduled daily refresh.

CREATE MATERIALIZED VIEW IF NOT EXISTS {CATALOG_NAME}.urbanride.mv_city_daily_metrics
COMMENT 'Gold layer: Daily aggregated metrics by city'
SCHEDULE REFRESH EVERY 1 DAY
AS
SELECT
city, DATE(trip_start_timestamp) as trip_date,
COUNT(*) as total_trips,
COUNT(DISTINCT driver_id) as active_drivers,
COUNT(DISTINCT rider_id) as active_riders,
SUM(total_amount) as total_revenue,
SUM(distance_miles) as total_distance,
SUM(tip_amount) as total_tips
FROM {CATALOG_NAME}.{DATABASE}.mv_trips_silver
WHERE status = 'completed'
GROUP BY city, DATE(trip_start_timestamp)

print("Gold MV created: mv_city_daily_metrics (reads from Silver MV, refreshes daily)")

Gold 2: Vehicle performance

With this materialized view, you can aggregate performance metrics by vehicle type and city.

CREATE MATERIALIZED VIEW IF NOT EXISTS {CATALOG_NAME}.{DATABASE}.mv_vehicle_performance
COMMENT 'Gold layer: Vehicle type performance metrics'
SCHEDULE REFRESH EVERY 1 DAY
AS
SELECT
vehicle_type, city,
COUNT(*) as trip_count,
SUM(total_amount) as total_revenue,
SUM(distance_miles) as total_distance,
SUM(tip_amount) as total_tips
FROM {CATALOG_NAME}.{DATABASE}.mv_trips_silver
WHERE status = 'completed'
GROUP BY vehicle_type, city

print("Gold MV created: mv_vehicle_performance (reads from Silver MV, refreshes daily)")

Dependency chain

The complete pipeline dependency is:

trips_bronze (table)
└── mv_trips_silver (materialized view)
    ├── mv_city_daily_metrics (MV on MV, daily schedule)
    └── mv_vehicle_performance (MV on MV, daily schedule)

Each layer is defined by a single SQL statement. There are no DAGs to maintain, no job definitions to deploy, and no watermark tracking to implement.

Step 6: Query the Gold layer

Query the Gold materialized views to see aggregated business metrics.

City daily metrics Gold table

SELECT city, trip_date, total_trips, active_drivers,
ROUND(total_revenue, 2) as revenue,
ROUND(total_revenue / total_trips, 2) as avg_per_trip
FROM {CATALOG_NAME}.{DATABASE}.mv_city_daily_metrics
ORDER BY trip_date DESC, revenue DESC LIMIT 15

The output should look like the following screenshot:

City daily metrics results with trips, active drivers, and revenue per city

Figure 16: City daily metrics from the Gold materialized view

Vehicle performance Gold table

SELECT vehicle_type, city, trip_count,
ROUND(total_revenue, 2) as revenue,
ROUND(total_revenue / trip_count, 2) as avg_per_trip
FROM {CATALOG_NAME}.{DATABASE}.mv_vehicle_performance
ORDER BY revenue DESC

The output should look like the following screenshot:

Vehicle performance results with trip counts and revenue by vehicle type and city

Figure 17: Vehicle performance metrics from the Gold materialized view

The Gold layer gives you pre-aggregated, business-ready metrics without writing aggregation jobs.

Step 7: Data propagation demo

This section demonstrates how changes propagate through the layers using INSERT, UPDATE (MERGE), and DELETE operations followed by incremental refresh. In production, the scheduled refresh handles this automatically. We trigger it manually here for demonstration purposes.

INSERT new records

Insert new trip records into the Bronze table.

INSERT INTO {CATALOG_NAME}.{DATABASE}.trips_bronze VALUES
('DEMO_TRIP_001', 'DRIVER_999', 'RIDER_888', 'Seattle', 'UberX',
47.6062, -122.3321, 47.6205, -122.3493,
'2024-12-15 14:30:00', '2024-12-15 14:50:00',
20, 5.2, 1.0, 10.0, 15.0, 3.0, 18.0, 'credit_card', 5, 'completed'),
('DEMO_TRIP_002', 'DRIVER_888', 'RIDER_777', 'Seattle', 'XL',
47.6101, -122.3300, 47.6550, -122.3080,
'2024-12-15 15:00:00', '2024-12-15 15:35:00',
35, 8.5, 1.5, 15.0, 30.0, 5.0, 35.0, 'cash', 4, 'completed'),
('DEMO_TRIP_003', 'DRIVER_777', 'RIDER_666', Portland, 'Comfort',
30.2672, -97.7431, 30.2800, -97.7400,
'2024-12-15 16:00:00', '2024-12-15 16:15:00',
15, 3.0, 1.0, 8.0, 12.0, 2.0, 14.0, 'credit_card', 5, 'completed')

print("Inserted 3 new trips into Bronze")

Refresh Silver (incremental)

Refresh the Silver materialized view. Iceberg materialized view processes only three new records.

REFRESH MATERIALIZED VIEW {CATALOG_NAME}.{DATABASE}.mv_trips_silver"

Verify the new records propagated

SELECT trip_id, city, total_amount, ROUND(revenue_per_mile, 2) as rev_per_mile, rating_category
FROM {CATALOG_NAME}.{DATABASE}.mv_trips_silver
WHERE trip_id LIKE 'DEMO_TRIP_%' ORDER BY trip_id

The output should look like the following screenshot:

Silver materialized view showing three newly inserted demo trips

Figure 18: The Silver materialized view showing the three newly inserted demo trips

Refresh Gold (cascading from the Silver materialized view)

Refresh the Gold materialized view. It reads from the refreshed Silver materialized view and processes only the incremental changes.

REFRESH MATERIALIZED VIEW {CATALOG_NAME}.{DATABASE}.mv_city_daily_metrics

Verify the Gold layer reflects the new trips

SELECT city, trip_date, total_trips, ROUND(total_revenue, 2) as revenue
FROM {CATALOG_NAME}.{DATABASE}.mv_city_daily_metrics
WHERE trip_date = '2024-12-15' ORDER BY city

The output should look like the following screenshot:

City daily metrics reflecting the newly added trips for December 15, 2024

Figure 19: City daily metrics reflecting the new trips for 2024-12-15

UPDATE through MERGE

Use MERGE to update existing records in Bronze, then refresh incrementally.

MERGE INTO {CATALOG_NAME}.{DATABASE}.trips_bronze AS target
USING (SELECT 'DEMO_TRIP_002' as trip_id, 5 as new_rating, 20.0 as new_tip) AS source
ON target.trip_id = source.trip_id
WHEN MATCHED THEN UPDATE SET
target.rating = source.new_rating,
target.tip_amount = source.new_tip,
target.total_amount = target.trip_fare + source.new_tip

Refresh Silver and verify

REFRESH MATERIALIZED VIEW {CATALOG_NAME}.{DATABASE}.mv_trips_silver")

SELECT trip_id, rating, rating_category, tip_amount, total_amount,
ROUND(revenue_per_mile, 2) as rev_per_mile
FROM {CATALOG_NAME}.{DATABASE}.mv_trips_silver WHERE trip_id = 'DEMO_TRIP_002'

print("UPDATE propagated: rating 4->5, tip $5->$20, total $35->$50")

The output should look like the following screenshot:

Silver materialized view showing the updated rating and tip for DEMO_TRIP_002

Figure 20: The Silver materialized view showing the updated rating and tip for the demo trip

Step 8: Cleanup

Drop materialized views, tables, the namespace, and delete the S3 Tables bucket to fully clean up resources.

# Drop MVs (Gold first, then Silver, due to dependency order)
spark.sql(f"DROP MATERIALIZED VIEW IF EXISTS {CATALOG_NAME}.{DATABASE}.mv_city_daily_metrics")
spark.sql(f"DROP MATERIALIZED VIEW IF EXISTS {CATALOG_NAME}.{DATABASE}.mv_vehicle_performance")
spark.sql(f"DROP MATERIALIZED VIEW IF EXISTS {CATALOG_NAME}.{DATABASE}.mv_trips_silver")
print("All materialized views dropped")

# Drop base table
spark.sql(f"DROP TABLE IF EXISTS {CATALOG_NAME}.{DATABASE}.trips_bronze")
print("Base table dropped")

# Drop the namespace
spark.sql(f"DROP NAMESPACE IF EXISTS {CATALOG_NAME}.{DATABASE} ")
print("Namespace dropped")

# Delete the S3 table bucket
import boto3
s3tables_client = boto3.client("s3tables")

# List and delete all remaining tables in the bucket
tables_response = s3tables_client.list_tables(
    tableBucketARN=TABLE_BUCKET_ARN, namespace="{DATABASE}"
)
for table in tables_response.get("tables", []):
    s3tables_client.delete_table(
        tableBucketARN=TABLE_BUCKET_ARN, namespace="{DATABASE}", name=table['name']
    )
    print(f" Deleted table: {table['name']}")

# Delete the namespace and bucket
s3tables_client.delete_namespace(tableBucketARN=TABLE_BUCKET_ARN, namespace="urbanride")
s3tables_client.delete_table_bucket(tableBucketARN=TABLE_BUCKET_ARN)
print(f"S3 table bucket deleted: {TABLE_BUCKET_NAME}")

Limitations and considerations

While materialized views remove most orchestration code, note the following:

No sub-hour freshness. The minimum schedule granularity is one hour (SCHEDULE REFRESH EVERY 1 HOUR).
Cascading refresh isn’t automatic. Refreshing Silver doesn’t trigger Gold in the same operation. Each layer refreshes on its own schedule or must be triggered sequentially.
Deletes require a FULL refresh. An incremental REFRESH that feeds the Silver layer detects inserts and updates through Iceberg metadata but cannot detect row removals. Use REFRESH ... FULL when delete propagation is needed.
SQL subset only. Some window functions, user-defined functions (UDFs), and complex expressions might not be supported in materialized view definitions.
Schema evolution requires recreation. If the source schema changes in a way that affects the materialized view definition, you must drop and recreate it.
AWS-specific extension. Iceberg materialized views are not part of the open-source Apache Iceberg specification. They aren’t portable to non-AWS environments.

Pricing

AWS bills materialized view auto-refresh at USD $0.44 per DPU-hour (4 vCPU, 16 GB memory), billed per second with a 1-minute minimum. When you configure scheduled refresh, the AWS Glue Data Catalog uses managed Spark compute to incrementally update the materialized view. You pay only for the compute time of each refresh run.

There are no separate charges for storing materialized view metadata in the Data Catalog (covered under standard catalog pricing: first million objects at no additional cost, then $1.00 per 100K objects/month). The materialized view data itself is stored as Iceberg files in S3 Tables or Amazon S3, charged at standard Amazon S3 storage rates.

Manual refreshes triggered from Spark (through Amazon Athena, Amazon EMR, or AWS Glue notebooks) are billed under those services’ respective compute pricing rather than the materialized view auto-refresh rate. For the latest pricing details, see the AWS Glue pricing page.

Estimated cost for this tutorial: Running through all steps once with 300 records typically consumes less than 0.5 DPU-hours total (~$0.22 in AWS Glue compute plus negligible Amazon S3 storage).

Summary

In this post, you built a Bronze → Silver → Gold medallion architecture using three SQL statements with nested materialized views and no orchestration code. The full pipeline creation took under 2 minutes, and incremental refreshes processed only changed data with no watermarks, no DAGs, no CDC plumbing.

To get started with your own data, create an Amazon SageMaker Unified Studio project, define your Bronze table, and express your transformation logic as Iceberg materialized views. For more information, see the Apache Iceberg materialized views documentation in the AWS Glue Developer Guide.

References

Using materialized views with AWS Glue

Query AWS Glue Data Catalog materialized views

Using materialized views with Amazon EMR

Working with Amazon S3 Tables and table buckets

About the authors

Build a dynamic streaming data lake with Apache Iceberg and Apache Flink

2026-09-02 Francisco Morillo

Post Syndicated from Francisco Morillo original https://aws.amazon.com/blogs/big-data/build-a-dynamic-streaming-data-lake-with-apache-iceberg-and-apache-flink/

Handling upstream schema changes is a common operational challenge in streaming data pipelines that write to a data lake. When a source schema changes, teams often face a difficult choice: restart the pipeline or perform a manual migration. A restart can pause ingestion and delay or lose in-flight data. A manual migration consumes engineering time and introduces the risk of schema inconsistencies while the data lake falls behind the source.

For example, consider an Apache Flink job that ingests order_events and writes to an Iceberg table. On Monday, the pipeline runs normally. By Wednesday, the upstream team adds a new loyalty_tier field and introduces a new interaction_events event type. Traditionally, you would need to stop the Flink job, update your schema definitions, and redeploy. With Apache Iceberg’s Dynamic Iceberg Sink on Amazon Managed Service for Apache Flink, the pipeline can handle both changes at the record level without disruption. The DynamicSink routes each event to the right Iceberg table and evolves table schemas as new columns appear, with no operator intervention.

Managed Service for Apache Flink is a fully managed AWS service that you can use to build and deploy streaming applications without setting up infrastructure and managing resources. Apache Flink’s distributed processing engine with exactly once processing guarantees through checkpointing paired with Apache Iceberg’s two-phase commit provides end-to-end consistency without duplications or data loss.

In this post, we show you how to build a dynamic streaming data lake that adapts to new event types and schema changes without stopping the pipeline. Using Apache Flink 2.3 and Apache Iceberg 1.11.0 on Managed Service for Apache Flink, we walk through the DataStream API patterns for per-record table routing and automatic schema evolution. The complete implementation is available in this GitHub repository.

Apache Iceberg dynamic sink

The Dynamic Iceberg Sink allows Flink to dynamically route records to multiple Iceberg tables based on user-defined logic. It also creates and updates tables on the fly and evolves both table schemas and partition specs during streaming execution, controlled through the DynamicRecord class, which eliminates the need for Flink job restarts when requirements change.

Per-record table routing with DynamicIcebergSink

The DynamicIcebergSink resolves the target table at the record level rather than at pipeline configuration time. Records flow through a DynamicRecordGenerator that, for each input, emits one or more DynamicRecord values. Each DynamicRecord carries its own target table ID, schema, partition spec, and row payload, so the sink knows where to write and how the table should look:

DynamicIcebergSink.forInput(events)
    .generator(generator)
    .catalogLoader(catalogLoader)
    .immediateTableUpdate(true)
    .cacheMaxSize(cacheMaxSize)
    .cacheRefreshMs(cacheRefreshMs)
    .append();

The generator receives each record and emits a DynamicRecord targeting a resolved table that looks as follows:

return new DynamicRecord(
    tableId,
    tableBranch,
    icebergSchema,
    rowData,
    partitionSpec,
    distributionMode,
    1);

The sink creates the table if it does not exist and evolves its schema when a record carries new columns. cacheMaxSize and cacheRefreshMs bound the sink’s per-table metadata cache, so a job that writes to many tables does not reload metadata on every record. immediateTableUpdate(true) controls how those catalog changes are applied, which the following section on automatic schema evolution explains. A single Flink job can ingest and route order_events, interaction_events, user_events, and future event types without additional sink definitions.

However, the sink also needs to know what the table looks like. That is why every DynamicRecord also carries the Iceberg schema so that DynamicIcebergSink can create the table on first sight and evolve it as new fields appear. The schema information can be inferred from the data or read from a schema registry.

Automatic schema evolution

Streaming sources add new fields over time, and DynamicIcebergSink handles them without a restart. Before writing each record, it compares the record’s schema against the target table. If the record has a new field, Iceberg adds it as an optional column and commits the change with the next data file. Existing files stay valid and no table rewrite is needed. When you query older files, the new column returns null.

The immediateTableUpdate setting controls where the catalog change happens. The GitHub sample repository sets immediateTableUpdate=true, so the writer subtask that sees the new schema applies the create or alter inline, before it emits the record. This gives the lowest latency but makes more concurrent calls to the catalog. When set to false, records that require a table change take a detour. Records whose table, schema, and partition spec already match the sink’s cached metadata go straight to the writers. Records that do need a change are routed, keyed by table name, to an update operator, so updates for the same table apply one at a time. Once the update commits and the cache refreshes, subsequent records match again and skip the detour. In steady state, with no schema changes arriving, this path adds no extra shuffle. Either way, the schema comparison and the resulting table change are the same.

Schema changes are non-destructive by default. The sink can add new columns, widen existing types (for example, int to long or float to double), relax a required column to optional, and drop columns. Importantly, DynamicIcebergSink does not support renaming columns at the time of writing.

Source schemas are identified in two ways: inferring the schema from source records (for example, JSON inference) and reading serialized records from a schema registry (for example, AWS Glue Schema Registry (GSR)). Schema evolution behavior for the Iceberg sink table depends on the schema source. JSON inference adds any new field it sees, with no contract. For example, this allows the job to initially infer a schema as an integer, and later expand to a long when larger values are detected. Schema registry serialized records define the policy using the registry’s compatibility rules (for example, BACKWARD). This means that incompatible producer changes are rejected when the schema is registered rather than at write time.

The partition spec travels on each DynamicRecord, so the sink applies it when it creates or updates the table. How our sample derives that spec is covered in the partitioning section.

Solution overview

The following diagram illustrates the solution architecture. A data generator (a local Java application) writes events to an Amazon Kinesis Data Stream. In Avro mode it also registers each event schema in the AWS Glue Schema Registry. A Managed Service for Apache Flink application consumes the stream, resolves a target Iceberg table for each record, and writes to Iceberg tables in Amazon S3, cataloged either in the AWS Glue Data Catalog or, for fully managed tables, in Amazon S3 Tables, a capability of Amazon S3.

Data generator sends events to Amazon Kinesis Data Streams, and Managed Service for Apache Flink routes each record to an Iceberg table in Amazon S3

Figure 1: Solution architecture for routing streaming records to per-event Iceberg tables on Managed Service for Apache Flink

At a high level, a single Managed Service for Apache Flink application reads raw records from Kinesis and resolves a target Iceberg table for each record. It uses the DynamicIcebergSink to create and evolve tables on demand. The same job handles many event types because the destination is decided per record, not per sink.

A note on stream topology: the examples assume one Kinesis stream carrying multiple event types, which keeps the walkthrough focused. This is not a requirement for the pattern. If your events arrive on separate streams (for example, one stream per producer or per domain), create one KinesisStreamsSource per stream and union them into a single DataStream before the sink. The routing generator chooses the destination table from the record itself, so many sources can fan into one DynamicIcebergSink and still land in the correct tables.

Unioning does not add shuffle cost. The sink always re-distributes records by an internal per-table writer key, so a unioned stream and N separate pipelines incur the same per-record exchange. The distribution mode each DynamicRecord carries only changes which writer subtask a row lands on, not whether a shuffle occurs. The real tradeoff is isolation. All tables share one writer pool, one commit aggregator, and one committer. A hot stream’s backpressure and checkpoint alignment therefore couple to every other stream, and writer parallelism is a single job-wide setting. Prefer one unioned pipeline when you have many small-to-medium event types that should pool capacity. Split into separate applications when one stream is high-volume enough to need its own writer parallelism and failure isolation.

DynamicIcebergSink needs a schema for every record. The sample provides two interchangeable ways to obtain it, implemented as two generator variants: Option 1 infers the schema from each JSON record at runtime. Option 2 reads the registered schema from AWS Glue Schema Registry. Everything downstream (routing, table creation, and schema evolution) is identical, and only the generator changes.

Option 1: Infer the schema from the JSON record

SchemaAgnosticRoutingGenerator implements Iceberg’s DynamicRecordGenerator. Its generate method maps the routing field to a table name, infers the schema, derives a partition spec, and emits a DynamicRecord through the collector:

@Override
public void generate(JsonNode json, Collector<DynamicRecord> out) {
    String tableName = determineTableName(json); // routing field -> table name
    TableIdentifier tableId = TableIdentifier.of(database, tableName);
    Schema schema = inferSchemaFromJson(json); // cached by schema signature
    RowData rowData = convertJsonToRowData(json, schema);
    PartitionSpec spec = buildPartitionSpec(schema); // cached per schema
    out.collect(new DynamicRecord(
        tableId, "main", schema, rowData, spec, DistributionMode.NONE, 4));
}

The table name comes from an explicit table-name field when present, otherwise from the routing field (event_type by default).

For schemaless or semi-structured JSON, the generator infers an Iceberg schema directly from each record. This is convenient, but inference is fundamentally lossy because JSON does not carry type information. The generator therefore applies deliberately conservative rules and selects a stable type rather than the narrowest one:

JSON value	Iceberg type
Integer	LongType (all integral values are widened to long)
String	StringType
Floating-point values	DoubleType
Boolean	BooleanType
ISO-8601 timestamps	TimestampType (microseconds)
Nested JSON object	StructType (with fields inferred recursively)
JSON array	ListType (with element type inferred from array contents)

Partitioning the routed tables

Partitioning is decided by our generator, not by the sink, and the same mechanism applies to both schema options: the JSON-inference and schema-registry generators share the partition-candidate logic. The open source DynamicIcebergSink applies whatever PartitionSpec each DynamicRecord carries. Our sample’s SchemaAgnosticRoutingGenerator builds that spec at runtime: it reads a list of candidate partition fields from the partition.candidates application property and derives a per-table spec from the fields it observes. For each table, buildPartitionSpec walks that list and keeps only the candidates present in the table’s schema.

The same list adapts to each table. A table with event_date and region is partitioned by identity(event_date) and identity(region). A table with none of the candidates is created unpartitioned. The resulting spec travels on each DynamicRecord, so the sink applies it when it first creates the table.

For example, with partition.candidates = event_time,region,product: a table whose schema has event_time and product is created partitioned by those two. A table with only event_time gets identity(event_time). A table with none of the candidates is created unpartitioned. Partition specs are not frozen at creation time either: the sink evolves them through Iceberg partition-spec evolution, adding a candidate field when it later appears in the table’s schema and removing one that disappears. This is a metadata-only change, so existing data files keep the spec they were written with.

Two operational practices follow. First, always include your event-time field among the candidates so every table is at least time-partitioned, and monitor for unpartitioned tables through the table’s $partitions metadata or its spec in the catalog: a producer that emits create_timestamp instead of event_time will silently create unpartitioned tables until the candidate list is updated. Second, be deliberate with generic fields like region. If a source produces high-cardinality values for a candidate field, you can correct the spec later. Evolution applies to newly written files only, so the small files already written remain until compaction rewrites them.

Note that the candidate list is global, not per table. It tracks every field you might partition on, and each table takes only the ones it has.

Option 2: Read the schema from a schema registry

Inference is convenient but lossy, and it offers no contract: nothing stops a producer from silently changing a field’s type or meaning. The second option removes the guesswork by reading the schema from a registry instead of the data. Many production streaming platforms standardize on strongly typed Avro schemas managed through AWS Glue Schema Registry. With GSR, producers register schemas explicitly, each record on Kinesis is Avro-encoded and prefixed with a schema-version ID, and the consumer decodes against the exact registered schema. That gives you three things JSON inference cannot: precise types (a long stays a long, a timestamp-micros stays a timestamp-micros), a governed evolution policy enforced at registration, and a single source of truth shared across producers and consumers.

The pattern works with any schema registry that gives consumers the writer’s schema per record. The sample implements it with AWS Glue Schema Registry, but the same generator shape applies to other registries.

The dynamic-sink-avro-sample module applies GSR-managed Avro schemas to the same dynamic routing and schema evolution pattern. For each record, AvroToDynamicRecordGenerator reads the schema-version ID and fetches the writer schema from GSR, caching it after the first lookup. It then converts that schema to an Iceberg schema, decodes the payload into RowData, and emits a DynamicRecord, exactly as the JSON generator does:

The sink wiring is identical to option 1. Only the generator changes, and because the source carries raw Avro bytes the input stream is byte[] rather than parsed JSON:

AvroToDynamicRecordGenerator generator = new AvroToDynamicRecordGenerator(
    awsRegion, registryName, database, partitionCandidates, branch);
DynamicIcebergSink.forInput(eventBytes)
    .generator(generator)
    // identical catalogLoader, immediateTableUpdate(true), cache, and write settings as option 1
    .append();

Because the schema comes from GSR rather than from inspecting bytes, the Avro-to-Iceberg type mapping is exact:

Category	Avro type	Iceberg type
Primitive	int	IntegerType
Primitive	long	LongType
Primitive	float	FloatType
Primitive	double	DoubleType
Primitive	string	StringType
Primitive	boolean	BooleanType
Logical	timestamp-millis	TimestampType (preserves millisecond precision)
Logical	timestamp-micros	TimestampType (preserves microsecond precision)
Logical	decimal	DecimalType
Complex	record	StructType (nested fields mapped recursively)
Complex	array	ListType (element type inferred from items schema)
Complex	map	MapType (keys are always StringType)

The GSR integration handles schema versioning transparently. As soon as a producer registers a new schema version containing additional fields, the Flink consumer deserializes the updated payload and evolves the Iceberg table to match, with no job restart.

Prerequisites

To follow along, you need the following:

An AWS account with permissions to create Amazon Kinesis Data Streams, Managed Service for Apache Flink applications, AWS Glue resources, and Amazon S3 buckets (plus Amazon S3 Tables if you choose that catalog).
The AWS Command Line Interface (AWS CLI) configured with credentials.
Node.js 18 or later and the AWS Cloud Development Kit (AWS CDK) CLI.
Java 17 or later and Apache Maven 3.9 or later, to build the data generator.
Docker running locally. The CDK build bundles the Flink application jars inside a Maven image.

Deploy and test the solution

The accompanying repository provisions everything through a single parameterized AWS CDK stack.

Install the CDK dependencies and bootstrap your environment (first time only):

cd cdk-infrastructure && npm install
npx cdk bootstrap aws://<account>/<region>

Deploy the variant you want to try:
```
npx cdk deploy -c appType=dynamic -c tableFormatVersion=2 # JSON inference variant
npx cdk deploy -c appType=dynamic-avro -c tableFormatVersion=2 # GSR Avro variant
```
Add -c catalogType=s3tables to either command to use Amazon S3 Tables instead of the AWS Glue Data Catalog. The walkthrough sets tableFormatVersion=2 so you can query the results with a broad range of engines. Omit it to use the default, Iceberg format version 3, when you query with a v3-aware engine such as Spark on Amazon EMR 7.12+ or AWS Glue ETL.

Start the application using the ApplicationName value from the stack outputs:

aws kinesisanalyticsv2 start-application --application-name <ApplicationName> --run-configuration 'ApplicationRestoreConfiguration={ApplicationRestoreType=SKIP_RESTORE_FROM_SNAPSHOT}'

Send test events with the included data generator. Start with the v1 payloads, which create the tables without the optional fields:
```
java -jar data-generator/target/data-generator-1.0-SNAPSHOT.jar <stream-name> <region> 100 60 v1
```
Then send v2 payloads, which add the userAgent and scrollDepth fields. This second run is the schema evolution you observe in the next step:
```
java -jar data-generator/target/data-generator-1.0-SNAPSHOT.jar <stream-name> <region> 100 60 v2
```
For the Avro variant, the generator registers each schema version in the AWS Glue Schema Registry as it sends:
```
java -jar data-generator/target/data-generator-1.0-SNAPSHOT.jar avro <stream-name> <region> <registry-name> 100 60
```
Query the routed tables in Amazon Athena. You should see one Iceberg table per event type appear in the database within a checkpoint interval, and after sending v2 events, the new fields (userAgent, scrollDepth) show up as optional columns on the same tables. The Iceberg metadata tables (for example, SELECT * FROM "db"."table$snapshots") show each commit the sink makes.

Clean up

When you finish testing, delete the resources to stop incurring charges:

cd cdk-infrastructure && npx cdk destroy

CDK removes the Kinesis Data Stream, the Managed Service for Apache Flink application, and the stack-created AWS Identity and Access Management (IAM) roles. Additionally, empty and delete the S3 warehouse bucket to remove the Iceberg data and metadata files, delete any schemas the Avro variant registered in the AWS Glue Schema Registry, and delete the table bucket contents if you used the S3 Tables catalog.

Conclusion

With Apache Iceberg 1.11.0 and Flink 2.3, you can build streaming data lake architectures that adapt to change without stopping the pipeline. With per-record routing, a single Flink application can write multiple event types to separate Iceberg tables, while automatic schema evolution keeps table definitions aligned with changing source data. Choosing AWS Glue Schema Registry over runtime JSON inference adds precise types and a governed evolution contract, and a configurable partition-candidate list keeps each routed table partitioned correctly without pre-declaring its schema.

The result is fewer pipeline redeployments, reduced operational overhead, and a data lake that remains synchronized with evolving application schemas.

To get started, follow the deploy and test section, then adapt the routing field and partition candidates to your own event types.

The full sample code is available in the accompanying GitHub repository.

About the authors

Observing and evaluating production agents using OpenSearch Agent Health

2026-09-01 Ulrich Hinze

Post Syndicated from Ulrich Hinze original https://aws.amazon.com/blogs/big-data/observing-and-evaluating-production-agents-using-opensearch-agent-health/

As AI agents are moving from experimental prototypes to production workloads, teams need visibility into what agents are doing and a systematic way to measure whether they’re doing it well. Traditional testing methodologies like unit and integration tests fall short for this task, as measuring an agent’s quality isn’t a straightforward true/false decision. Instead, agent observability and evaluations (evals for short) provide a two-legged solution to this problem. Agent observability captures the details of an agent’s behavior, and evals compare this behavior to the behavior that you want. With this approach, teams can monitor their agent’s quality over time and introduce agent-specific quality gates in their software development lifecycle.

In this post, we show how to combine an AI agent running on AWS with OpenSearch Agent Health for observability and evals. You will deploy an agent and its observability data pipeline to AWS, then use Agent Health as a local development tool connecting to your cloud resources.

Overview of solution

Agent observability and evaluations rely on OpenTelemetry traces to understand agent behavior. Traces describe the flow of a request through components of a system. OpenSearch Agent Health is a purpose-built tool for analyzing agent traces and running evaluations against an agent for quality control. Although Agent Health works with any open source OpenSearch installation, many AWS customers choose Amazon OpenSearch Ingestion and Amazon OpenSearch Service for ingesting and storing their OpenTelemetry data. You can connect OpenSearch Agent Health to these AWS resources to fetch live data and store its own configuration and evaluation history.

The following diagram shows the overall architecture of the solution presented in this post:

Figure 1: Solution overview

The individual parts are:

AWS Amplify for hosting an assistant-ui chat interface. Connects to the agent backend using the Agent-User Interaction (AG-UI) protocol.
Sample ecommerce AI agent using Strands Agents SDK, deployed to Amazon Bedrock AgentCore runtime, exposing an AG-UI Server-Sent Events (SSE) endpoint. This agent has access to multiple tools, such as product search and shopping basket operations. For this sample project, the tool calls are all simulated within the agent runtime rather than including API calls to other systems. The agent emits messages, reasoning steps, and tool calls as OpenTelemetry traces.
Large language models (LLMs) on Amazon Bedrock. One model (Amazon Nova 2 Lite) is used to power the agent, the other model (Anthropic Claude Opus 4.6) is used to evaluate the agent behavior.
Amazon OpenSearch Ingestion for collecting and transforming the raw agent traces and loading them into an Amazon OpenSearch Service domain. Agent traces have the same structure as regular OpenTelemetry traces, with the addition of generative AI semantics (for example, tool calls and token usage). This means a regular OpenTelemetry pipeline configuration can be used to process agent traces.
OpenSearch Agent Health for analyzing traces and running evaluation test cases and benchmarks against the agent. Agent Health uses the same AG-UI endpoint as the front-end application. It authenticates to the application, to Amazon Bedrock for model functionality, and to Amazon OpenSearch Service using AWS SigV4 authentication.

Walkthrough

In this walkthrough, we showcase how you can use Agent Health and Strands to measure and improve your agent’s quality over time.

We follow these steps:

Deploy solution to AWS and test the application.
Start Agent Health locally and connect it to cloud resources.
Explore agent traces and run evaluations.

We have created a GitHub repository for you to follow along.

Prerequisites

For this walkthrough, you should have the following prerequisites:

An AWS account
Git
Node.js
AWS Cloud Development Kit (AWS CDK)

Deploy solution to AWS and test the application

In this section, you check out the repository and deploy the infrastructure to AWS. Be aware that these steps create AWS resources that incur cost. We cover cleanup steps at the end of this post.

First, clone the repository to a local directory:

git clone https://github.com/aws-samples/sample-agent-health-with-amazon-opensearch-service && cd sample-agent-health-with-amazon-opensearch-service

Switch to the infra folder and install dependencies:

cd infra && npm install

Before you can start the deployment, determine the AWS Identity and Access Management (IAM) user or role that you will use to start Agent Health later on. In many cases, this will be the same role that you use to deploy the infrastructure. Set this ARN in your environment by issuing the following command:

export AGENT_HEALTH_READER_ARN=arn:aws:iam::<YOUR_ACCOUNT_ID>:role/<YOUR_ROLE_NAME>

Bootstrap your AWS account for use with AWS CDK:

cdk bootstrap -c agentHealthReaderArn=$AGENT_HEALTH_READER_ARN

Run the infrastructure deployment. Review and acknowledge IAM statement changes when prompted. This takes around 25 minutes to complete:

cdk deploy -R -c agentHealthReaderArn=$AGENT_HEALTH_READER_ARN

The -R parameter defines that if something fails during this deployment, the successfully provisioned resources are retained. Be aware that this command creates AWS resources, incurring cost. Review the cleanup section at the end of this post for removing all created resources.

When the deploy command finishes successfully, you should see an output like the following:

...
AgentObservabilityStack

✨ Deployment time: 1402.42 s

Outputs:
AgentObservabilityStack.AgentEndpoint = https://bedrock-agentcore.us-east-1.amazonaws.com/runtimes/arn%3Aaws%3Abedrock-agentcore%3Aus-east-1%3A123456789012%3Aruntime%2Fretail_agent-abcdefghij/invocations?qualifier=AgentObservabilityStAgentRuntimeEndpointABCDEFGH
AgentObservabilityStack.ChatUrl = https://main.abcdefghijklmn.amplifyapp.com
...

Next, create a user for your application. Retrieve the CDK output value for AgentObservabilityStack.UserPoolId. Create a user for the application using the user pool ID, an email address, and a strong password (minimum eight characters including uppercase, lowercase, letter, and digit):

export COGNITO_EMAIL=<YOUR_EMAIL>
export COGNITO_PASSWORD=<YOUR_PASSWORD>
export USER_POOL=<YOUR_USER_POOL_ID>
aws cognito-idp admin-create-user --user-pool-id $USER_POOL --username $COGNITO_EMAIL --message-action SUPPRESS --user-attributes Name=email_verified,Value=true
aws cognito-idp admin-set-user-password --user-pool-id $USER_POOL --username $COGNITO_EMAIL --password "$COGNITO_PASSWORD" --permanent

You can now access the retail agent application. From the CDK output values, retrieve the value for AgentObservabilityStack.ChatUrl. Copy and paste this URL into your browser. Log in with your email and password. You should now see the agent interface:

Sample retail agent chat interface showing the ecommerce assistant ready for queries

Figure 2: Sample retail agent user interface

Experiment with the application. Here is an example sequence of queries you can put in:

Do you have books on Python?
Is this in stock?
Put it into my basket.
What else can you do for me?

Start Agent Health

Now that you have the infrastructure running, you can start OpenSearch Agent Health locally and connect it to your cloud resources.

The CDK infrastructure deployment created a file cdk-output.json, which contains all relevant configuration values for Agent Health. We’ve already created a file agent-health/agent-health.config.ts that pulls these values dynamically in your environment, so you can start Agent Health without any further configuration.

Open a terminal and start Agent Health by running the following command:

cd ../agent-health && npm install
npx @opensearch-project/agent-health

Open http://localhost:4001 in your browser to access Agent Health UI. Choose Agent Traces in the sidebar menu to access your agent’s traces. You should see traces from your previous interactions:

Agent Health Traces view listing agent traces captured from previous interactions

Figure 3: Agent traces. As Agent Health is in active development, this interface might have changed since the time of writing

Expand the trace and explore the information it contains, such as token count and agent trajectory (sequence of messages, reasoning steps, and tool calls).

If you’re unable to access the application or see any traces, verify the following:

Check Agent Health logs in your terminal for any errors. Also check whether Agent Health is running on an alternative port, like 4002 instead of 4001.
If there are permission errors when accessing traces from OpenSearch, verify that the AWS credentials in your terminal match the principal (user or role) that you specified under the agentHealthReaderArn CDK parameter during cdk deploy. This principal must have ESHttpGet:* IAM permissions. Agent Health uses your current AWS credentials to access the OpenSearch API for querying traces. The OpenSearch API is guarded by both IAM and OpenSearch fine-grained access control.

Create and run a test

Choose Test Cases and New Test Case. Fill out the required fields with the following information:

Name: Should add to cart.
Initial Prompt: Add some wireless headphones to my cart. Take any that you have in stock.
Expected Outcomes: PROD-001 added to cart.

Back in the test cases overview, select the created test case and choose Run Test. In the Configure Run dialog, choose Retail Assistant (production) for Agent, Tool Usage Efficiency for Evaluator, Claude Opus 4.8 for Judge Model, and choose Start Run.

Agent Health now runs the configured initial prompt against the agent. The agent completes the task and sends execution traces to OpenSearch. Agent Health uses an evaluation model to check both agent responses and traces on successful execution, according to the defined expected outcomes. After the test is completed, go through the different tabs to check the test results.

Agent Health evaluation report showing test results across multiple tabs

Figure 4: Agent Health evaluation report

If you’re unable to run the test, check the following:

Agent Health automatically creates an Amazon Cognito token for your user upon start, but this token can expire. Restarting Agent Health creates a new token. Verify that both the COGNITO_EMAIL and COGNITO_PASSWORD variables are still set in your terminal environment.

Beyond test cases

After running a single test case, choose Benchmarks in the sidebar menu. With Benchmarks, you can run multiple test cases in parallel and summarize their results. You can compare benchmark runs by choosing Evaluation Runs in the sidebar, where you can analyze trends in pass rate, cost, and duration over time. Lastly, choose Evaluators to define your own evaluation logic beyond the predefined ones.

You can also run Agent Health tests with its command-line interface, which is handy for automation and continuous integration (CI). The equivalent command of running the preceding test is:

npx @opensearch-project/agent-health run -t <TEST_CASE_ID> -a "Retail Assistant" -e system-tool-usage --judge-model claude-opus-4.8 -e system-tool-usage

where TEST_CASE_ID can be retrieved from the browser URL when you visit the Agent Health UI (test case IDs start with tc-).

Agent Health stores all test cases, other configuration, and reports locally on disk in the agent-health/agent-health-data directory.

Cleaning up

To avoid incurring future charges, delete the resources:

cd ../infra && cdk destroy -c agentHealthReaderArn=$AGENT_HEALTH_READER_ARN

Conclusion

In this post, you learned to set up and use OpenSearch Agent Health for production agent observability and evaluations. To discover more features, see the Agent Health documentation pages. You can discuss and request additional features, and get help with setup, through the issues in the GitHub project. For more information, see the observability documentation for Amazon OpenSearch Service, where you can learn about the features available to build observability for both agents and traditional systems using OpenSearch. To investigate issues in production AI agents, see the recent post Unified observability in Amazon OpenSearch Service.

About the authors

Accelerate Apache Spark debugging on Amazon EMR with AWS DevOps Agent

2026-09-01 Kalyan Janaki

Post Syndicated from Kalyan Janaki original https://aws.amazon.com/blogs/big-data/accelerate-apache-spark-debugging-on-amazon-emr-with-aws-devops-agent/

When an Apache Spark job fails on Amazon EMR, the root cause can hide in executor logs, memory profiles, or application code. As data pipelines grow in complexity, correlating logs, metrics, and traces across multiple services requires significant operational effort. AWS DevOps Agent handles this investigation autonomously while keeping operators in the loop to review findings and approve fixes. From a single chat prompt, it produces a root cause and mitigation plan, often without any human involvement beyond the initial question.

The native AWS API tools in AWS DevOps Agent don’t extend into Spark-internal artifacts. Sometimes those tools can’t reach the evidence that pins down the root cause: a Spark History Server event log, executor Python worker memory, or a line of code that allocated too much. In these cases, AWS DevOps Agent can describe symptoms (“the executor exited with code 1”) but can’t identify the actual antipattern that caused them.

This post shows how to extend AWS DevOps Agent to investigate failures in Apache Spark workloads on Amazon EMR. You register the Apache Spark Troubleshooting Agent for Amazon EMR, a managed Model Context Protocol (MCP) server hosted by AWS, as a custom capability provider in your AWS DevOps Agent space. You route the traffic over AWS PrivateLink so MCP calls never traverse the public internet. Then you watch a single agent chat session investigate a deliberately failing Spark job, from Amazon CloudWatch alarm to line-numbered root cause, in about two minutes.

Prerequisites

Before you begin, make sure you have the following:

An AWS account with permissions to deploy AWS CloudFormation stacks that create AWS Identity and Access Management (IAM) roles and Amazon Virtual Private Cloud (Amazon VPC) resources.
The latest version of the AWS Command Line Interface (AWS CLI), Boto3, and Botocore, installed and configured with credentials for the same account and AWS Region.
This walkthrough assumes you are familiar with AWS DevOps Agent and know how to trigger an investigation. For an introduction, see Getting started with AWS DevOps Agent.

How AWS DevOps Agent discovers custom tools through MCP

Model Context Protocol (MCP) is an open standard that defines how AI agents discover and invoke external tools. AWS DevOps Agent supports connecting to custom MCP servers, which means you can expose new capabilities to it without modifying the agent itself. When you connect an MCP server to AWS DevOps Agent, the agent automatically discovers the available tools, understands their schemas, and calls them as part of its investigation workflow. You build and connect the MCP server, and the agent handles the rest.

MCP tools sit alongside the agent’s built-in AWS API tools. During a single investigation, the agent can interleave calls to cloudwatch.describe-alarms, emr-serverless.get-job-run, and a custom MCP tool such as analyze_spark_workload. The agent picks the right one for each subtask. You augment the agent’s reach without replacing what it already does.

For this integration, you don’t build an MCP server. The Apache Spark Troubleshooting Agent for Amazon EMR is itself a managed MCP server, hosted by AWS at a regional endpoint. Your job is to register that endpoint with AWS DevOps Agent and authorize the agent to call it. This requires a network path from the agent to the endpoint, plus an IAM role for AWS Signature Version 4 request signing.

Why Spark internals visibility matters

The actual root cause for a Spark failure usually lives somewhere none of those APIs (such as Amazon CloudWatch Logs Insights, AWS CloudTrail, or Amazon EMR step-status calls) can reach:

The Apache Spark Troubleshooting Agent for Amazon EMR reads the following sources.

The Spark History Server event log is a per-job archive in Amazon Simple Storage Service (Amazon S3) with stage timings, task-level metrics, executor utilization, shuffle read/write volumes, and garbage-collection pauses. Amazon EMR exposes this data through the Spark UI on Amazon EMR Serverless, Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2), and Amazon EMR on Amazon Elastic Kubernetes Service (Amazon EKS), but interpreting signals like data skew, executor memory pressure, or stages that take significantly longer than expected requires familiarity with Spark internals.

The Spark query plan — the logical and physical plan the driver compiled. Without it, you can’t identify antipatterns such as unnecessary data repartitioning or missing broadcast hints that trigger expensive shuffles.
The application source code in Amazon S3 — the .py or .jar code artifact the job ran. Without it, you can’t quote the offending line of a mapPartitions user-defined function or an inefficient collect().
The Python worker process telemetry — the PySpark worker is a separate Python subprocess outside the Java Virtual Machine’s (JVM) managed memory. When it crashes from spark.executor.pyspark.memory exhaustion, the JVM driver sees a generic “executor exited unexpectedly” message. The actual cause is invisible to standard JVM-level logs.

When the agent invokes analyze_spark_workload during an investigation, it returns a structured analysis with the antipattern identified at the line level, the offending stage isolated, and a concrete fix: both code changes and configuration changes.

Integrating AWS DevOps Agent with Apache Spark Troubleshooting MCP

This section explains how AWS DevOps Agent connects to the Apache Spark Troubleshooting Agent through a private MCP endpoint and orchestrates the investigation workflow.

How it works

Architecture diagram showing AWS DevOps Agent connecting to the Apache Spark Troubleshooting Agent over AWS PrivateLink

Figure 1: Integration architecture between AWS DevOps Agent and the Apache Spark Troubleshooting Agent for Amazon EMR over AWS PrivateLink

You submit an investigation prompt in AWS DevOps Agent.
AWS DevOps Agent sends a SigV4-signed MCP call into your Amazon VPC through the AWS DevOps Agent private connection.
The private connection forwards the request to the Interface VPC Endpoint.
The endpoint routes the request over AWS PrivateLink to the Apache Spark Troubleshooting Agent for Amazon EMR, which AWS manages.
The MCP service reads from your data sources (Amazon EMR, Amazon S3, Amazon CloudWatch Logs) using the same IAM role AWS DevOps Agent assumed for the call.
When a CloudWatch alarm transitions to ALARM state (for example, a failed-jobs alarm for your Amazon EMR Serverless application), AWS DevOps Agent automatically triggers an investigation without manual intervention.
AWS DevOps Agent decides which tools to call based on the prompt. For a Spark failure, that includes the Apache Spark Troubleshooting MCP server you registered as a capability provider.
Each MCP request is signed with AWS Signature Version 4 using the IAM role assigned to the capability provider. The request travels from AWS DevOps Agent into your Amazon VPC through the private connection. This private connection is a managed VPC Lattice resource gateway you created during setup.
From the resource gateway, the request flows to the Interface VPC Endpoint for the Amazon SageMaker Unified Studio MCP service, then on to the Apache Spark Troubleshooting Agent. The traffic stays entirely on the AWS network.
The MCP server reads the inputs it needs from your AWS account using the IAM role that you assigned to the capability provider during MCP server registration. This role grants access to the Spark History Server event log and application source code in Amazon S3, the driver and executor stdout streams in Amazon CloudWatch Logs, and the job-run metadata from Amazon EMR Serverless.
The MCP server returns its diagnostic findings to AWS DevOps Agent. The agent then analyzes the results, identifies the root cause, and presents recommended fixes both code-level and configuration-level in your chat.

Setting up the demo

As part of this demo, this post includes a sample AWS CloudFormation template, tested in the us-east-1 Region, that provisions the following resources for the walkthrough:

A dedicated Amazon Virtual Private Cloud (Amazon VPC) with two private subnets in Availability Zones supported by the Apache Spark Troubleshooting Agent for Amazon EMR.
An Interface VPC Endpoint for the Apache Spark Troubleshooting Agent for Amazon EMR.
An IAM role that AWS DevOps Agent assumes to invoke the Apache Spark Troubleshooting MCP server with AWS Signature Version 4.
A deliberately failing PySpark workload running on Amazon EMR Serverless, including the Amazon EMR Serverless application, the Spark execution role, and the demo logs stored in Amazon S3 bucket.
An Amazon CloudWatch alarm that fires when the demo job fails. This alarm is used as the trigger for the agent investigation later in this section.

Step 1: Clone the repository

Clone the git repository for the CloudFormation template, PySpark script, and Parquet data.

git clone https://github.com/aws-samples/sample-aws-data-processing-and-analytics.git

Step 2: Deploy the AWS CloudFormation stack

Deploy the template using the following AWS CLI command.

cd sample-aws-data-processing-and-analytics/blogs/devops-agent-spark-mcp-integration

aws cloudformation create-stack \
  --stack-name spark-troubleshooting-demo \
  --template-body file://cloudformation/spark-troubleshooting-devops-agent-blog.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-1

The stack reaches CREATE_COMPLETE in approximately 4–6 minutes. When it does, capture the following stack outputs, which you paste into the AWS DevOps Agent console in the next two steps:

DemoVpcId — the VPC ID for the AWS DevOps Agent private connection.
DemoSubnetIds — the two subnet IDs for the AWS DevOps Agent private connection.
SMUSVpcEndpointSecurityGroupId — the security group ID.
TroubleshootingRoleArn — the IAM role Amazon Resource Name (ARN).
MCPEndpointURL — the MCP endpoint URL to register.
FailedJobsAlarmName — the CloudWatch alarm name to reference in your investigation prompt.
DemoBucket — the S3 bucket name where you copy the demo script and Parquet data.

To retrieve all outputs at once, use the following AWS CLI command.

aws cloudformation describe-stacks \
  --region us-east-1 \
  --stack-name spark-troubleshooting-demo \
  --query "Stacks[0].Outputs" --output table

# Get your bucket name from the stack outputs
DEMO_BUCKET=$(aws cloudformation describe-stacks --stack-name spark-troubleshooting-demo --region us-east-1 --query 'Stacks[0].Outputs[?OutputKey==`DemoBucket`].OutputValue' --output text)

# Copy the script
cd sample-aws-data-processing-and-analytics/blogs/devops-agent-spark-mcp-integration
aws s3 cp scripts/customer_events_aggregator.py s3://$DEMO_BUCKET/customer_events_aggregator.py

# Copy the Parquet data
aws s3 cp data/ s3://$DEMO_BUCKET/data/ --recursive

Step 3: Create an agent space

The agent space defines which AWS account and Region the agent monitors, which IAM role it assumes, and which capability providers, including MCP servers, it can call.

Follow the steps in Creating an Agent Space in the AWS DevOps Agent User Guide. When completing those steps, use the following values:

Parameter	Value
Name	data-pipeline-troubleshooting
Region	us-east-1
Agent Space role	Choose Auto-create a new DevOps Agent role — the console generates a DevOpsAgentRole-AgentSpace* role with AIOpsAssistantPolicy attached
Optional integrations	Not required

After the agent space reaches Active status, proceed to create the private connection.

Step 4: Create the AWS DevOps Agent private connection

AWS DevOps Agent uses the private connection to reach into your Amazon VPC. Follow the steps in Connecting to privately hosted tools in the AWS DevOps Agent User Guide. You can use either the console or the AWS CLI command documented under Create a private connection.

When completing those steps, use the following values from your CloudFormation stack outputs:

Parameter	Value
Name	A descriptive name (for example, spark-private)
VPC	DemoVpcId from your stack outputs
Subnets	Both subnet IDs from DemoSubnetIds
Security group	SMUSVpcEndpointSecurityGroupId
TCP port ranges (Advanced configuration)	443
Host address (Service target details)	sagemaker-unified-studio-mcp.us-east-1.api.aws
DNS resolution	In VPC (private DNS)
Certificate public key	None

After the connection reaches Active status, proceed to Step 5.

Step 5: Register the Apache Spark Troubleshooting MCP server as a capability provider

With the private connection in place, register the MCP server as a capability provider. Follow the steps in Registering an MCP server at the account level in the AWS DevOps Agent User Guide.

When completing those steps, use the following values:

Parameter	Value
Name	spark-troubleshooting
Endpoint URL	MCPEndpointURL from your stack outputs
Connect to endpoint using a private connection	Selected

Step 6: Add the MCP server to the agent space

With the MCP server registered, you need a workspace where investigations run. The agent space defines which AWS account and Region the agent monitors, which IAM role it assumes, and which capability providers, including MCP servers, it can call.

In the MCP Server section, choose Add.

MCP Server section of the agent space detail page with an Add button

Figure 2: MCP Server section of the agent space detail page

In the Add a capability dialog, locate spark-troubleshooting in the list of registered MCP servers and choose Add.

Add a capability dialog listing the spark-troubleshooting MCP server

Figure 3: Add a capability with the spark-troubleshooting MCP server listed

On the Select MCP server tools page, both tools that the Apache Spark Troubleshooting Agent for Amazon EMR publishes are listed: analyze_spark_workload and analyze_spark_history_server_endpoint. Select both checkboxes, then choose Save.

Select MCP server tools page with both Spark troubleshooting tools checked

Figure 4: The Select MCP server tools page with both Spark troubleshooting tools selected

The agent space connects to the MCP server, lists its tools, and displays 2 Available / 2 Connected. Both tools are now part of your agent’s catalog.

MCP Server section showing spark-troubleshooting connected with two available and two connected tools

Figure 5: MCP Server section showing spark-troubleshooting connected with both tools available

Seeing it in action

To see the integration end to end, you submit a PySpark job, watch the CloudWatch alarm move to ALARM, and then ask AWS DevOps Agent to investigate using the alarm name.

The failing workload

The CloudFormation template provisioned an Amazon EMR Serverless application called analytics-events-platform and configured a sample PySpark job, customer_events_aggregator.py. The script simulates a common Python-side memory bug: a mapPartitions user-defined function accumulates 11 copies of every input row in an in-memory Python list before yielding results, while the job runs with spark.executor.pyspark.memory=256m. The Python worker process exceeds the 256 MB cap, the kernel kills it, Spark retries four times, and the stage is marked failed.

Submit the failing job

Run the DemoSubmitJobCommand from your stack outputs in your terminal. It looks like this:

aws emr-serverless start-job-run \
  --region us-east-1 \
  --application-id <DemoApplicationId> \
  --execution-role-arn <DemoExecutionRoleArn> \
  --name daily-customer-events-rollup \
  --job-driver '{"sparkSubmit":{"entryPoint":"s3://<DemoBucket>/customer_events_aggregator.py","entryPointArguments":["<DemoBucket>"],"sparkSubmitParameters":"--conf spark.executor.cores=2 --conf spark.executor.memory=1g --conf spark.executor.pyspark.memory=256m --conf spark.executor.instances=2"}}' \
  --configuration-overrides '{"monitoringConfiguration":{"s3MonitoringConfiguration":{"logUri":"s3://<DemoBucket>/logs/"}}}'

The command returns a jobRunId. Note it down. You will see it later in the agent’s investigation.

The job goes through PENDING to SCHEDULED to RUNNING to FAILED and reaches FAILED state in roughly four minutes.

Watch the CloudWatch alarm fire

The CloudFormation template also created a CloudWatch alarm named <DemoApplicationId>-FailedJobs (the exact name is in the FailedJobsAlarmName stack output). The alarm watches the FailedJobs metric in the AWS/EMRServerless namespace, scoped to your demo application, and flips to ALARM within a minute or two of the job failing.

Open the Amazon CloudWatch console, choose Alarms in the left navigation pane, and confirm the alarm is in In alarm state.

Amazon CloudWatch console alarm detail page showing the FailedJobs alarm in alarm state

Figure 6: The Amazon CloudWatch alarm detail page showing the FailedJobs alarm in the In alarm state

Ask AWS DevOps Agent to investigate

Open your AWS DevOps Agent space.
In the left navigation pane, choose Operator Access, then choose Incidents.
Choose Start an investigation.
Paste the following prompt, replacing <FailedJobsAlarmName> with the value from your stack outputs:

CloudWatch alarm in us-east-1 just went into ALARM state. Investigate why and recommend a fix

AWS DevOps Agent Start an investigation panel with the alarm prompt entered

Figure 7: AWS DevOps Agent Start an investigation panel with the Amazon CloudWatch alarm investigation prompt

The agent’s investigation chains together native AWS API tools and the Apache Spark Troubleshooting MCP tool you registered:

use_aws cloudwatch describe-alarms — fetches the alarm definition and reads its metric dimensions, identifying that the alarm is scoped to Amazon EMR Serverless application <DemoApplicationId>.
use_aws emr-serverless list-job-runs — finds the most recent FAILED job run on that application.
use_aws emr-serverless get-job-run — pulls the FAILED run’s metadata and last-known error.
spark-troubleshooting analyze_spark_workload — invokes the Apache Spark Troubleshooting Agent for Amazon EMR through the MCP capability provider, passing the application ID and job run ID. This is where the deep analysis happens.

Review the root cause and fix

When the investigation completes, AWS DevOps Agent presents the results across two tabs: Investigation timeline and Root cause.

The Investigation timeline shows every step the agent took: skills loaded, native AWS API calls made, and the moment it called the analyze_spark_workload MCP tool to analyze the failed Spark job. Each entry is expandable so you can audit the inputs and outputs.

Investigation timeline listing the agent tool calls and the MCP invocation

Figure 8: Investigation timeline tab showing the sequence of agent tool calls and the spark-troubleshooting MCP invocation

The Root cause tab is where the answer lands. It is organized into three sections that mirror what an experienced engineer would write in an incident report:

Root cause tab showing impact, root causes, and key findings for the memory failure

Figure 9: The Root cause tab showing the impact summary, identified root causes, and key findings for the Spark memory exhaustion failure

Impact — what failed, when, and for how long. For our demo, this calls out that the daily-customer-events-rollup job on the analytics-events-platform application failed with a MemoryError and that the alarm transitioned to ALARM state at the time of the failure.
Root causes — the actual antipattern. The agent identifies that customer_events_aggregator.py combines three compounding issues: an expand_event function (line 23) that amplifies each input row 11×, a repartition(1) that funnels all data into a single partition on a single executor, and a collect() (line 31) that pulls the amplified dataset back to the driver. All three run with only 1 GB of executor memory.
Key findings — supporting facts behind the diagnosis, including the executor memory configuration, the application’s maximum capacity, and how the agent confirmed each fact from the analyzed artifacts.

Both the antipattern identification and the supporting evidence come from artifacts the agent could only reach through the MCP tool: the application source code in Amazon S3, the Spark History Server event log, and the query plan. Without the Apache Spark Troubleshooting Agent for Amazon EMR plugged in, AWS DevOps Agent would have stopped at “the executor exited with a memory error.”

Clean up

To avoid ongoing charges, delete the resources you created. Some resources are managed by the AWS DevOps Agent console and must be removed there first. Otherwise, the CloudFormation stack deletion fails.

In the AWS DevOps Agent console, open your data-pipeline-troubleshooting agent space, choose the MCP Server section, select spark-troubleshooting, and choose Remove.
From the Agent spaces list, select data-pipeline-troubleshooting and choose Delete.
In Capability Providers, select spark-troubleshooting and choose Deregister.
In Capability Providers → Private connections, select smus-spark-private and choose Delete.
Delete the AWS CloudFormation stack. This removes the Amazon VPC, the Interface VPC Endpoint, the security group, the IAM role, the Amazon EMR Serverless application, the Spark execution role, the Amazon CloudWatch alarm, and the demo logs bucket.

aws cloudformation delete-stack \
  --region us-east-1 \
  --stack-name spark-troubleshooting-demo

Conclusion

In this post, you connected the Apache Spark Troubleshooting Agent for Amazon EMR to AWS DevOps Agent as a custom MCP capability provider. You kept the traffic on the AWS network with AWS PrivateLink, and ran a failing PySpark job to see the integration end to end. A CloudWatch alarm fired, you asked the agent to investigate, and a single chat session returned the root cause along with code and configuration fixes.

You can extend this pattern beyond the demo scenario. Consider connecting the MCP server to agent spaces that monitor your production Amazon EMR environment. Any Spark job that writes a History Server event log becomes diagnosable through the same workflow.

To continue learning, explore the following resources:

AWS DevOps Agent documentation — learn how to create agent spaces, configure integrations, and manage investigations.
Apache Spark Troubleshooting Agent for Amazon EMR setup guide — detailed prerequisites and configuration options for the MCP server.
Connecting MCP servers to AWS DevOps Agent — register additional MCP servers to expand your agent’s capabilities.
Sample code on GitHub — clone the CloudFormation template, PySpark script, and Parquet data used in this walkthrough.

If you’ve already integrated the Apache Spark Troubleshooting Agent into your operational workflow, or if you’re exploring other MCP-based extensions for AWS DevOps Agent, we want to hear about your experience. Share your thoughts and questions in the comments.

About the authors

Scheduling email campaigns at scale with Amazon EventBridge Scheduler

2026-09-01 Oluwaseun Ademuwagun

Post Syndicated from Oluwaseun Ademuwagun original https://aws.amazon.com/blogs/compute/scheduling-email-campaigns-at-scale-with-amazon-eventbridge-scheduler/

Scheduling email campaigns becomes more complex when you need to send email to millions of recipients at the unique time best suited for each customer. Consider these examples:

A flash sale might need to hit inboxes at 9 AM local time across every time zone.
A follow-up email (often known as a drip sequence) might need to send a second message exactly 3 days after the first message per subscriber.
A re-engagement campaign might target users who haven’t logged in for 30 days.

The scheduling requirements involve multiple considerations. You’re sending hundreds of millions of messages, each at its own optimal moment personalized to the recipient’s time zone and behavior.

In this post, we walk through how to use Amazon EventBridge Scheduler to personalize email notifications to each recipient. We create one schedule per recipient to deliver each email at its individually optimal moment, with zero idle compute cost. We also show how Amazon EventBridge Scheduler handles higher volumes. Amazon EventBridge Scheduler supports billions of schedules. By default, you have a quota of 10 million schedules.

Solution overview

When every recipient has their own ideal delivery time, you need a scheduling layer that can hold billions of individual send intents and fire each one at the right moment. Most teams reach for one of three familiar patterns, each with tradeoffs that become painful at scale.

Batch cron jobs: A job runs every hour, queries for all messages due in the next window, and sends them out. Recipients get email in imprecise hourly batches. At scale, the batch job itself becomes a bottleneck, processing millions of rows per run, competing for database connections, and creating a sudden spike in load on the email provider.
Delay queues: You can use Amazon Simple Queue Service (Amazon SQS) as a delay queue. A delay queue postpones the delivery of new messages to a customer for a set time. A limitation of this approach is that Amazon SQS caps delays at 15 minutes.
Third-party campaign tools: Offload to a SaaS email platform. This works until you need tight integration with your application data, custom send-time optimization, or control over delivery infrastructure. You’re also paying per-recipient fees that compound at scale.

All three approaches either sacrifice precision (batching), hit architectural limits (delay queues), or surrender control (third-party tools).

The building block approach

Amazon EventBridge Scheduler treats each email send as a discrete scheduled action. Instead of “process all messages due this hour,” you express the intent directly: “send this email to this person at this time.” Amazon EventBridge Scheduler holds that intent with zero compute cost until the moment arrives, then triggers the scheduled action. See the Amazon EventBridge Scheduler User Guide for the full API reference and current service quotas.

For email campaigns, Amazon EventBridge Scheduler becomes the send-time dispatcher, the component that schedules every email in a campaign for its individually optimal moment, whether that’s timezone-adjusted, behavior-triggered, or sequence-driven.

Architecture diagram

The architecture follows an event-driven, per-recipient scheduling pattern for an email campaign. To start the campaign, you first define the target audience and the content they receive. Next, you need a way to create the per-recipient schedule. To do that for a campaign that can contain millions of recipients, you need a scalable mechanism to create the schedules. You can achieve this with an AWS Step Functions state machine, a serverless workflow service that coordinates multiple AWS services into structured, visual workflows called state machines. In this solution, we orchestrate the creation of the schedules by using a Distributed Map state within the state machine, which lets us fan out and accelerate schedule creation. It does this by splitting a large dataset into chunks and processing them across thousands of parallel child executions. It reads the recipient list from Amazon Simple Storage Service (Amazon S3), applies time zone logic per recipient, and creates an individual Amazon EventBridge Scheduler resource for each recipient in parallel. After the workflow creates all schedules, the execution completes.

The actual email delivery happens later, entirely decoupled from the campaign creation step. At the scheduled time, Amazon EventBridge Scheduler invokes Amazon Simple Email Service (Amazon SES) directly, passing the template name and personalization data as template variables. For campaigns requiring complex personalization logic (conditional content, real-time suppression checks, or data enrichment), you can optionally route through an AWS Lambda function before SES. If you need to adjust timing or content for specific recipients, you can update their individual schedules directly without reprocessing the entire campaign.

Figure 1: Per-recipient email scheduling architecture with Amazon EventBridge Scheduler

Walkthrough

The solution uses four core components that work together: a campaign manager to define send-time rules, Step Functions Distributed Map to fan out and accelerate schedule creation, Amazon EventBridge Scheduler to hold each per-recipient intent and deliver through Amazon SES directly, and automatic cleanup through schedule self-deletion.

How it works

Create the campaign: A marketer defines the campaign: audience segment, email template, and send-time rules (for example, “9 AM in each recipient’s local time zone” or “24 hours before a Black Friday sale”).
Campaign manager fans out: An AWS Step Functions workflow uses Distributed Map to iterate over the recipient list and create one Amazon EventBridge Scheduler schedule per recipient per campaign step directly through SDK integration. Each schedule encodes the exact send time for that individual.
Amazon EventBridge Scheduler fires at the right moment: At each recipient’s scheduled time, Amazon EventBridge Scheduler invokes Amazon SES directly through a universal target, passing the template name and personalization data (recipient name and attributes) as template variables.
SES personalizes and sends: Amazon SES renders the email template with the provided data and delivers the message.
Schedule self-deletes: ActionAfterCompletion='DELETE' prevents the accumulation of spent schedules.

Prerequisites

To follow along with this walkthrough, you need the following:

AWS account and permissions: An active AWS account with permissions to create Amazon EventBridge Scheduler schedules, AWS Step Functions state machines, and Amazon SES identities, along with an AWS Identity and Access Management (IAM) role for Amazon EventBridge Scheduler to invoke Amazon SES.
Development environment: Python 3.13 or later, AWS SDK for Python (Boto3) version 1.26 or later, and AWS Command Line Interface v2 (AWS CLI v2).
Amazon SES configuration: Move your Amazon SES account out of sandbox mode to allow sending to arbitrary recipients.

Scaling the fan-out with Step Functions

For campaigns with millions of recipients, use AWS Step Functions Distributed Map to parallelize schedule creation. When you want to activate a campaign, you trigger a Step Functions workflow. This workflow fans out and creates schedules across the recipient list by using a Distributed Map with direct SDK integration. The direct SDK integration between Step Functions and Amazon EventBridge Scheduler lets each child execution call CreateSchedule directly. The following state machine definition reads recipients from an Amazon S3 CSV file and creates schedules in parallel:

{
  "Comment": "Fan out campaign schedule creation via direct SDK integration",
  "StartAt": "EnsureScheduleGroup",
  "States": {
    "EnsureScheduleGroup": {
      "Type": "Task",
      "Resource": "arn:aws:states:::aws-sdk:scheduler:createScheduleGroup",
      "Parameters": {
        "Name.$": "States.Format('campaign-{}', $.campaign_id)"
      },
      "ResultPath": null,
      "Catch": [
        {
          "ErrorEquals": [
            "Scheduler.ConflictException"
          ],
          "ResultPath": null,
          "Next": "FanOutRecipients"
        }
      ],
      "Next": "FanOutRecipients"
    },
    "FanOutRecipients": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "DISTRIBUTED",
          "ExecutionType": "STANDARD"
        },
        "StartAt": "BuildScheduleInput",
        "States": {
          "BuildScheduleInput": {
            "Type": "Pass",
            "Parameters": {
              "schedule_name.$": "States.Format('campaign-{}-{}', $.campaign_id, $.recipient.id)",
              "group_name.$": "States.Format('campaign-{}', $.campaign_id)",
              "schedule_expression.$": "States.Format('at({}T{}:00:00)', $.send_date_date, $.send_hour)",
              "timezone.$": "$.recipient.timezone",
              "target_input": {
                "FromEmailAddress": "[email protected]",
                "Destination": {
                  "ToAddresses.$": "States.Array($.recipient.email)"
                },
                "Content": {
                  "Template": {
                    "TemplateName.$": "$.template_id",
                    "TemplateData.$": "States.JsonToString($.recipient.attributes)"
                  }
                }
              }
            },
            "Next": "CreateSchedule"
          },
          "CreateSchedule": {
            "Type": "Task",
            "Resource": "arn:aws:states:::aws-sdk:scheduler:createSchedule",
            "Retry": [
              {
                "ErrorEquals": [
                  "Scheduler.SdkClientException"
                ],
                "IntervalSeconds": 2,
                "MaxAttempts": 3,
                "BackoffRate": 2
              }
            ],
            "Parameters": {
              "Name.$": "$.schedule_name",
              "GroupName.$": "$.group_name",
              "ScheduleExpression.$": "$.schedule_expression",
              "ScheduleExpressionTimezone.$": "$.timezone",
              "FlexibleTimeWindow": {
                "Mode": "FLEXIBLE",
                "MaximumWindowInMinutes": 5
              },
              "Target": {
                "Arn": "arn:aws:scheduler:::aws-sdk:sesv2:sendEmail",
                "RoleArn": "arn:aws:iam::976764934189:role/CampaignFanOutRole-dev",
                "Input.$": "States.JsonToString($.target_input)",
                "RetryPolicy": {
                  "MaximumEventAgeInSeconds": 7200,
                  "MaximumRetryAttempts": 5
                }
              },
              "ActionAfterCompletion": "DELETE"
            },
            "ResultPath": null,
            "End": true
          }
        }
      },
      "ItemReader": {
        "Resource": "arn:aws:states:::s3:getObject",
        "ReaderConfig": {
          "InputType": "CSV",
          "CSVHeaderLocation": "FIRST_ROW"
        },
        "Parameters": {
          "Bucket.$": "$$.Execution.Input.recipient_bucket",
          "Key.$": "$$.Execution.Input.recipient_key"
        }
      },
      "ItemSelector": {
        "campaign_id.$": "$$.Execution.Input.campaign_id",
        "template_id.$": "$$.Execution.Input.template_id",
        "send_date_date.$": "$$.Execution.Input.send_date_date",
        "send_hour.$": "$$.Execution.Input.send_hour",
        "recipient": {
          "id.$": "$$.Map.Item.Value.id",
          "email.$": "$$.Map.Item.Value.email",
          "timezone.$": "$$.Map.Item.Value.timezone",
          "attributes": {
            "first_name.$": "$$.Map.Item.Value.first_name",
            "signup_date.$": "$$.Map.Item.Value.signup_date"
          }
        }
      },
      "MaxConcurrency": 1000,
      "ResultPath": null,
      "End": true
    }
  }
}

Concurrency alignment with Amazon EventBridge Scheduler API limits

Step Functions Distributed Map supports up to 10,000 concurrent child workflows. Each child calls the CreateSchedule API directly, which has a default rate limit of 5,000 TPS. This limit is sufficient for most campaigns. If your campaign volumes require higher throughput, check your current quotas in the Service Quotas console and request an increase.

To avoid throttling, set MaxConcurrency below the CreateSchedule TPS quota. A value of 2,500 provides a comfortable buffer to account for bursts and retries without requiring a quota change. For larger campaigns, request an increase through AWS Service Quotas (adjustable to tens of thousands) and raise MaxConcurrency to match.

Canceling a campaign

A schedule group is an Amazon EventBridge Scheduler resource used to organize schedules. For this use case, we have a schedule group per campaign. If you need to pull a campaign (error in content, legal issue, or strategy change), you can cancel all scheduled sends for that campaign by deleting the entire schedule group. The following code shows how to cancel all pending sends for a campaign:

def cancel_campaign(campaign_id):
    """Cancel all pending sends for a campaign by deleting its schedule group."""
    scheduler.delete_schedule_group(
        Name=f'campaign-{campaign_id}'
    )

Operational considerations

Moving to production introduces a few scaling and reliability concerns to plan for.

Handling invocation spikes at delivery time

When a mass campaign schedules millions of messages for the same time, this creates cascading pressure across two limits:

Amazon EventBridge Scheduler invocations throttle limit: The default is 1,000 TPS per AWS Region, and it is adjustable to tens of thousands of TPS through AWS Service Quotas. Amazon EventBridge Scheduler queues invocations internally and retries with exponential backoff when the downstream target throttles.
Amazon SES sending quotas: Your SES account has a per-second sending rate. If the effective invocation rate exceeds this, messages fail with throttling errors. Align Amazon SES sending quotas with your campaign volume. Check your current SES quota in the Service Quotas console and request an increase before launching large campaigns. See Amazon SES best practices for deliverability at scale.

To handle an invocation spike, we recommend using the FlexibleTimeWindow feature of Amazon EventBridge Scheduler. Setting MaximumWindowInMinutes lets Amazon EventBridge Scheduler spread invocations across a time window rather than firing them all at the exact second. Size the window based on your campaign: divide the total schedules by your effective TPS to determine the minimum spread needed. For example, 500,000 schedules at 5,000 TPS need at least a 2-minute window.

Cost model

You pay for Amazon EventBridge Scheduler on a per-invocation basis.

Cleanup

To avoid ongoing charges, delete the resources created during this walkthrough:

Delete any runtime-created schedule groups.

aws scheduler delete-schedule-group --name campaign-<campaign-id>

Delete the Step Functions state machine.

aws stepfunctions delete-state-machine \
    --state-machine-arn arn:aws:states:us-east-1:<account-id>:stateMachine:CampaignFanOut

Note: If you have active schedules still waiting to fire, deleting the schedule group will cancel all pending sends.

IAM role for Amazon EventBridge Scheduler and Step Functions

The Step Functions state machine needs an execution role with permissions to create schedules, send email, and pass the role to the Amazon EventBridge Scheduler service. Amazon EventBridge Scheduler needs permissions to call SES. The following policy shows the combined permissions for both scenarios:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowPassRoleToScheduler",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/CampaignFanOutRole",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": "scheduler.amazonaws.com"
        }
      }
    },
    {
      "Sid": "AllowSESSend",
      "Effect": "Allow",
      "Action": [
        "ses:SendEmail",
        "ses:SendTemplatedEmail"
      ],
      "Resource": "arn:aws:ses:<REGION>:<ACCOUNT_ID>:identity/[email protected]"
    },
    {
      "Sid": "DistributedMapExecution",
      "Effect": "Allow",
      "Action": [
        "states:StartExecution",
        "states:DescribeExecution",
        "states:StopExecution"
      ],
      "Resource": [
        "arn:aws:states:<REGION>:<ACCOUNT_ID>:stateMachine:CampaignFanOut",
        "arn:aws:states:<REGION>:<ACCOUNT_ID>:execution:CampaignFanOut:*"
      ]
    },
    {
      "Sid": "ReadRecipientsBucket",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::campaign-recipients-<ACCOUNT_ID>",
        "arn:aws:s3:::campaign-recipients-<ACCOUNT_ID>/*"
      ]
    },
    {
      "Sid": "CreateSchedules",
      "Effect": "Allow",
      "Action": "scheduler:CreateSchedule",
      "Resource": "arn:aws:scheduler:<REGION>:<ACCOUNT_ID>:schedule/campaign-*"
    },
    {
      "Sid": "CreateScheduleGroups",
      "Effect": "Allow",
      "Action": "scheduler:CreateScheduleGroup",
      "Resource": "arn:aws:scheduler:<REGION>:<ACCOUNT_ID>:schedule-group/campaign-*"
    },
    {
      "Sid": "PassRoleToScheduler",
      "Effect": "Allow",
      "Action": "iam:PassRole",
      "Resource": "arn:aws:iam::<ACCOUNT_ID>:role/SchedulerCampaignRole",
      "Condition": {
        "StringEquals": {
          "iam:PassedToService": "scheduler.amazonaws.com"
        }
      }
    }
  ]
}

This policy scopes the scheduler:CreateSchedule and scheduler:CreateScheduleGroup actions to resources prefixed with campaign-*, following least-privilege principles.

A condition restricts the iam:PassRole permission so that it can only pass the role to the Amazon EventBridge Scheduler service.

Conclusion

In this post, we walked through how to use Amazon EventBridge Scheduler to personalize email campaign delivery for each recipient. An email campaign system has two core problems: deciding what to send and deciding when to send it. Most teams over-engineer the “when” with polling infrastructure, batch jobs, and queue chains. Amazon EventBridge Scheduler collapses that into a single CreateSchedule API call per recipient.

To get started, explore Amazon EventBridge Scheduler on the AWS Management Console. Browse Serverless Land patterns for more than 20 Amazon EventBridge Scheduler patterns and other use cases beyond email campaigns.

Suggested tags: Amazon EventBridge, architecture, events, modernization, serverless.

Measuring and improving search quality with Amazon OpenSearch Service

2026-09-01 Aruna Govindaraju

Post Syndicated from Aruna Govindaraju original https://aws.amazon.com/blogs/big-data/measuring-and-improving-search-quality-with-amazon-opensearch-service/

Search is the front door of many applications, yet most teams struggle to answer a deceptively simple question: “Is my search actually returning relevant results?” Query logs tell you what users typed, not what they saw, what they selected, or why they left. When search feels broken, the culprit is rarely the engine. It’s the lack of deliberate signal collection, measurement, and a feedback loop to act on it.

You can close this gap on Amazon OpenSearch Service using User Behavior Insights (UBI), an open schema standard for capturing search behavior, and Search Relevance Workbench (SRW), a toolkit for measuring and evaluating search quality. Your application generates the UBI-formatted records. Together, UBI and SRW give you a repeatable framework: collect signals, turn them into relevance judgments, and validate every change before it ships.

In this post, we show you how to capture UBI data on an Amazon OpenSearch Service domain and use those signals to evaluate search quality. This is the first post in a two-part series. We build the foundation here, and Part 2 covers automating the workflow end to end.

The challenge: You can’t improve what you can’t measure

Consider a shopper searching for “handbag” on an ecommerce site. The catalog has 16 products (tote bags, duffel bags, laptop bags), but every title only says “bag.” The search returns zero results. Most shoppers leave. A patient one retries with “bag” and finds what they were looking for.

Your server log recorded that first query as a clean sub-second response: no error, no alert, no signal. What it missed entirely was a customer with purchase intent. That customer hit a vocabulary gap between how they search and how you write your catalog. Zoom out and apply this lens to misspelled queries, poor handling of long-tail searches, and abandoned sessions. The blind spot is larger than you think.

There’s a second problem: click signals are position biased. Users select the first result far more than the fifth, regardless of relevance, so raw click counts reflect where results appeared, not whether they deserved to be there. Any judgment derived from clicks must correct for this bias. We return to it when generating judgments.

Capturing behavioral data with UBI

UBI defines two indices. The ubi_queries index holds one record per executed query: the text the user typed, the full query that ran (filters and facets included), and the IDs of the documents returned. The ubi_events index holds every subsequent user action: impressions, hovers, clicks, add-to-carts, each stamped with the result position and the product’s business identifier (object_id). A shared query_id links every event back to the query that triggered it. Two additional identifiers complete the picture: client_id tracks the browser across visits, and session_id scopes events to a single visit.

A query record captures what the user asked and which document IDs the engine returned, including zero-result cases like the handbag search, which appears as a record with an empty result list. Here’s the shopper’s follow-up search for “bag”:

{
  "query_id": "1bf736d4-d673-4763-9193-4bc8a2282115",
  "client_id": "9a9968ac-664b-42d7-9a9e-96f412b5ab49",
  "user_query": "bag",
  "query": "{"multi_match": {"query": "bag", "fields": ["title", "description", "category", "brand"]}}",
  "query_response_hit_ids": [
    "3760170840499",
    "8400000000042"
  ],
  "timestamp": "2026-07-23T07:53:35.264Z",
  "application": "retail-shop"
}

The UBI queries schema reference documents the complete query schema, including the mandatory attributes.

The event record captures what the user did next. For each result rendered, emit an impression event. When the user selects a result, emit a click event. Here is the impression event for the first result of the bag search:

{
  "action_name": "impression",
  "query_id": "1bf736d4-d673-4763-9193-4bc8a2282115",
  "client_id": "9a9968ac-664b-42d7-9a9e-96f412b5ab49",
  "session_id": "0f2e6f2a-8f4e-4f60-9f6e-2a1b3c4d5e6f",
  "user_query": "bag",
  "timestamp": "2026-07-23T07:53:41.112Z",
  "event_attributes": {
    "position": {
      "ordinal": 1
    },
    "object": {
      "object_id": "3760170840499",
      "object_id_field": "object_id"
    }
  }
}

event_attributes also accepts custom fields of your own alongside the standard position and object structures. The action_name attribute is critical: The judgment model you use later consumes only impression and click events. Treat a paginated results page as the same logical query: reuse the query_id and record absolute positions. The UBI events schema reference documents the complete event schema.

Collecting UBI data on Amazon OpenSearch Service

Behavioral data (what results ranked, what users saw, what they selected) exists only in the application layer. Your application owns the records, and Amazon OpenSearch Ingestion (OSI), a fully managed, serverless data collector powered by Data Prepper, provides the managed delivery path. Your application sends the records as SigV4-signed HTTP POST requests to the OSI pipeline endpoints. Route browser events through your backend for signing. One thing to understand before you write any code: Your application generates and owns the query_id attribute. The application creates the ID when it runs a search and stamps it on every subsequent event the user produces, until the user issues a new search or the session ends.

Prerequisites

To follow along, you need an Amazon OpenSearch Service domain running OpenSearch 3.5 or later with the OpenSearch UI application, permissions to create OpenSearch Ingestion pipelines with an AWS Identity and Access Management (IAM) pipeline role, and a search application you can instrument to emit behavioral records.

Create the UBI indices

Before you start collecting user metrics, you need the two indices in place with the right mappings. Field types matter here: query_id as keyword supports exact joins between queries and events, timestamp as date supports time-range queries, and event_attributes as dynamic means you can extend events with custom fields without schema changes.

Create ubi_queries first in Dev Tools. It holds the query-side records. We abbreviated the mappings here. Refer to the published queries-mapping.json file for the complete version:

PUT ubi_queries
{
  "mappings": {
    "properties": {
      "query_id": { "type": "keyword" },
      "client_id": { "type": "keyword" },
      "user_query": { "type": "keyword" },
      "query_response_hit_ids": { "type": "keyword" },
      "timestamp": {
        "type": "date",
        "format": "strict_date_time"
      },
      "application": { "type": "keyword" }
    }
  }
}

Then create ubi_events. It holds every user action that follows (refer to the full events-mapping.json file):

PUT ubi_events
{
  "mappings": {
    "properties": {
      "query_id": { "type": "keyword", "ignore_above": 100 },
      "action_name": { "type": "keyword", "ignore_above": 100 },
      "client_id": { "type": "keyword", "ignore_above": 100 },
      "session_id": { "type": "keyword", "ignore_above": 100 },
      "user_query": { "type": "keyword" },
      "timestamp": {
        "type": "date",
        "format": "strict_date_time"
      },
      "event_attributes": {
        "dynamic": true,
        "properties": {
          "position": {
            "properties": {
              "ordinal": { "type": "integer" }
            }
          },
          "object": {
            "properties": {
              "object_id": { "type": "keyword" },
              "object_id_field": { "type": "keyword" }
            }
          }
        }
      }
    }
  }
}

With both indices created, the next step is routing data into them. You can deliver UBI data to your domain in several ways. This post uses OSI pipelines, shown end to end in the diagram that follows the setup.

Set up the OSI pipelines

Create two OSI pipelines: one for queries and another for events. Each pipeline exposes an HTTP source endpoint that your application writes to (shown on each pipeline’s console page) and sinks data to the corresponding index. The following configuration defines the events pipeline:

version: '2'
ubi-events:
  source:
    http:
      path: /ubi/events
      max_request_length: 10mb
  processor:
    - date:
        from_time_received: true
  sink:
    - opensearch:
        hosts: ["https://<domain-endpoint>"]
        aws:
          serverless: false
          region: <region>
          sts_role_arn: <pipeline-role-arn>
        index_type: custom
        index: ubi_events
    - s3:
        aws:
          region: <region>
          sts_role_arn: <pipeline-role-arn>
        object_key:
          path_prefix: 'ubi_events/%{yyyy}/%{MM}/%{dd}'
        bucket: <bucket-name>
        threshold:
          maximum_size: 50mb
          event_collect_timeout: 60s
        codec:
          ndjson:

Note: the queries pipeline follows the same pattern, with /ubi/queries as the path and ubi_queries as the sink index and S3 prefix. Create the pipeline role yourself or let OpenSearch Ingestion create it. If your domain uses fine-grained access control, also map the pipeline role to a backend role so the domain accepts the pipeline’s writes. Refer to the tutorial Collecting UBI-formatted data in Amazon OpenSearch Service for detailed steps.

With the pipelines running, your application can start sending data. The following diagram illustrates the end-to-end flow:

UBI collection flow from the search application through OpenSearch Ingestion into the ubi_queries and ubi_events indices

Figure 1: The UBI collection pattern on Amazon OpenSearch Service

The workflow consists of the following steps:

Users interact with your search application.
The application sends signed query records to the OSI HTTP endpoint.
OSI writes queries to the ubi_queries index.
Users interact with the results, viewing and selecting documents.
The application sends signed event records, carrying the same query_id, to the OSI HTTP endpoint.
OSI writes events to the ubi_events index.
Optionally, both pipelines archive records to Amazon Simple Storage Service (Amazon S3).
Search Relevance Workbench (OpenSearch UI) works with the collected data in the ubi_queries and ubi_events indices.

Note: if you’re already collecting site analytics through an existing third-party tool, you don’t need to replace it. Map your search-related events (queries, clicks, and conversions) into the UBI schema and store them in OpenSearch. That’s enough to unlock the out-of-the-box evaluation framework, implicit judgment generation, and the full SRW metrics pipeline, without defining a single custom metric from scratch.

Visualize the data collected

After the UBI behavior metrics start to trickle in, you can review the data in the Discover tab on the OpenSearch UI dashboard. Filtering ubi_queries for empty result lists ranks your vocabulary gaps. You can also visualize the data collected through the sample User Behavior Insights (UBI) dashboards in OpenSearch.

OpenSearch Discover view of UBI records for the zero-result handbag query and the follow-up bag query

Figure 2: UBI records in Discover, showing the zero-result handbag query and the follow-up bag query with its impressions and pagination events

With data flowing into your indices, keep these things in mind as you scale to production:

Keep telemetry off the search critical path – Queue records and forward them asynchronously. Losing a fraction of behavioral data is statistically harmless. Blocking users isn’t.
Manage volume deliberately – Batch impression events, and if you sample, sample whole queries rather than individual events to preserve the click-through ratios that drive judgments.
Isolate analytical load for larger deployments – Route pipelines to a separate analysis domain with the same engine version, mappings, and analyzers as production. This keeps behavioral writes from touching live search latency.
Plan for retention and integrity – Register the UBI mappings as an index template and apply an Index State Management (ISM) retention policy as your indices grow. You should validate and rate-limit the event write path, and cover query text and client identifiers with your data retention policy.

Evaluating search quality with Search Relevance Workbench

With ubi_queries and ubi_events collecting data, you now have the signals needed to evaluate search quality. Search Relevance Workbench, generally available in the OpenSearch UI from Amazon OpenSearch Service 3.5, turns those signals into structured experiments: comparing query configurations, scoring results against relevance judgments, and surfacing metrics that guide iterative tuning.

The Search Relevance Workbench home screen in the OpenSearch UI

Figure 3: Search Relevance Workbench in the OpenSearch UI

SRW experiments rely on three components. You set them up once, then reuse them across every experiment you run: a query set (the fixed queries you evaluate against), search configurations (the query structures you want to compare), and a judgment list (the relevance ground truth). The following sections walk through each one.

Step 1: Create a query set

A query set is the fixed collection of queries you evaluate against. Keeping it fixed makes results comparable across experiments. Effective query sets reflect real traffic, not intuition. You can seed one from your top queries, a random sample, or a hand-picked mix that includes long-tail and low-performing queries. Alternatively, SRW can sample directly from ubi_queries using Probability-Proportional-to-Size (PPS) sampling, which selects queries in proportion to how often users issue them. This approach represents frequent queries like “bag”, so your metrics reflect search quality as users experience it.

Query set creation screen sampling queries from real traffic in the ubi_queries index

Figure 4: Creating a query set sampled from real traffic in ubi_queries

Step 2: Define search configurations

A search configuration defines how a search executes: the index, the query structure, and a %SearchText% placeholder that SRW replaces with each query in your set. Creating two configurations and running them against the same query set and judgment list is how you validate a change before any user sees it.

As an example, here we define two configurations: a baseline multi_match query (retail_query) and a variant that boosts title matches (retail_boosted_query), so we can measure whether the boost actually helps ranking.

retail_query retail_boosted_query

{
  "query": {
    "multi_match": {
      "query": "%SearchText%",
      "fields": [
        "title",
        "description",
        "category",
        "brand"
      ]
    }
  }
}

{
  "query": {
    "multi_match": {
      "query": "%SearchText%",
      "fields": [
        "title^2",
        "description",
        "category",
        "brand"
      ]
    }
  }
}

Configurations go beyond query variants: a candidate can be an entirely different retrieval strategy, like hybrid search combining keyword and neural retrieval. You can use judgments to rate query-document pairs independently of your retrieval approach. You can test a semantic or hybrid approach offline against your existing traffic before shipping it.

Step 3: Create the judgment list

A judgment is a relevance rating for a query-document pair: the ground truth that quality metrics measure against. You can create judgments that are explicit (from stakeholders or a large language model acting as judge), imported, or implicit (derived from behavior). Here we use implicit judgments derived from UBI selection behavior, scored using the Clicks Over Expected Clicks (COEC) model. The COEC model helps correct position bias by comparing each document’s actual click rate against the expected rate for its rank position. Documents that outperform their position score as relevant. Those that users select because they ranked first score near average.

Judgment list creation screen with the Implicit click-based type and COEC click model selected

Figure 5: Creating an implicit judgment list with the Implicit (Click based) type and the COEC click model

Three things to get right before you run experiments:

object_id in your events must match the document _id from your product catalog. The search configurations you define return this _id, which lets SRW join judgments to results.
Implicit judgments are statistical. They need volume and query coverage. As a working rule of thumb, aim for hundreds to thousands of real sessions per query to separate signal from noise.
Max Rank controls how deep in the result list events count. If users paginate, set it beyond a single page. We use 20 here.

Step 4: Run experiments

This post uses three SRW capabilities: Query Analysis, Query Set Comparison, and Search Evaluation. Query Analysis is a quick eyeball check: compare two configurations side by side for a specific query to see exactly what changed and why the metrics moved. The other two answer harder questions with numbers: how good a configuration is, and how two configurations compare against real relevance signals.

Query Set Comparison (also called pairwise comparison) takes two configurations and computes ranking similarity. Jaccard overlap measures how much the two result lists share, while Rank-Biased Overlap (RBO) weights agreement at the top of the list more heavily. Near-identical scores mean the change will barely register with users. Low overlap means a real ranking shift worth reviewing carefully before shipping. In this run, the two configurations score 0.93 Jaccard and 0.92 RBO, a modest but real shift. SRW cannot score zero-result queries like “handbag”: They show zero similarity in a comparison and Failed in an evaluation, a signal they need a different fix than ranking adjustments.

Query Set Comparison results showing Jaccard and Rank-Biased Overlap scores for the two configurations

Figure 6: Query Set Comparison showing Jaccard and Rank-Biased Overlap between the two configurations

Search Evaluation (also called pointwise evaluation) scores one configuration against your query set and judgment list across four metrics, each computed over the top k results (k=10 by default):

Metric	What it measures	What it tells you
Coverage@k	Proportion of returned documents that have judgments	How much to trust the other three metrics. Low Coverage means many results were never judged
Precision@k	Fraction of the top k results that are relevant	How many irrelevant results appear on the first page
MAP@k (Mean Average Precision)	Precision averaged across ranks, rewarding relevant documents placed early	Whether relevant results appear early, even when Precision ties
NDCG@k (Normalized Discounted Cumulative Gain)	Graded judgment values, discounted by position (rank 1 counts more than rank 9)	Whether the best results appear first. The primary comparison metric

Each pointwise experiment evaluates one configuration. To compare candidates, run one experiment per configuration and compare the results. In this run, the baseline (retail_query) scores Coverage@10 of 1.0, Precision@10 of 1.0, MAP@10 of 0.95, and NDCG@10 of 0.93, with the zero-result “handbag” query showing as Failed in the per-query detail.

Search evaluation results showing Coverage, Precision, MAP, and NDCG at 10 with per-query detail

Figure 7: Search evaluation results for one configuration: Coverage, Precision, MAP, and NDCG at 10, with per-query detail

From measurement to improvement

The preceding experiments are the harness. The following are common levers to test with it. Express each as a new search configuration, evaluate it against the same query set and judgment list, and adopt it only if the metrics move:

Synonyms – One option for addressing known vocabulary gaps is to build synonyms. A search-time synonym token filter treats “handbag” and “bag” as equivalent, and with Amazon OpenSearch Service, you can hot deploy custom synonym packages without reindexing.
Field weights – Adjust the fields and boosts in a multi_match query, like the title^2 variant tested earlier.
Semantic retrieval – A hybrid query combines keyword and neural scores, addressing vocabulary mismatch as a class rather than term by term. Judgments evaluate it offline exactly like a lexical candidate.
Reranking – A rerank processor in a search pipeline reorders the top results using a cross-encoder model.

Clean up

To avoid future charges, delete the resources you created for this walkthrough:

Delete the two OpenSearch Ingestion pipelines. To reuse them later, stop them instead. A stopped pipeline keeps its configuration and incurs no OpenSearch Compute Unit (OCU) hour charges.
If you configured the optional Amazon S3 archive, delete the archived objects (or the bucket).
If you keep the domain, optionally delete the ubi_queries and ubi_events indices and the query sets, judgment lists, and experiments you created. These live on the domain and incur no separate charges.
If you created the domain specifically for this post, delete it to remove everything, including the resources in the previous step. Deleting a domain is irreversible. Don’t delete a domain that serves other workloads.

Conclusion

UBI collects the evidence, COEC turns it into judgments, and SRW experiments deliver the verdict: Coverage, Precision, MAP, and NDCG in place of guesswork. Ship the winning configuration, keep collecting, and the next round of judgments shows whether the improvement holds with real behavior. Where there used to be an opinion, there is now a number.

Everything here follows a repeatable pattern, and repeatable patterns lend themselves to automation. Part 2 walks through the Search Relevance Agent, available through the AI Assistant chat (the Ask AI button) in the OpenSearch UI. The agent analyzes your UBI signals, generates tuning hypotheses, and validates them offline before recommending changes. The pipeline you built in this post is the foundation. Stay tuned for Part 2.

To go deeper on the evaluation features, refer to the Search Relevance Workbench documentation.

About the authors

Automate IAM Identity Center governance with continuous discovery and reporting

2026-08-31 Jonathan Nguyen

Post Syndicated from Jonathan Nguyen original https://aws.amazon.com/blogs/security/automate-iam-identity-center-governance-with-continuous-discovery-and-reporting/

AWS IAM Identity Center integrates with external identity provider (IdP) to provide customers with a centralized authentication and authorization solution for AWS resources across AWS Organizations. AWS continues to invest into IAM Identity Center with a growing number of AWS services that natively integrate with IAM Identity Center. As your AWS organization scales, maintaining visibility into who has access to which applications and enforcing governance policies across accounts and Regions becomes increasingly complex. Identity Center helps address this by centralizing authentication and authorization for AWS resources across your organization, integrating with your external identity provider and a growing number of AWS services. However, as adoption scales, tracking access assignments and enforcing governance policies consistently becomes its own challenge.

This blog post focuses on planning your integration between an identity provider and IAM Identity Center for managed applications in your organization. We also walk through deploying and using an automated Identity Center discovery and reporting sample solution to help answer the governance and security questions:

Which users or groups have access to which AWS applications?
Who last accessed a specific AWS application and when?
Which users and groups are assigned to which IAM Identity Center applications across organization and AWS Regions?
How can you quickly generate reports to assist with compliance audits or security reviews?

The sample solution will identify associated AWS applications and the corresponding user and group assignments for the IAM Identity Center instances within your organization. The output is stored in a queryable format and generates CSV files for downstream analysis or reporting.

Plan identity governance for Identity Center application assignments

There are four key areas to start on when planning how to manage delegation and provisioning access across IAM Identity Center managed AWS applications. Bring together key stakeholders across security, governance, application, and business teams to make sure the implementation and integration will fit into the overall identity governance strategy.

Who can provision managed AWS applications: You can implement the IAM restrictions for creation of new AWS resources within AWS accounts in your organization. For example, if you restrict provisioning into a production AWS account to only infrastructure as code (IaC) IAM roles, you would continue implementing restrictions using AWS identity policies, service control policies (SCP), resource control policies (RCP), or IaC policy evaluation tools like Open Policy Agent (OPA) or Checkov.
Who manages user and group assignments: The managed application administrator handles authorization to managed applications within an AWS account. It’s recommended to clearly define roles and responsibilities across the workflow. You would have an IaC pipeline manage the integrated AWS resource provisioning with IAM Identity Center, then another workflow to allow requests to manage user and group membership for the managed application.
How authentication flows from the IdP to AWS resources: Users will authenticate into Identity Center, then be authorized to access AWS managed applications. From there, they will be authorized to access the associated AWS service and resources tied to the managed application. Depending on the AWS service, the associated downstream resources might have their own IAM principals that the users can access.
Mapping IdP identities to AWS resource access: There needs to be a link for workforce users and groups in your IdP, to Identity Center managed applications, and to downstream resources and permissions. Identifying the relationship will help you understand access within your AWS environment. Trusted identity propagation (TIP) is an additional feature of Identity Center that provides an end to end trail of the identity to the downstream service.

Create and manage an Identity Center application assignment lifeycle

As a security best practice, you should enable delegated administration when managing Identity Center within an AWS organization instances.

After you have IAM Identity Center set up within an organization instance, your member AWS accounts can start creating associated AWS resources. Within each member AWS account, the IAM principals that provision AWS resources will need two types of service-specific IAM permissions:

The first type of IAM permissions will be specific to the AWS service you want to provision. For example, to create an Amazon SageMaker AI domain, you would need the same IAM permissions to create the SageMaker AI domain and the downstream AWS resources SageMaker AI might use.
The second type of IAM permissions is specific to IAM Identity Center. The IAM principal used to create the resource, in this example SageMaker AI, will also need permissions to manage applications within the Identity Center instance.

{
	"Version": "2012-10-17",
	"Statement":
	[
		{
            "Effect": "Allow",
            "Action": [
                "sso:CreateManagedApplicationInstance",
                "sso:GetManagedApplicationInstance",
                "sso:DeleteManagedApplicationInstance",
                "sso:DescribeRegisteredRegions"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "sso:CreateApplication",
                "sso:DescribeApplication",
                "sso:DeleteApplication",
                "sso:PutApplicationGrant",
                "sso:PutApplicationAuthenticationMethod",
                "sso:PutApplicationAccessScope"
            ],
            "Resource": 
            [
                "arn:aws:sso::<INSERT-ACCOUNT-ID>:application/ssoins-<INSERT-INSTANCE-ID>/apl-*"
            ]
        }
    ]
}

IAM Identity Center application Amazon Resource Names (ARNs) follow a different standard naming convention that isn’t based on the original resource name that was provided during resource creation. For example, when a user creates an Amazon Simple Storage Service (Amazon S3) bucket and sets a specific bucket name, that bucket name is included in the ARN: arn:[partition]:s3:::[bucket-name]. Identity Center application ARNs use unique identifiers (GUIDs) generated at creation time.

Manage access for an Identity Center application

After the IAM Identity Center application is created, you will need to manage access to the Identity Center application and associated AWS resources. To continue with the SageMaker AI domain example, after the domain is created, an authorized IAM principal will need to assign Identity Center users or groups from the Identity Center instance to the domain. For Identity Center, you will need two types of Identity Center IAM permissions.

The first type of IAM permissions is used to list IAM Identity Center users and groups within the Identity Center instance. This is needed to read and select specific IAM users or groups to assign to an Identity Center application.

{
    "Version": "2012-10-17",
    "Statement":
    [
        {
            "Sid": "ListIdentityCenterUsers",
            "Effect": "Allow",
            "Action":
            [
                "identitystore:ListUsers",
                "identitystore:DescribeUser",
                "identitystore:ListGroups",
                "identitystore:DescribeGroup",
                "identitystore:ListGroupMemberships"
            ],
            "Resource": "*"
        }
    ]
}

Although IAM Identity Center users and groups have a GUID, the GUIDs aren’t clearly linked to the resource friendly names. For example, a group name could be Read-Only and the resource GUID could be 1234567890-abcdef12-3456-7890-abcd-ef1234567890 in the identity store. Additionally, the IAM actions to list users or groups require the AllUsers or AllGroups parameter. Because List actions require access to users and groups, a restrictive IAM policy can’t be used to prevent IAM principals from seeing a subset of users or groups within the identity store. The second type of IAM permission is used to create and manage application assignments for the Identity Center application within the Identity Center instance.

{
    "Version": "2012-10-17",
    "Statement":
    [
        {
            "Sid": "ManageApplicationAssignments",
            "Effect": "Allow",
            "Action": 
            [
                "sso:CreateApplicationAssignment",
                "sso:DeleteApplicationAssignment",
                "sso:ListApplicationAssignments",
                "sso:PutApplicationAssignmentConfiguration"
            ],
            "Resource":
            [
                "arn:aws:sso::<INSERT-ACCOUNT-ID>:application/ssoins-<INSERT-INSTANCE-ID>/apl-*"
            ]
        }
    ]
}

Because the IAM Identity Center application ARN is created using a unique application ID during creation, it’s not recommended to implement an IAM policy restricting authorized IAM principals to manage specific Identity Center applications. For example, to limit the application assignments to only a specific set of applications, you would need to:

Create the AWS resource with IAM Identity Center as the authentication mechanism
Query the Identity Center application ARN for the associated AWS resource
Identify the IAM principal that will be used for application assignments
Create or update an IAM policy associated to that IAM principal to allow application assignments for that specific application
Create or update an SCP to restrict application assignment to that specific IAM principal

In lieu of implementing resource restrictions within identity policies, you should limit management of IAM Identity Center application and application assignments to a limited number of authorized IAM principals. In addition, it is recommended to implement detective and reactive capabilities to manage Identity Center application assignments.

Plan your naming conventions and automation strategy

IAM Identity Center provides several APIs to capture information about your AWS organization instances, applications, and assignments. Before implementing automation or guardrails, you should develop a methodical approach and understand what outcome you’re working backwards from. Start by defining naming conventions and deciding what parts of the workflow you want to centralize.

Determine a naming convention for groups within your IdP: For example: AWS_<ACCT#>_<AWS_Service>_<LOB>_<ENV>_<AppName>. The IdP group name would look like: AWS_123412341234_SageMaker_Data_PROD_GTLabel.
Define the naming convention for AWS resources for your Identity Center integrated applications: For example: <AWS_Service>_<LOB>_<AppName>. The AWS resource name would look like: SageMaker_Data_GTLabel.
Define the naming convention for Identity Center application names: For example: <AWS_Service>_<LOB>_<ENV>_<AppName>. The Identity Center application name would look like: SageMaker_Data_PROD_GTLabel.
Decide on the restrictions that you want to implement within your AWS environment. Depending on your enterprise’s security standard, you can implement specific restrictions based on mapping of a similar combination of ENV (environment), AWS service, LOB (line of business), or application name.
Choose the portions of the application workflow that you want to centralize. This could include creating the application, making application assignments, or remediating issues.

As more configurations and permissions are centralized, additional overhead and bottlenecks can be introduced. It’s important to find the right balance for your enterprise. For example, if you centralize application assignments, each application team will need to submit a request to modify assignments that will be reviewed by a centralized team and could result in a delayed response. Conversely, if each application team handles their own assignments, there’s a risk that application assignments won’t align to enterprise security standards.

By understanding your goals and how you want to reach them, you can tailor the sample solution accordingly. Getting alignment on this requires planning and coordination across multiple teams within your organization. When thinking about more customized authorization logic—such as using provisioned AWS resource metadata—you should review how the specific AWS service integrates with IAM Identity Center managed applications. For example, if you want to find the Identity Center application ARN for a specific AWS resource, such as a SageMaker AI domain, use the following approach. A reverse lookup is necessary because AWS services create Identity Center applications with GUID-based ARNs that aren’t easily discoverable.

#!/bin/bash

DOMAIN_ID="d-xxxxxxxxxxxx"

REGION="xx-xxxx-x"

# Step 1: Get SageMaker domain details
echo "=== SageMaker Domain Details ==="
DOMAIN_INFO=$(aws sagemaker describe-domain \
--region $REGION \
--domain-id $DOMAIN_ID)

# Step 2: Extract Identity Center application ARN
SSO_APP_ARN=$(echo $DOMAIN_INFO | jq -r '.SingleSignOnApplicationArn')
echo "Identity Center App ARN: $SSO_APP_ARN"

IAM Identity Center automation sample solutions

The sample-iam-idc-application-discovery-reporting solution hosted on GitHub consists of two separate AWS CDK stacks:

IAM Identity Center governance reporting stack (/identity-center-reporting directory) – Provides automated discovery and report generation (using CSV files)
IAM Identity Center remediation stack (/identity-center-remediation directory) – Provides real-time enforcement and notifications

The recommendation is to deploy the reporting stack first to establish baseline visibility, then deploy the remediation stack for enforcement.

The following diagram depicts that IAM Identity Center governance architecture.

Figure 1: IAM Identity Center application assignment reporting architecture diagram

The reporting sample deploys the following resources:

Amazon EventBridge – Rule invokes the discovery workflow daily at 2:00 AM UTC (configurable)
AWS Step Functions – Orchestrates the multi-stage discovery workflow across instances, applications,and assignments
AWS Lambda – Takes the following actions:
1. Discovers IAM Identity Center instances across the organization and member accounts
2. Application discovery that enumerates the applications configured in each Identity Center instance
3. Assignment discovery maps users and groups to applications, resolving friendly names from the Identity Store
Amazon DynamoDB – Stores the discovered instances, applications, and assignments, encrypted with an AWS Key Management Service (AWS KMS) customer-managed key
Amazon API Gateway – Provides an IAM-authenticated REST API for a Lambda function to generate and export reports as CSV files
Amazon S3 – Stores the encrypted CSV file exports, with lifecycle policies and time-limited Amazon S3 presigned download URLs

Deploy the IAM Identity Center reporting sample

The following procedure deploys the automated discovery and reporting infrastructure using AWS Cloud Development Kit (AWS CDK). Make sure you have the following prerequisites in place, then continue with the steps to set up the solution.

Prerequisites

You need to have the following to test the solution in this post.

An AWS organization with an IAM Identity Center organization instance with delegated administrator access configured
IAM Identity Center configured with at least one instance
AWS Command Line Interface (AWS CLI) configured with appropriate credentials
Python 3.12 & Node.js 18 or later installed for CDK deployment

To deploy the IAM Identity Center reporting solution, run the following commands:

Clone the solution repository:

git clone https://github.com/aws-samples/sample-iam-idc-application-discovery-reporting
cd identity-center-reporting

Install dependencies:

python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Bootstrap the CDK (if not already done):

cdk bootstrap aws://<INSERT-ACCOUNT-ID>/<INSERT-REGION>

Deploy the sample solution:

export IDC_EXTERNAL_ID="$(uuidgen)" # alternatively you can set this value — member-account roles need the same value

cdk deploy --parameters AllowedIpRange=10.0.0.0/8 --parameters CrossAccountExternalId="$IDC_EXTERNAL_ID"

Note: AllowedIPRange is optional but recommended as a security best practice. The parameter will add a network restriction to download the Amazon S3 presigned URL export.

Optional: For AWS account-level Identity Center instance discovery, a cross-account IAM role is required.
python scripts/deploy-cross-account-roles.py --external-id "$IDC_EXTERNAL_ID"

Figure 2: Successful AWS CDK deployment of the reporting stack

After the stack is successfully deployed, obtain the CDK output values for the API Gateway URL and S3 bucket name. If using a command line to deploy, these values will be displayed after the stack successfully deploys. It can also be found in the AWS Management Console as AWS CloudFormation stack output. The output will be used for generating reports in the following sections.

Note that this stack is for the reporting stack only. Reactive monitoring and deployment are described in the next section.

After the reporting stack is successfully deployed, the automation will run on a daily schedule. The first discovery run executes immediately after deployment. You can monitor discovery execution history and detailed logs through the the AWS Step Functions console. Review the detailed Lambda function logs in Amazon CloudWatch Logs. Query discovered instances, applications, and assignments through the DynamoDB console for one-time analysis.

Generate reports for Identity Center application assignments

To generate on-demand reports as CSV files from the REST API:

Set env variables for Sigv4 authentication

  export AWS_REGION="<REPLACE-REGION>"
  export API_ID="<REPLACE-API-ID>"
  eval "$(aws configure export-credentials --profile "<YOUR-PROFILE>" --format env)"

Export applications

  curl -sS --fail-with-body \
    --aws-sigv4 "aws:amz:${AWS_REGION}:execute-api" \
    --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
    --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
    "https://${API_ID}.execute-api.${AWS_REGION}.amazonaws.com/prod/export/applications" \
    -o applications.json

Export assignments with user and group names

  curl -sS --fail-with-body \
    --aws-sigv4 "aws:amz:${AWS_REGION}:execute-api" \
    --user "${AWS_ACCESS_KEY_ID}:${AWS_SECRET_ACCESS_KEY}" \
    --header "x-amz-security-token: ${AWS_SESSION_TOKEN}" \
    "https://${API_ID}.execute-api.${AWS_REGION}.amazonaws.com/prod/export/assignments" \
    -o assignments.json

The API returns a JSON response with a presigned Amazon S3 URL that’s valid for 15 minutes:

{
    "message": "CSV export generated successfully",
    "download_url": "https://<bucket>.s3.amazonaws.com/exports/applications/2026/06/22/applications_export_20260722_184538.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&...",
    "filename": "applications_export_20260722_184538.csv",
    "s3_key": "exports/applications/2026/07/22/applications_export_20260722_184538.csv",
    "file_size_bytes": 6514,
    "export_type": "applications",
    "generated_at": "2026-07-22T18:45:38Z",
    "expires_at": "2026-07-22T19:00:38Z",
    "request_id": "a1b2c3d4-...."
}

{
    "message": "CSV export generated successfully",
    "download_url": "https://<bucket>.s3.amazonaws.com/exports/applications/2026/07/22/applications_export_20260722_184538.csv?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=900&...",
    "filename": "applications_export_20260722_184538.csv",
    "s3_key": "exports/applications/2026/07/22/applications_export_20260722_184538.csv",
    "file_size_bytes": 6514,
    "export_type": "applications",
    "generated_at": "2026-07-22T18:45:38Z",
    "expires_at": "2026-07-22T19:00:38Z",
    "request_id": "a1b2c3d4-...."
}

The generated CSV files include enriched data with friendly names:

Instance ARN	Account ID	Application name	Principal type	Principal name	Status
arn:aws:sso:::instance/…	123456789012	SageMaker_PROD	GROUP	Engineering-Team-Dev	ACTIVE
arn:aws:sso:::instance/…	123456789012	OpenSearch_PROD	USER	[email protected]	ACTIVE

You can use the generated CSV files to help identify anomalies or non-compliant assignments, such as:

Each PROD application should only have GROUP assignments. OpenSearch_PROD has a USER principal type and so is non-compliant.
Each PROD application should only allow PROD groups assigned. SageMaker_PROD has a DEV group name (Engineering-Team-Dev) assigned and so is non-compliant.

Based on the testing and analysis of the output from the IAM Identity Center governance reporting sample solution, it’s important to start thinking about what restrictions to put in place for application assignments. It’s also important to conduct this exercise before taking action within the Identity Center remediation sample solution in the next section.

IAM Identity Center remediation

The following diagram shows the IAM Identity Center remediation architecture.

Figure 3: IAM Identity Center remediation architecture diagram

The IAM Identity Center remediation sample solution deploys the following resources:

Amazon EventBridge – Matches IAM Identity Center assignment and profile events from CloudTrail (sso.amazonaws.com) and invokes the monitor function across the following IAM actions:
1. CreateApplicationAssignment
2. DeleteApplicationAssignment
3. PutApplicationAssignmentConfiguration
4. AssociateProfile
5. DisassociateProfile
6. CreateProfile
7. UpdateProfile
8. DeleteProfile
Lambda – Resolves the application and group names, validates the assignment against your naming convention, and notifies or remediates based on the configured mode
Amazon Simple Notification Service (Amazon SNS) – Publishes alerts for non-compliant assignments to subscribers (for example, email)
Amazon Simple Queue Service (Amazon SQS) – Captures events the Lambda function fails to process for later inspection
AWS KMS – Customer-managed key to encrypt the Lambda environment variables, CloudWatch logs, SNS topic, and dead-letter queue
Amazon CloudWatch – Log group stores the function’s structured, encrypted logs as an audit trail

Flexible naming policies support regex-based pattern matching for specific organizational requirements. The automation actions are logged to CloudWatch with structured JSON for additional analysis and reporting.

The following procedure deploys the remediation infrastructure using AWS Cloud Development Kit (AWS CDK). Make sure you have the following prerequisites in place, then continue with the steps to set up the solution.

Prerequisites

You need the following to run the remediation solution:

An AWS organization with an IAM Identity Center organization instance with delegated administrator access configured
IAM Identity Center configured with at least one instance and an IdP
Access to create groups within the integrated IdP
AWS Command Line Interface (AWS CLI) configured with appropriate credentials
Python 3.12 & Node.js 18 or later installed for CDK deployment

Provide your IAM instance ARN and the account ID where IAM Identity Center is administered:

git clone https://github.com/aws-samples/sample-iam-idc-application-discovery-reporting # only needed if you did not clone in the previous reporting section
cd identity-center-remediation
cdk deploy --context enableAutoDeletion=false --parameters IdentityCenterInstanceArn=arn:aws:sso:::instance/ssoins-<INSERT-ORG-INSTANCE-ID> --parameters ManagementAccountId=<INSERT-MANAGEMENT-ACCOUNT>

Note: If you don’t pass a parameter for GroupNameRegex, the default action of the sample solution is to verify the group name appears as a whole word in the application name: Case-insensitive, splitting on -, _, and spaces, so ReadOnly matches sagemaker_readonly but read does not. If different validation is needed, the sample can be deployed with the regex value for GroupNameRegex.

After the solution is deployed, we will walk through testing both a compliant and non-compliant application assignment.

Gather Identity Center and application information

For this blog, we have already created two groups within the IdP that is integrated into an IAM Identity Center instance. We also already created two applications within Identity Center instance to use. Next, we’ll need to gather information specific to the environment to run through each example.

Obtain the IAM Identity Center instance ARN and set the value.

INSTANCE_ARN=$(aws sso-admin list-instances --region <REPLACE-REGION> --query "Instances[0].InstanceArn" --output text)

echo "$INSTANCE_ARN"

Obtain the IAM Identity Center identity store ID

IDENTITY_STORE_ID=$(aws sso-admin list-instances --region <REPLACE-REGION> --query "Instances[0].IdentityStoreId" --output text)

echo "$IDENTITY_STORE_ID"

Get existing groups in IAM Identity Center

aws identitystore list-groups --identity-store-id $IDENTITY_STORE_ID --query "Groups[].{Name:DisplayName,Id:GroupId}" --output table

Get existing enabled applications in IAM Identity Center

aws sso-admin list-applications --instance-arn $INSTANCE_ARN --query "Applications[?Status=='ENABLED'].{Name:Name,ARN:ApplicationArn}" --output table

After you have the output for IAM Identity Center groups and applications, select two groups and one application that you want to test with. You will need to set additional variables for each group GUID and application ARN. In this example, I select the following two groups (ReadOnly and Developer) for testing and set the environment variables using export:

Group #1 Name: ReadOnly

export GRP_READONLY=abc12345-1234-1234-1234-abcdef123456
export GRP_DEVELOPER=abc12345-1234-1234-1234-abcdef123457
export APP_READONLY="arn:aws:sso::<INSERT-ACCOUNT-ID>:application/<INSERT-INSTANCE-ARN>/<INSERT-APPLICATION-ARN>"

Group #2 Name: Developer
Application Name: sagemaker_readonly

As part of this validation, the sample solution verifies the group name appears as a whole word in the application name: Case-insensitive, splitting on the -, _, characters and spaces, so ReadOnly matches sagemaker_readonly but read does not. For different use-cases, The GroupNameRegex parameter can be used during deployment.

Test compliant and non-compliant assignments

Run the following command to add the ReadOnly group assignment to the sagemaker_readonly application:

aws sso-admin create-application-assignment --application-arn $APP_READONLY --principal-id $GRP_READONLY --principal-type GROUP

The group assignment request meets the validation criteria because the application name sagemaker_readonly contains the group name ReadOnly. The output logs for this validation exist within the associated lambda function CloudWatch log group /aws/lambda/identity-center-app-monitor”.

In this example, the logs will show:

✓ COMPLIANT - Group name found in application name

applicationName="sagemaker_readonly” groupName="ReadOnly”

Remediation action determined: NONE

Run the following command to try to add the Developer group assignment to the sagemaker_readonly application:

aws sso-admin create-application-assignment --application-arn $APP_READONLY --principal-id $GRP_DEVELOPER --principal-type GROUP

The group assignment request doesn’t meet the validation criteria because the application name sagemaker_readonly doesn’t contain the group name Developer. The output logs for this validation exists within the associated lambda function CloudWatch log group /aws/lambda/identity-center-app-monitor. Note that the remediation action listed shows NOTIFICATION_ONLY, meaning it only sent a notification to the configured SNS topic and did not take action. If you want the group assignment to be deleted, the value should be set to enableAutoDeletion=true.

In this example, the logs will show:

✗ NON-COMPLIANT - Group name not found in application name

applicationName="sagemaker_readonly” groupName="Developer”

Remediation action determined: NOTIFICATION_ONLY

SNS notification sent successfully

The SNS message will look like:

{
	"eventType": "NON_COMPLIANT_ASSIGNMENT",
	"applicationName": "sagemaker_readonly",
	"groupName": "Developer",
    "action": "NOTIFICATION_ONLY",
    "status": "SUCCESS",
    "applicationArn": "arn:aws:sso::1234:application/ssoins-1234/apl-1234",
    "groupId": "abc12345-1234-1234-1234-abcdef123457",
    "initiatedBy": { 
    	"type": "AssumedRole", 
    	"arn": "arn:aws:sts::1234:assumed-role/.../you" 
	}
}

Scheduled reporting gives baseline visibility into IAM Identity Center managed applications. Event-driven monitoring can provide near real-time notification or enforcement. Together, these sample solutions can help align and scale Identity Center with your governance and security standards through both historical analysis and immediate response.

Clean up

For each deployed CDK stack, run the following commands in the AWS account where it was deployed.

To delete the remediation stack, run the following commands:

cd sample-iam-idc-application-discovery-reporting/identity-center-remediation
cdk destroy

To delete the reporting stack, run the following commands:

cd sample-iam-idc-application-discovery-reporting/identity-center-reporting
cdk destroy

IAM governance automation at scale

Achieving effective IAM Identity Center governance at scale requires moving beyond manual processes to automated, continuous monitoring and reporting. The following high-level steps can provide an Identity center governance framework:

Deploy the sample automation with an IAM principal that has access in your delegated administrator account.
Establish a baseline by running your first discovery and reviewing the generated reports.
Configure naming policies to match your organization’s security conventions.
Deploy the event-driven monitoring capabilities to enable real-time policy enforcement and automated response based on the security policies.
Start in notification mode to establish a baseline before enabling auto-remediation.
Integrate with your governance tools by connecting the API endpoints to your compliance dashboards or ITSM tools.
Move to auto-remediation once you have validated policies are working as expected.

Conclusion

Managing AWS IAM Identity Center at scale doesn’t have to be a manual, time-consuming process. By implementing automated discovery and reporting combined with real-time event-driven monitoring, you can maintain continuous visibility into your organization’s identity and access landscape, respond immediately to policy violations, and enforce governance policies consistently across your organization. Automation reduces operational work, strengthens security, speeds up incident response, and maintains compliance. Start by deploying these solutions to gain visibility and enable real-time enforcement.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on AWS re:Post or contact AWS Support.

Build a real-time event pipeline with Spark Real-Time Mode on AWS Glue 6.0

2026-08-31 Shoukat Ghouse

Post Syndicated from Shoukat Ghouse original https://aws.amazon.com/blogs/big-data/build-a-real-time-event-pipeline-with-spark-real-time-mode-on-aws-glue-6-0/

Real-time event pipelines rarely get to work with a uniform schema. Whether it’s IoT metrics, ecommerce clickstreams, or financial pricing vectors, each event type brings its own schema. An equity trade and a rates trade, for instance, carry almost entirely different fields. Ingesting these multi-schema streams has traditionally forced suboptimal architectural choices. You build separate tables for each event type or maintain a wide STRUCT where every possible field across all event types must be declared upfront (fast reads, but sparse and rigid). The other option is to flatten everything into an unwieldy schema with hundreds of columns. To sidestep that maintenance burden, many teams dump events into a plain JSON string column that introduces significant performance penalty. Querying a single nested field requires your engine to deserialize the entire JSON blob for every row. At scale, you burn compute and budget scanning terabytes of raw text to extract a few bytes of data.

Adding to the challenge, these pipelines typically demand mixed processing speeds. You need a real-time path (not near-real-time) to flag anomalies or high-risk events with sub-second latency, while simultaneously pushing those same events into analytical storage for deep historical analysis in batch.

With AWS Glue 6.0, you can tackle all of these challenges (schema heterogeneity, JSON scanning overhead, and mixed-latency requirements) from a single pipeline. Built on Apache Spark 4.1 with Apache Iceberg v3 support, AWS Glue 6.0 brings Variant columns, Variant shredding, Spark Real-Time Mode (RTM), and Arrow-native user-defined functions (UDFs) to a fully managed, serverless environment.

In this post, we walk you through how to build this multi-layer architecture using a financial services use case: a market risk pipeline processing trade pricing vectors. While the example is finance, the patterns apply wherever you deal with heterogeneous schemas, expensive JSON parsing, and mixed real-time/batch requirements such as IoT device fleets, multi-tenant SaaS platforms, logistics tracking, and beyond. We will show you how to flag high-risk trades with sub-second latency, stream everything into an Iceberg v3 data lake as Variants, and run batch Value at Risk (VaR) computations efficiently using Arrow-native UDFs.

Solution overview

A bank’s Market Risk team receives a continuous stream of trade pricing vectors from front-office systems. Each trade event carries:

A trade ID and book/desk IDs.
A pricing vector as semi-structured data. The schema varies by asset class (for example, equities carry risk sensitivities known as Greeks such as delta/gamma, foreign exchange (FX) carries volatility surfaces, rates carry curve sensitivities).
A region ID for jurisdictional reporting (uses a column DEFAULT value, so rows that omit it get auto-populated).

The team needs three things from this stream, each at a different speed. We build the pipeline in three layers, each addressing a distinct requirement with a purpose-built AWS Glue 6.0 capability.

Layer 1: Real-time trade position breach detection (sub-second latency)

Positions must be updated in sub-second time, not seconds of traditional micro-batch streaming. For a team monitoring position limits, those seconds mean trades can breach limits before the system reacts. Spark Real-Time Mode (RTM) eliminates the micro-batch boundary entirely, letting records flow continuously through the pipeline so that high-risk trades trigger alerts within sub-second latency of arrival.

Layer 2: Near-real-time analytical lakehouse (seconds latency)

Every trade must land in a queryable data lake within seconds, with heterogeneous pricing vectors stored without declaring a fixed schema upfront. The Iceberg v3 Variant type handles this natively. The raw semi-structured payload goes into a single column regardless of asset class schema. At write time, Variant shredding automatically extracts fields observed in the data into typed Parquet columns, so downstream analytical queries read only the columns they need without deserializing the full blob. Trade amendments and cancellations are handled at a lower cost with deletion vectors (merge-on-read), and column DEFAULT values reduce boilerplate in ingestion code.

Layer 3: Batch risk computation (minutes to hours latency)

Risk metrics like Value at Risk (VaR) must be computed in Python across millions of trades. Traditional row-by-row pickle serialization between the Java Virtual Machine (JVM) and Python is the bottleneck. Arrow-native UDFs process data as vectorized columnar batches, eliminating serialization overhead and accelerating Python-based risk calculations.

The solution uses three separate AWS Glue 6.0 jobs, each independently scalable:

Real-time path (Scala, gluestreaming): Reads trades from Amazon Managed Streaming for Apache Kafka (Amazon MSK), enriches them with risk scores and breach flags, and writes alerts to a downstream Kafka topic. It runs with a fixed set of workers that are always on. Downstream fraud detection and position limit systems consume the alerts topic for real-time blocking decisions.
Near-real-time path (PySpark, gluestreaming): Reads from the same MSK topic and lands the full trade history into an Iceberg v3 table. It uses Glue auto scaling and can scale down between batches, keeping costs lower.
Batch analytics (PySpark, glueetl): Reads from the Iceberg v3 table, extracts fields using variant_get, computes VaR across the portfolio, and writes aggregated risk reports to a downstream summary table.

The following diagram illustrates the solution architecture.

Architecture diagram of a real-time market risk pipeline on AWS Glue 6.0. All compute runs inside a VPC within an AWS Account. A Sample Trades Producer (AWS Glue job, simulating front office trading systems) publishes to an Amazon MSK topic named trade-risk-vectors. From MSK, three processing paths branch out. The Real-Time Path uses an AWS Glue 6.0 Spark Real-Time Mode job in Scala for continuous processing (JSON extraction, risk scoring, breach detection), writing alerts to an Amazon MSK trade-alerts topic that feeds CloudWatch Alarms, SNS notifications, and position limit systems. The Near-Real-Time Path uses an AWS Glue 6.0 micro-batch PySpark job that applies PARSE_JSON to Variant, TIMESTAMP_NTZ with nanosecond precision, and shredding, writing to an Amazon S3 Apache Iceberg v3 table named trade_risk_vectors. The Batch Consumption path reads that table with an AWS Glue 6.0 PySpark job using variant_get extraction and an Arrow UDF for Value at Risk computation and jurisdiction classification, writing to an Amazon S3 Iceberg v3 table named daily_risk_summary that feeds downstream analytics. Amazon S3, AWS Glue Data Catalog, and CloudWatch are regional services shown outside the VPC but inside the AWS Account, accessed privately through VPC endpoints.

Figure 1: Real-time market risk pipeline on AWS Glue 6.0

Prerequisites

To follow along with this post, you need the following:

An AWS account in a Region where AWS Glue 6.0 is available.
An AWS Identity and Access Management (IAM) role with permissions to deploy AWS CloudFormation stacks and create resources including AWS Glue, Amazon MSK, AWS Lambda, Amazon Simple Storage Service (Amazon S3), and the AWS Glue Data Catalog.

Deploy the CloudFormation stack

We provide an AWS CloudFormation template that provisions all the resources needed for this walkthrough.

The stack provisions the following resources:

An Amazon MSK cluster with two topics: trade-risk-vectors (input) and trade-alerts (real-time alerts output).
An Amazon S3 bucket for Iceberg table storage and streaming checkpoints.
An AWS Glue database (risk_analytics_<account-id>_glue6b1).
An IAM role (GlueRole-<account-id>-glue6b1) with permissions for Glue, MSK, S3, and CloudWatch.
Virtual private cloud (VPC) networking: A Glue network connection (connection-<account-id>-glue6b1), S3 gateway endpoint, and Glue interface endpoint.
AWS Glue job rtm-alerts-<account-id>-glue6b1 (Scala): This job reads trades from MSK, scores risk in real time using Spark RTM, writes alerts to the trade-alerts topic.
AWS Glue job nrt-ingestion-<account-id>-glue6b1 (PySpark): This job reads trades from MSK, writes to Iceberg v3 table with Variant + shredding enabled.
AWS Glue job batch-var-<account-id>-glue6b1 (PySpark): This job reads from Iceberg v3 table, computes VaR with Arrow UDF, demonstrates deletion vectors.
AWS Glue job producer-<account-id>-glue6b1-helper (PySpark): This job generates sample trade events (equities, FX, rates) to the trade-risk-vectors topic.

Deploy the CloudFormation stack:

Download the CloudFormation template from the GitHub repository.
Sign in to the AWS CloudFormation console
Choose Create stack > With new resources > Upload a template file, and upload the downloaded template.
Enter the following parameters:
- VpcId: Your VPC ID.
- SubnetIds: At least two subnets in different Availability Zones.
- SecurityGroupId: A dedicated security group that allows all inbound TCP traffic from itself (self-referencing rule).
- RouteTableId: The main route table for your VPC.
Acknowledge the IAM capabilities and choose Create stack.

Stack creation takes approximately 20 minutes.

After the stack completes, open the AWS Glue console and start the jobs in this order:

Start rtm-alerts-<account-id>-glue6b1 and nrt-ingestion-<account-id>-glue6b1.
Once both show RUNNING, start producer-<account-id>-glue6b1-helper.
After the producer finishes (~3.5 minutes), run batch-var-<account-id>-glue6b1 for risk aggregation.

The consumers must be running before the producer starts so that trades are scored in real time and landed in the Iceberg table as they arrive. The batch job runs last because it reads from the Iceberg table that the near-real-time path populates.

Understand the Iceberg v3 table design

The CloudFormation stack provisions Glue jobs that create two Iceberg v3 tables, trade_risk_vectors (primary trade store) and daily_risk_summary (batch VaR output), using new data types and features:

VARIANT: Stores semi-structured pricing vectors without requiring a fixed schema.
DEFAULT values: Automatically applies provided defaults when fields aren’t provided.
Deletion vectors (merge-on-read): Enables fast row-level updates and deletes.

Open the AWS Glue console under Data Catalog > Tables > trade_risk_vectors.

AWS Glue Data Catalog console showing the trade_risk_vectors table with its Variant and default-valued columns

Figure 2: The trade_risk_vectors table in the AWS Glue Data Catalog

The following is the Create Table command:

CREATE TABLE {TABLE} (
    trade_id STRING, book_id STRING, desk STRING,
    asset_class STRING DEFAULT 'UNKNOWN',
    execution_time STRING,
    pricing_vector VARIANT,
    var_contribution DOUBLE DEFAULT 0.0,
    risk_weight DOUBLE DEFAULT 1.0,
    trade_date DATE, region STRING DEFAULT 'EMEA'
) USING iceberg
TBLPROPERTIES ('format-version'='3', 'write.delete.mode'='merge-on-read',
    'write.update.mode'='merge-on-read',
    'write.parquet.shred-variants'='true')
PARTITIONED BY (trade_date, asset_class)

Note the use of DEFAULT values for asset_class, var_contribution, risk_weight, and region. This is an Iceberg v3 feature that applies defaults automatically when values aren’t provided during writes, reducing boilerplate in ingestion code. The pricing_vector column is defined as a Variant type, and write.parquet.shred-variants='true' automatically extracts Variant fields into separate typed Parquet columns at write time for faster downstream queries.

Sample trade event generator

The CloudFormation stack includes a Glue job (producer-<accountid>-glue6b1-helper) that produces realistic trade events to the trade-risk-vectors MSK topic. Each event carries a pricing_vector with a completely different schema per asset class. This is exactly the problem Variant solves.

Equity trade (greeks, scenarios with sector/region breakdowns):

JSON pricing vector for an equity trade showing greeks and per-sector and per-region scenario breakdowns

Figure 3: Sample equity trade pricing vector

Rates trade (curve sensitivities per tenor, calibration params):

JSON pricing vector for a rates trade showing curve sensitivities per tenor and calibration parameters

Figure 4: Sample rates trade pricing vector

Completely different structures: greeks vs curve sensitivities, BlackScholes vs HullWhite. Both land in the same pricing_vector VARIANT column with no schema changes required.

Ingest trades with Spark Real-Time Mode

Traditional Spark Structured Streaming uses micro-batches: collect records, schedule a job, process, commit, wait. Even with small batches, the fixed overhead of planning and scheduling adds noticeable latency per batch. For a risk team monitoring position limits, the delay can let a trade breach a limit before the system reacts.

The following Scala job reads trade events from Amazon MSK, applies lightweight risk rules based on data directly available in the event, and writes alerts to a Kafka topic, all with sub-second latency. The real-time path intentionally avoids external lookups (market data, volatility surfaces) to stay fast. The full VaR computation happens later in the batch layer where latency is less critical.

You can view the complete job code in the AWS Glue console under the rtm-alerts-<accountid>-glue6b1 job. Additionally, all the scripts are available in the GitHub repository.

Scala real-time job code that reads from Kafka, scores risk, and writes alerts to a Kafka topic

Figure 5: Scala real-time job that scores trades and writes alerts

The Trigger.RealTime("1 minute") is what distinguishes this from a traditional micro-batch. Records flow through the pipeline continuously. Records are processed the instant they arrive. The 1-minute parameter controls how often Spark checkpoints its progress for recovery. It does not control how often records are processed. RTM on AWS Glue 6.0 currently supports Kafka-source, stateless, Scala workloads with fixed workers (no auto scaling) and update output mode only. This makes it ideal for stateless transformations that require sub-second latency, such as the filter, enrich, score, and route pattern shown here. The heavier computation (VaR, aggregations) runs in the micro-batch/batch layer where sub-second latency is less critical.

The real-time path acts as a circuit breaker: trades over $50M notional are flagged CRITICAL, over $25M flagged HIGH. Downstream systems consume the trade-alerts topic and can block or escalate before the next trade executes. The detailed VaR computation (which requires market data, volatility surfaces, and the full pricing vector) runs in the batch consumption layer where latency is less sensitive.

After the streaming phase completes, the job reads back from the trade-alerts topic and measures end-to-end latency. It compares two MSK timestamps: when the trade was received by MSK from the producer, and when the alert was received by MSK from RTM.

To verify the alerts and latency, open the Amazon CloudWatch console > Log groups > /aws-glue/jobs/output and select the RTM job’s log stream. You will see the alert summary showing each flagged trade with its end-to-end latency. The following is a sample.

CloudWatch log output listing flagged trades with CRITICAL and HIGH labels and their end-to-end latency

Figure 6: CloudWatch output showing flagged trades and end-to-end latency

Store trades in Iceberg v3 with Variant shredding enabled

The near-real-time path reads from the same MSK topic but writes to an Iceberg v3 table using standard micro-batch streaming. This job runs separately with auto scaling enabled, scaling between batches, keeping costs lower than the always-on real-time path.

You can view the complete job code in the AWS Glue console under the nrt-ingestion-<accountId>-glue6b1 job. The critical aspects are the Variant conversion and the Iceberg write:

PySpark code applying PARSE_JSON to build a Variant column and writing to the Iceberg v3 table

Figure 7: Near-real-time PySpark job writing trades to Iceberg v3 as a Variant

The PARSE_JSON() function converts the raw pricing vector into a native Variant, regardless of the asset class schema. Whether the incoming trade is an equity with greeks, an FX option with a volatility surface, or a rates swap with curve sensitivities, it all goes into the same column. Since the table has write.parquet.shred-variants enabled, fields observed in the initial sample are automatically extracted into typed Parquet columns for fast downstream queries.

How shredding works

During the write process, Spark automatically extracts the Variant fields it observes into separate typed Parquet columns at write time, a feature called shredding. At the start of each write, the engine buffers a sample of rows (controlled by write.parquet.variant-inference-buffer-size), infers which fields exist and their types, then uses that schema to shred all subsequent rows in the file. Every field observed in that sample gets its own typed column, including nested objects. Rows that lack a particular field simply store NULL in that shredded column. For our risk table, fields like $.greeks.delta, $.dv01, and $.model all live in their own typed Parquet columns, even if only one asset class carries a specific field. The result: faster read performance because queries access only the typed columns they need, skipping the rest of the document entirely. Shredding is transparent to queries. variant_get() calls work the same way whether the field is shredded or not. The query engine automatically routes to the shredded column when available, falling back to the binary Variant blob for fields that aren’t part of the inferred schema.

Note: Shredding adds write latency because the engine must infer the schema and write additional typed columns. In this pipeline, we enable shredding on the near-real-time path and absorb that cost, since the downstream read benefits (batch VaR, ad-hoc queries, audit) far outweigh the write penalty. For latency-sensitive pipelines where every millisecond on the write path matters, you can disable shredding on the streaming table and instead write shredded data in a separate batch job that reads from the unshredded table and inserts into a shredded copy. This approach trades architectural simplicity for lower ingestion latency.

Build the batch consumption layer

The third AWS Glue 6.0 job reads from the Iceberg v3 table, extracts risk metrics from the Variant column, computes VaR using an Arrow-native UDF, and writes aggregated results to a summary table.

You can view the complete job code in the AWS Glue console under the batch-var-<accountid>-glue6b1 job. The key aspects are the variant_get extraction from deeply nested structures and the Arrow-native UDF:

PySpark code using variant_get to extract deeply nested fields from the Variant column

Figure 8: Extracting nested Variant fields with variant_get

Notice how variant_get reaches into arbitrarily nested structures: $.greeks.delta (2 levels), $.scenarios[0].breakdown.by_sector.financials (5 levels), $.model_params.calibration.fit_error (4 levels). All with the same function call. No pre-flattening, no schema-per-asset-class tables, no ETL to restructure the data before querying.

Once the risk metrics are extracted, we need to run a Monte Carlo-style Historical VaR that simulates 1,000 daily profit and loss (P&L) scenarios per trade and returns the 99th percentile loss. This is where the @arrow_udf decorator comes in.

Python Arrow UDF code running a Monte Carlo Historical VaR simulation for each trade

Figure 9: Arrow-native UDF computing Historical VaR

The @arrow_udf decorator is new in Spark 4.1. Your function receives and returns pyarrow.Array directly, operating on the entire batch of rows at once. There is no pickle serialization, no row-by-row invocation, and no Pandas conversion. Data flows as native Arrow columnar arrays between the JVM and Python. For compute-heavy operations like VaR across hundreds of thousands of rows, this can be significantly faster than traditional scalar UDFs. Additionally, you can use the built-in UDF profilers to identify performance and memory bottlenecks in compute-heavy UDFs such as VaR calculations.

Handle late trade corrections with deletion vectors

In financial markets, trade amendments and cancellations are common. The batch VaR job demonstrates this after completing the risk computation. It amends one trade and cancels another.

PySpark code amending one trade and cancelling another in the Iceberg table

Figure 10: Amending and cancelling trades with merge-on-read

Prior to Iceberg v3, row-level deletes required either rewriting entire data files (copy-on-write) or maintaining separate positional delete files that store (file_path, row_position) pairs as Parquet rows (merge-on-read). Both approaches are expensive at scale. Copy-on-write rewrites gigabytes for a single amendment, and positional deletes degrade read performance as delete files accumulate (each read must parse and hash-join all delete records against the data file).

Because we configured the table with write.delete.mode='merge-on-read', UPDATEs and DELETEs write deletion vectors instead of positional delete files used in Iceberg v2. A deletion vector is a Roaring Bitmap stored in a Puffin file (.puffin), one per affected data file, marking which row positions are deleted. At read time, the engine loads a single bitmap and skips flagged positions with a bit check. No file joins, no linear scan through multiple delete files. The bitmap is compact regardless of how many rows are deleted, and read performance remains predictable as amendments accumulate.

The batch job also verifies the deletion vectors were created. You can see the results in the job’s output logs.

Job output confirming deletion vector Puffin files were created for the affected data files

Figure 11: Output verifying deletion vectors were created

Clean up

To avoid incurring further charges, delete the CloudFormation stack. This removes all resources provisioned as part of this post, including the S3 bucket, Glue jobs, Iceberg tables, MSK cluster, and IAM roles.

Conclusion

In this post, we built a multi-layer market risk pipeline using AWS Glue 6.0 (real-time alerting, near-real-time ingestion, and batch analytics):

Spark Real-Time Mode (RTM) on the real-time path delivers sub-second trade scoring and breach alerting, eliminating the micro-batch boundary so position limits are enforced before the next trade executes.
Iceberg v3 Variant on the near-real-time path stores heterogeneous pricing vectors without schema flattening. One table handles equities, FX, and rates with different schemas per row.
Variant shredding delivers faster reads by automatically extracting fields into separate typed Parquet columns at write time with no manual tuning required.
Arrow-native UDFs eliminate pickle serialization overhead for Python-based risk calculations, processing data as vectorized columnar batches on the batch layer.
Deletion vectors handle trade amendments and cancellations without costly data file rewrites, using compact Roaring Bitmaps instead of accumulating positional delete files.
Default values reduce boilerplate in ingestion code.

To get started with AWS Glue 6.0, see the AWS Glue documentation. For more information about Apache Iceberg v3, see the Iceberg specification.

About the authors

Optimize EKS operations with agents: Reduce MTTR with AWS DevOps Agent and a Kubernetes Operator

2026-08-31 HoSeong Lee

Post Syndicated from HoSeong Lee original https://aws.amazon.com/blogs/devops/optimize-eks-operations-with-agents-reduce-mttr-with-aws-devops-agent-and-a-kubernetes-operator/

Introduction

Running workloads on Amazon Elastic Kubernetes Service (Amazon EKS) can involve managing failures like OOMKilled or IP exhaustion. Engineers must repeatedly collect pod logs, trace events, and check node logs—a process that slows at night/weekends, with critical data lost when pods are deleted or nodes become unhealthy. This collection phase is pure overhead on mean time to resolution (MTTR): the incident stays open while an engineer gathers data that a machine could have captured the moment the failure occurred. Automating it shortens MTTR and lets the on-call engineer start at the analysis step instead of the data-gathering step.

Existing AI tools have limitations: K8sGPT only analyzes current resource state, and Amazon Bedrock Agents requires manual tool integration and pipeline setup. Neither provides end-to-end automated incident investigation.

AWS DevOps Agent addresses these gaps—a frontier agent that connects code repositories, observability tools, CI/CD pipelines, and skills to autonomously analyze root causes. This post shows how to build an automated incident response pipeline using the DevOps Agent Operator, a Kubernetes Operator that detects EKS failures and triggers DevOps Agent investigations automatically.

Solution overview

AWS DevOps Agent provides powerful incident analysis. However, it does not detect pod failures inside an EKS cluster on its own. To start an investigation, an external source must trigger DevOps Agent through a webhook. When this trigger occurs, two conditions must be met:

Immediate failure detection: You must detect the failure before the pod is rescheduled or deleted.
Sufficient context: You must send the data that the analysis needs, such as the manifest, logs, events, and node information.

The DevOps Agent Operator is a Kubernetes Operator that meets both conditions automatically.

Why use an Operator?

DevOps Agent runs only when something calls it through a webhook or a manual trigger. In 24/7 operations, doing this manually is not practical. Kubernetes keeps events for only about an hour, restarted containers overwrite their logs, and deleted pods lose them entirely. If you do not collect data right after a failure, the key evidence is gone for good.

DevOps Agent can already run describe and logs with kubectl, and tools like Datadog can detect failures and trigger it.

A separate Operator still adds value for three reasons:

Proactive preservation of volatile data: The Operator detects state changes in milliseconds via watch and preserves data to S3/CloudWatch instantly—before external tool delays (metric collection, alert evaluation, webhook delivery) let evidence disappear.
Selective collection of node-level data: kubectl exposes only container-level and event data, but root causes often live deeper in the node—for example, OOMKilled traces to node dmesg, and IP exhaustion details are in IPAMD introspection. Because the Operator knows the real-time pod-to-node mapping, it collects only what each failure type needs from the exact node.
Encoding operational knowledge in code: The Operator pattern captures human expertise in code, applying different strategies per failure type—dmesg/memory for OOMKilled, previous logs/restart history for CrashLoopBackOff, IPAMD/ENI mappings for IP exhaustion—directly improving analysis accuracy.

In short, the Operator captures evidence at the failure site before it disappears and collects data beyond the reach of kubectl, giving DevOps Agent the best possible material to analyze.

Note: If data collection or an upload to Amazon S3 or CloudWatch Logs fails, the reconcile returns an error and the pod is requeued with exponential backoff rather than dropped, and throttled AWS API requests are retried automatically. The Operator also runs a single reconcile worker and marks each pod with a processed annotation, so a mass failure—for example, 100 replicas crashing at once—is handled one pod at a time and each pod is reported only once. For noisy clusters, WEBHOOK_MIN_SEVERITY and WEBHOOK_SKIP_CATEGORIES let you narrow which failures trigger an investigation.

Architecture

Figure 1. End-to-end flow from failure detection to investigation.

The preceding diagram shows the full flow. The DevOps Agent Operator detects a failure inside the EKS cluster and sends the context to AWS DevOps Agent.

Getting started

Prerequisites

Region availability: AWS DevOps Agent is available in six AWS Regions—US East (N. Virginia), US West (Oregon), Europe (Frankfurt), Europe (Ireland), Asia Pacific (Sydney), and Asia Pacific (Tokyo). Create your Agent Space in one of these Regions.
Node type: Node-level log collection uses AWS Systems Manager Run Command against the EC2 instance that ran the failed pod, so it requires Amazon EKS managed node groups or self-managed EC2 nodes. On AWS Fargate, the Operator still collects Kubernetes-level data—the pod manifest, events, and container logs—but node-level data such as dmesg output and IPAMD introspection is not available.
Systems Manager registration: Attach the AmazonSSMManagedInstanceCore policy to your node group’s IAM role so the nodes appear as managed nodes. Without it, node-level collection is skipped and only Kubernetes-level data is collected.

Setting up this solution involves two steps.

The first step is to configure the Agent Space for DevOps Agent. You connect the sources that DevOps Agent needs to analyze an incident, such as code repositories and observability tools. You also set up a generic webhook to receive failure information from the Operator.

The second step is to deploy the DevOps Agent Operator to the EKS cluster. When the Operator detects a pod failure, it collects the context and sends it automatically to the webhook that you set up in the first step.

After you complete these steps, you have an end-to-end pipeline. When a pod failure occurs, DevOps Agent starts an investigation automatically.

Step 1: Configure the Agent Space for DevOps Agent

Configure the webhook

DevOps Agent supports two types of webhooks:

Integration-specific webhooks: Created automatically when you set up an integration with an external solution, such as Slack or Datadog.
Generic webhooks: Created manually to trigger an investigation from sources that an external integration does not cover.

The DevOps Agent Operator uses a generic webhook. It maintains security through HMAC-SHA256 authentication.

For detailed setup instructions, see the following documentation. This post creates a generic webhook as an example.

Configuring webhook access

Configure the pipeline

You connect GitHub or GitLab so that DevOps Agent can track deployment events and correlate code changes with failures.

Register GitHub or GitLab at the AWS account level.
Connect the repositories that you want to monitor to the Agent Space.

With this connection, DevOps Agent can analyze the recent deployment history and code changes when a failure occurs. DevOps Agent currently supports GitHub and GitLab. For GitLab, you can use both the managed instance and a self-managed instance that is reachable from outside.

For detailed setup instructions, see the following documentation. This post uses GitHub as an example.

Connecting to CI/CD pipelines

Configure communication

DevOps Agent joins your team’s existing communication channels to share its investigation activity. When you connect Slack, you can follow the full process in real time, from failure detection to completed analysis.

For detailed setup instructions, see the following documentation. This post uses Slack as an example.

Connecting to ticketing and chat

Step 2: Deploy the DevOps Agent Operator

To install the DevOps Agent Operator, complete the prerequisite steps and the Operator deployment steps in order.

Before you continue, download the source code. You can find the source code at the following link: DevOps Agent Operator source code

1. Prerequisite steps

Before you deploy the Operator to an existing EKS cluster, complete the following prerequisite steps.

1.1. Create an IAM policy for SSM, Amazon S3, and CloudWatch

cat >devops-agent-operator-permission.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "SSMCommandExecution",
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand",
        "ssm:GetCommandInvocation"
      ],
      "Resource": [
        "arn:aws:ec2:<aws-region>:*:instance/*",
        "arn:aws:ssm:<aws-region>:*:*"
      ]
    },
    {
      "Sid": "S3LogStorage",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject"
      ],
      "Resource": "arn:aws:s3:::<s3-bucket-name>/*"
    },
    {
      "Sid": "S3BucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": "arn:aws:s3:::<s3-bucket-name>"
    },
    {
      "Sid": "CloudWatchLogsIncidentStorage",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:<aws-region>:*:log-group:/<cloudwatch-log-group-name>:*"
    }
  ]
}
EOF

Note: To keep this example readable, the policy allows ssm:SendCommand on EC2 instances in the account. In production, restrict it to your cluster’s nodes with an IAM condition key—for example, a StringEquals condition on ssm:resourceTag/eks:cluster-name in a statement that targets only the instance ARN—so that the Operator cannot run commands on unrelated instances. Keep the AWS-RunShellScript document ARN in a separate statement without the condition, because a document carries no instance tags and a single combined statement would deny the call.

Next, create the policy from this file.

aws iam create-policy \
    --policy-name devops-agent-operator-policy \
    --policy-document file://devops-agent-operator-permission.json

1.2. Create a trust policy

cat >devops-agent-operator-trust-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowEksAuthToAssumeRoleForPodIdentity",
            "Effect": "Allow",
            "Principal": {
                "Service": "pods.eks.amazonaws.com"
            },
            "Action": [
                "sts:AssumeRole",
                "sts:TagSession"
            ]
        }
    ]
}
EOF

1.3. Create an IAM role

aws iam create-role \
    --role-name devops-agent-operator-role \
    --assume-role-policy-document file://devops-agent-operator-trust-policy.json

aws iam attach-role-policy --role-name devops-agent-operator-role --policy-arn=arn:aws:iam::<aws-account-id>:policy/devops-agent-operator-policy

1.4. Associate Pod Identity

EKS Pod Identity associates Kubernetes service accounts directly with IAM roles, enabling pods to access AWS services like Amazon CloudWatch under the principle of least privilege. For more information, see Learn how EKS Pod Identity grants pods access to AWS services.

Pod Identity requires the eks-pod-identity-agent add-on, which is not installed on existing clusters by default. If your cluster does not have it yet, add it first:

aws eks create-addon \
    --cluster-name <eks-cluster-name> \
    --addon-name eks-pod-identity-agent

Then create the association:

aws eks create-pod-identity-association
  --cluster-name <eks-cluster-name>
  --namespace devops-agent-operator-system
  --service-account devops-agent-operator
  --role-arn arn:aws:iam::<aws-account-id>:role/devops-agent-operator-role

2. Build the image

Because the Operator is a reference implementation, the sample provides source code only—no prebuilt container image.
Build the image with the Dockerfile at the following location and push it to a registry that you control, which also keeps the image that runs in your cluster inside your own supply chain. Then use that image to deploy the Operator.
Building the image locally requires Go 1.25 or later. The Operator is built against the Kubernetes 1.35 client libraries and uses only the core Pod, Node, and Event APIs.

Dockerfile location

For example, suppose that you create a separate repository from all the files under Devops Agent Operator – Sample

You can then build the image through CI/CD with the following GitHub Action as a reference.

name: Build and Push container images to GitHub Container Registry
jobs:
  ...
  build-and-push:
    name: Build and Push Image
    runs-on: ubuntu-latest
    needs: create-tag
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Setup Go
        uses: actions/setup-go@v5
        with:
          go-version-file: go.mod
      - name: Login to GitHub Container Registry
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.repository_owner }}
          password: ${{ secrets.WRITE_REGISTRY_TOKEN }}
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: Build and Push
        uses: docker/build-push-action@v6
        with:
          context: .
          file: Dockerfile
          push: true
          provenance: false
          no-cache: true
          tags: |
            "ghcr.io/${{ github.repository_owner }}/devops-agent-operator:${{ needs.create-tag.outputs.sha_short }}"
            "ghcr.io/${{ github.repository_owner }}/devops-agent-operator:latest"

3. Deploy the Operator

The following steps are based on the example YAML files for the DevOps Agent Operator. Download the repository, or run the following command to download the files, and then continue.

curl -s https://api.github.com/repos/aws-samples/kr-tech-blog-sample-code/contents/containers/devops-agent-operator/examples?ref=main | jq -r '.[].download_url' | xargs -n1 curl -O

3.1. Set the environment variables

Open the 05-deployment.yaml file, and then change the following variables to values that match your environment.

containers:
- name: manager
    # Use the image that you built in step 2
    image: <operator-image>:latest
    ...
    env:
    # Required settings
    - name: DEVOPS_AGENT_WEBHOOK_URL
        value: "<devops-agent-webhook-url>"
    ...
    - name: EKS_CLUSTER_NAME
        value: "<eks-cluster-name>"
    - name: AWS_REGION
        value: "<aws-region>"
    - name: AWS_ACCOUNT_ID
        value: "<aws-account-id>"
    # Optional settings
    - name: ENABLE_SSM_COLLECTION
        value: "true"
    - name: CLOUDWATCH_LOG_GROUP
        value: "<cloudwatch-log-group-name>"

Also change the 04-configmap.yaml file to values that match your environment.

data:
  # Comma-separated list of namespaces to watch (empty = all namespaces)
  WATCH_NAMESPACES: ""
  # Comma-separated list of namespaces to exclude
  EXCLUDE_NAMESPACES: "kube-system,kube-public,kube-node-lease"
  # Enable AWS SSM node log collection (requires IAM permissions)
  ENABLE_SSM_COLLECTION: "true"
  # AWS region for SSM and S3
  AWS_REGION: "<aws-region>"
  ...

In a shared or multi-tenant cluster, set WATCH_NAMESPACES to the namespaces that your team owns so that the Operator does not collect data from other teams’ workloads. If you leave it empty, the Operator watches every namespace except those listed in EXCLUDE_NAMESPACES.

Note: DevOps Agent references the collected data only while it investigates the incident, so you do not need to retain it long-term. Keeping a short retention period on the CloudWatch log group—and a matching S3 Lifecycle expiration rule on the bucket—keeps the storage cost of this solution minimal.
# Expire the incident logs in CloudWatch Logs after 14 days
aws logs put-retention-policy \
    --log-group-name <cloudwatch-log-group-name> \
    --retention-in-days 14

# Expire the incident objects in Amazon S3 after 14 days
aws s3api put-bucket-lifecycle-configuration \
    --bucket <s3-bucket-name> \
    --lifecycle-configuration '{"Rules":[{"ID":"expire-incident-data","Status":"Enabled","Filter":{"Prefix":"incidents/"},"Expiration":{"Days":14}}]}'

3.2. Create the webhook secret

Edit the 06-webhook-secret.yaml file:

stringData:
  webhook-secret: "<webhook-secret>"

3.3. Deploy the Kubernetes resources

kubectl apply -f .

The example deployment runs a single replica with leader election enabled, so you can raise the replica count for availability without two Operators processing the same failure.

3.4. Verify the deployment

# Check the pod status
kubectl get pods -n devops-agent-operator-system

# Check the logs
kubectl logs -f deployment/devops-agent-operator \
  -n devops-agent-operator-system

When the Operator works correctly, it produces the following logs:

Configuration loaded
Log collector initialized (sinceMinutes: 15)
Webhook client initialized
CloudWatch Logs client initialized
S3 client initialized
Starting workers (worker count: 1)

Use case: Automated analysis of an OOMKilled failure

The following scenario shows how the DevOps Agent Operator and DevOps Agent work together. In this environment, Slack is connected as the notification channel for DevOps Agent, and GitHub is connected as the pipeline.

Scenario

In this scenario, a developer pushed a code change to add a new feature to the web-python service and built a new container image. The developer then updated the running web-python deployment in the EKS cluster with the newly built image.

After the new version rolled out successfully, the developer verified that other services were unaffected. Shortly after, a Slack notification arrived. DevOps Agent reported that the pod that was just deployed had terminated with an OOMKilled status, and that it was investigating the related incident.

kubectl get pods -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[*].ready,STATUS:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount,IMAGE:.spec.containers[*].image'
NAME READY STATUS RESTARTS IMAGE
web-python-56b9874b88-tdljd true Running 0 <your-registry>/web-python:sha-96cd2b0

# Deploy the new version
kubectl get pods -o custom-columns='NAME:.metadata.name,READY:.status.containerStatuses[*].ready,STATUS:.status.phase,RESTARTS:.status.containerStatuses[*].restartCount,IMAGE:.spec.containers[*].image'
NAME READY STATUS RESTARTS IMAGE
web-python-645b4f7867-lvqgr true Running 0 <your-registry>/web-python:sha-15d1398

The following steps describe what happens after the pod with the new image is deployed.

Step-by-step flow

1. Failure detection

The kubelet detects the OOM termination of the web-python container and updates the pod status. The informer in the DevOps Agent Operator receives this change in real time. It detects the change from the previous state (Running) to the current failure state (OOMKilled).

kubectl describe po web-python-645b4f7867-lvqgr
Name: web-python-645b4f7867-lvqgr
Namespace: default
...
Annotations: devops-agent.io/failure-type: OOMKilled
                  devops-agent.io/processed: true
                  devops-agent.io/processed-at: 2026-05-30T07:24:43Z

2. Kubernetes-level data collection

As soon as the Operator detects the failure, it collects Kubernetes-level data including pod manifests, pod logs, previous crash logs, and OOM-related event timelines.

3. Node-level data collection

It then gathers node-level data such as kubelet, containerd, and ipamd logs, disk/memory/network usage, and the kernel OOM killer log from dmesg output.

4. Data storage

Based on your configuration, the Operator stores the collected data in CloudWatch Logs and Amazon S3. DevOps Agent can reference the data in CloudWatch Logs during the investigation when it needs to.

5. DevOps Agent trigger

The Operator sends a webhook request that includes an HMAC-SHA256 signature to DevOps Agent. The payload includes investigation instructions for the AI agent.

The DevOps Agent Operator handles steps 1 through 5. You can also see these steps in the logs of the Operator pod.

# 1. Failure detection
2026-05-30T07:24:05Z INFO Failure detected {"controller": "pod", ... "pod": {"name":"web-python-645b4f7867-lvqgr","namespace":"default"}, "failureType": "OOMKilled", "container": "web-python", "exitCode": 137}

# 2-3. Data collection
2026-05-30T07:24:06Z INFO ssm-collector Collecting node logs via SSM {"node": "ip-192-168-1-10.ec2.internal", "instanceID": "i-0123456789abcdef0"}
...

# 4. Data storage
2026-05-30T07:24:08Z INFO cloudwatch CloudWatch Logs upload completed {"logGroup": "cw-log-group-devops-agent-operator", "logStream": "incidents/2026-05-30T07-24-05Z/default/web-python-645b4f7867-lvqgr", "eventsCount": 13}
...

# 5. DevOps Agent trigger
...
2026-05-30T07:24:08Z INFO webhook Webhook request with S3 reference successful {"incidentId": "2026-05-30T07-24-05Z/default/web-python-645b4f7867-lvqgr", "status": 200}

6-7. DevOps Agent Investigation

As DevOps Agent starts the investigation, it shares the incident and its investigation status in the Slack channel that you configured for communication. Through this notification, the engineer can open the Agent Space and follow the investigation in real time.

Slack message from the AWS DevOps Agent app reading "Investigation started: Pod OOMKilled: default/web-python-645b4f7867-lvqgr", with a link to view the investigation.

Figure 2. DevOps Agent announces the OOMKilled incident in Slack.

Skill-based investigation: DevOps Agent automatically selects the skill that matches the incident type. Following the OOMKilled skill, it systematically performs the steps to check the memory configuration, analyze usage patterns, and review the code change history.

The Investigation timeline tab showing the payload sent by the Operator—cluster, pod name, node, failure type OOMKilled, exit code 137, and the Amazon S3 data location—followed by the skill file that DevOps Agent read.

Figure 3. The investigation timeline opens with the payload the Operator sent.

Correlation analysis: In addition to the troubleshooting data that it receives, DevOps Agent connects the following sources for its analysis:

GitHub: Checks recent code changes for memory-related modifications.
CloudWatch: Checks memory usage trends in Container Insights.

In this scenario, you can see that DevOps Agent starts its analysis from the data that the Operator uploaded to CloudWatch Logs, as the skill specifies.

The timeline showing the OOMKilled symptom and four investigation tasks running in parallel: search-app-logs, search-performance-metrics, search-code-repos, and check-cloudtrail-changes.

Figure 4. DevOps Agent runs four investigation tasks in parallel.

The code repository task listing the files DevOps Agent read from the connected repository, including the Kubernetes deployment manifest, the application source, the Dockerfile, and the requirements file.

Figure 5. DevOps Agent reads the manifest and application code from the connected repository.

The skill also specifies the relationship between the GitHub repository that you connected as a pipeline and the container image. DevOps Agent uses this information to review the code changes that occurred recently.

This information helps DevOps Agent identify the root cause of the incident.

Two findings on the timeline: an unbounded processed_records list leaking about 20 Mi per minute, and a deployment updated to the image that contains the leaking code.

Figure 6. Two findings: the unbounded list and the deployment that introduced it.

8. Analysis results

DevOps Agent organizes the analysis results:

Investigation Timeline: This tab shows the agent’s investigation steps—which skills it referenced and what data it analyzed.
This view helps you optimize the skill to guide investigations more efficiently.
Root causes: This section summarizes the root cause from the overall investigation.

Unbounded `processed_records` list in web-python application causes memory leak at ~20Mi/min
The Python Flask application in image `<your-registry>/web-python:sha-15d1398` contains a background worker thread (`_cache_worker`) that generates 500 records every 2 seconds and appends processed results to an in-memory list called `processed_records`. Unlike the `cache` list which has eviction logic capped at 80MB (`CACHE_SIZE_MB`), the `processed_records` list has NO eviction or size limit — it grows unboundedly. With Python/Flask overhead (~30MB) + the cache growing toward its 80MB cap, the remaining headroom within the 200Mi container memory limit is exhausted in approximately 10 minutes. This was confirmed by two consecutive pod instances (lvqgr and 7rwdj) both being OOMKilled after exactly ~10 minutes of runtime.

With the investigation from DevOps Agent, the engineer can identify the cause of the problem.

In the preceding example, you can see how the agent identifies a critical memory leak in the recently changed service code. It then reasons about the cause of the OOM event together with the commit ID.

The Root cause tab showing the memory leak summary with two supporting observations: the pod being OOMKilled twice within 30 minutes under a 200Mi limit, and the audit log entry for the image change.

Figure 7. The Root cause tab with its supporting observations.

9. Analysis and mitigation plan through chat

The engineer reviews the results and, when needed, can ask DevOps Agent follow-up questions:

“Check whether other services show a similar memory growth pattern.”
“Will fixing it with approach A help solve the problem?”

In the following example, the engineer asks whether increasing the pod memory limit will help solve the problem. The agent responds based on its investigation.

A chat panel where the engineer asks whether raising the pod memory limit to 250Mi would mitigate the issue, and DevOps Agent answers that it would only add about 2.5 minutes before the same OOMKill.

Figure 8. Follow-up chat on whether a higher memory limit would help.

As this shows, DevOps Agent goes beyond simple problem analysis. It uses the context that it accumulated during the investigation to respond to the engineer’s follow-up questions with detailed explanations.

In this scenario, the problem is a logic issue in the source code. For that reason, DevOps Agent could not provide a clear plan at the Kubernetes or AWS infrastructure level. However, based on the root cause, you can receive a mitigation plan related to a rollback.

The Mitigation plan tab proposing a rollback of the web-python deployment to the previous image, with numbered preparation steps and the AWS CLI commands to verify the cluster first.

Figure 9. The Mitigation plan tab proposes a rollback.

Conclusion

In this post, we introduced the DevOps Agent Operator – a Kubernetes Operator that automatically detects EKS workload failures, collects diagnostic data, and triggers AWS DevOps Agent for root cause analysis.

By combining these two tools, engineers gain the following benefits:

Faster response: Automatic data collection and analysis as soon as a failure occurs, even during nights and weekends.
No loss of information: Immediate preservation of all troubleshooting data before a pod is rescheduled or deleted.
Comprehensive analysis: DevOps Agent analyzes code repositories, observability tools, and CI/CD pipelines together to trace root causes that are hard to find with a single tool.
Organizational knowledge: Through skills, the solution reflects your team’s operational knowledge, enabling incident response with consistent quality.
Continuous improvement: Proactive recommendations based on accumulated incident data help prevent future incidents.

Looking ahead, there are several ways to extend this solution:

Support for more resource types: Extend monitoring beyond pods to Job, CronJob, Deployment, and StatefulSet.
MCP server integration: DevOps Agent supports Model Context Protocol (MCP) servers, enabling advanced workflows such as querying additional resources during analysis or performing pattern analysis on past incidents.
Proactive pattern analysis: As incident data accumulates in Amazon S3 and CloudWatch Logs, DevOps Agent can identify recurring patterns – such as “OOMKilled repeats every Monday morning” – and recommend preventive measures.

The DevOps Agent Operator project is open source on GitHub. It is a reference implementation rather than a supported product: use it as a working example of how to encode your own detection conditions and collection strategy for the failures your team actually sees.

To try it yourself, clone the repository, follow the deployment steps in this post, and point the Operator at your own Agent Space webhook. Start with a non-production cluster and a narrow WATCH_NAMESPACES list, then widen the scope once you see the investigations that DevOps Agent produces.

References

Extend your data perimeter to the AWS Management Console with Private Access

2026-08-28 Madhur Kulkarni

Post Syndicated from Madhur Kulkarni original https://aws.amazon.com/blogs/security/extend-your-data-perimeter-to-the-aws-management-console-with-private-access/

Organizations in regulated industries such as financial services, government, defense, and healthcare restrict their sensitive workloads to isolated network environments with no access to the public internet. Until now, customers could restrict AWS Management Console access to authorized AWS accounts and corporate networks, but the console itself required internet connectivity. This was creating tension between operational convenience and network security controls.

We’re happy to announce that AWS Management Console Private Access is now generally available with support for virtual private clouds (VPCs) without internet connectivity. Organizations in regulated industries that restrict workloads to isolated network environments can now route all traffic for supported service consoles—including authentication flows, static assets (JavaScript, CSS, images), console-only APIs, and AWS service API calls—through AWS PrivateLink VPC endpoints, eliminating the need for an internet gateway, NAT gateway, or any route to the public internet. This capability is available in all AWS commercial Regions for a select set of supported service consoles.

In 2023, we launched AWS Management Console Private Access, which you can use to connect to the console by routing console, sign-in, and service API calls through VPC endpoints. However, accessing the console required internet connectivity for static assets and console-only APIs. This meant security teams faced a choice: allow internet connectivity to use the console or deny console access to operators working in network-isolated environments.

With this launch, AWS Management Console Private Access addresses two common scenarios:

Console traffic over internet restricted networks: Traffic for supported service consoles now flows entirely through your VPC endpoints—no proxy allowlists to maintain, no TLS-intercepting proxies to operate, and no CLI-only workflows to accept as a compromise. The same path works seamlessly from Amazon WorkSpaces, Amazon Elastic Compute Cloud (Amazon EC2) instances, and on-premises networks connected through AWS Direct Connect or AWS Site-to-Site VPN. Combined with sign-in resource control policies (RCPs) and sign-in resource policies, you can ensure that console authentication only succeeds from expected networks—even if valid credentials are presented elsewhere, the session is denied. Teams that previously relied on restricted egress rules or manual domain allowlists now get full console access with the same network controls they already trust.
Data-exfiltration prevention: Private Access enables you to restrict which AWS accounts and organizational identities can use the AWS Management Console from within your VPC. This prevents access from personal accounts and from accounts outside your organization. Attach a VPC endpoint policy with an aws:ResourceOrgID condition, and console actions are automatically scoped to resources inside your organization. Sign-in RCPs add a second layer by ensuring authentication only succeeds from networks within your perimeter. Together, these controls prevent supported service consoles from being used to access resources in accounts outside your organization—such as personal accounts—without requiring complex network-layer workarounds.

In this post, you will learn how AWS Management Console Private Access works in environments without internet connectivity, and how to layer access controls using VPC endpoint policies and sign-in resource control policies (RCPs) to strengthen your data perimeter.

Solution overview

AWS Management Console Private Access and sign-in resource control policies are a natural extension of the service control policies (SCPs), resource control policies, and VPC endpoint policies you already use for API traffic; now applied to the console session itself. The same data perimeter controls for identity, resource, and network that protect your programmatic access now protect interactive browser sessions too.

Perimeter	Control objective	Policy construct	Implementation Steps
Identity	Only trusted identities can access my resources	Sign-In RCPs and RBPs	Restrict which principals can sign in to the console. Before authentication, `signin:PrincipalArn` is available for exemptions only. After authentication, RCPs restrict at the organization, account, or principal level (`aws:PrincipalOrgID`, `aws:PrincipalAccount`, `aws:PrincipalArn`); RBPs restrict at the account or principal level.
Identity	Only trusted identities are allowed from my network	Console VPC endpoint policy and Sign-In VPC endpoint policy	Console endpoint: `aws:PrincipalOrgID` or `aws:PrincipalAccount` on signed-in identities. Sign-In endpoint: `aws:ResourceOrgID` or `aws:ResourceAccount` before authentication, principal and resource keys after authentication. Blocks sign-in to accounts outside your organization, such as personal accounts, from your network.
Resource	My identities can access only trusted resources	SCP	Resource perimeter SCP with `aws:ResourceOrgID` follows your principals into every console session; each service API call the console makes on their behalf is denied if the target resource is outside your organization.
Resource	Only trusted resources can be accessed from my network	Console VPC endpoint policy and service VPC endpoint policies	Console endpoint policy with `aws:ResourceOrgID` and `aws:ResourceAccount` scopes what the console can reach through your network.
Network	My identities can access resources only from expected networks	SCP	Network perimeter SCPs that use aws:SourceVpc deny your principals’ service calls from outside expected networks. With Private Access, requests proxied by the console to supported services carry `aws:SourceVpc` set to the VPC hosting your Private Access endpoints. Direct browser requests carry VPC context only when the service has its own VPC endpoint, so configure endpoints for every service you use. AWS recommends conditioning on `aws:SourceVpc` rather than specific `aws:SourceVpce` values.
Network	My resources can only be accessed from expected networks	Sign-In RBPs and RCPs and network perimeter RCPs	Sign-In policies deny console authentication from unexpected networks using `aws:SourceIp`, `aws:SourceVpc`, `aws:SourceVpce`, and `aws:VpcSourceIp` in both pre-authentication and post-authentication statements. Network perimeter RCPs apply the same network conditions to your data resources for any access path.

With this launch, Console Private Access routes browser traffic for supported service consoles through VPC endpoints, including:

Authentication flows – Sign-in, credential exchange, and session token requests
Static assets – JavaScript, CSS, and images that render the console UI
Service console API calls – The backend requests made when users interact with service consoles

How traffic flows from a workload in a private VPC through the three Private Access endpoints, with no path to the public internet (shown in Figure 1):

The operator’s browser requests <region>.console.aws.amazon.com.
The corporate DNS forwarder forwards the query to an Amazon Route 53 Resolver inbound endpoint configured within the VPC, which forwards the traffic to the console VPC endpoint.
Browser traffic flows from on-premises through Direct Connect (or AWS Site-to-Site VPN) to the VPC, and the VPC endpoint routes traffic to the console service over the AWS private network.
The console service redirects to the SignIn endpoint <region>.signin.aws.amazon.com to establish a browser session.
The DNS now resolves to the SignIn VPC endpoint’s private IP addresses, and browser traffic flows to the SignIn service over the AWS private network.
After entering credentials, the SignIn service evaluates VPC endpoint policies, in addition to resource-based policies (RBPs) and RCPs, then redirects back to the console.
The console evaluates VPC endpoint policies, loads static content from the console API VPC endpoint, and enforces identity and resource restrictions when making calls to AWS service APIs.
Users can now access the AWS Management Console over Private Access.

Figure 1: Network isolation architecture

Deploy a pilot of AWS Management Console Private Access

This high-level walkthrough sets up AWS Management Console Private Access for a single AWS Region within one organizational unit (OU). We recommend rolling out incrementally; validate each step before you expand to additional Regions and OUs.

If you want to validate the mechanics of a Private Access deployment before you build out the full solution, the Getting started with a test environment guide walks you through a minimal configuration: a single VPC with the three Private Access endpoints and a permissive policy. This gives you a working setup to experiment with, independent of the deployment described in the rest of this post. To understand how sign-in policies can verify a user’s network location when they access the console, see Controlling console access with resource-based policies and resource control policies.

Prerequisites

You must have the following prerequisites:

An AWS account that’s a member of an organization within AWS Organizations. The example RCP in Step 4 requires the management account access.
A VPC in your chosen AWS Region with optional connectivity from your corporate network. This can be an AWS Direct Connect connection, an AWS Site-to-Site VPN, or workloads running directly in the VPC such as Amazon WorkSpaces or Amazon EC2 browsing instances.
AWS CloudTrail enabled in the same Region to view ConsoleLogin events captured.
Familiarity with AWS PrivateLink interface endpoints and AWS Identity and Access Management (IAM) policy evaluation.

Step 1: Baseline current console access

Before changing anything, use CloudTrail to map how your users access the console today. Search for eventName = ConsoleLogin over a representative window (we recommend 30 days) and review the sourceIPAddress, vpcEndpointId, and awsRegion fields. Identify which identity types are in use: root user, IAM user, SAML federation, and AWS IAM Identity Center. Decide which OU or account you will pilot with.

Note: A misconfigured sign-in policy can lock users out of the console. Avoid piloting in a production or shared account. Instead, use a dedicated test account and configure a break-glass principal (covered in Step 5) before enabling access enforcement.

Step 2: Create the Private Access VPC endpoints

In your chosen Region, create or identify a VPC to host the endpoints, then create three interface VPC endpoints in that VPC:

com.amazonaws.<region>.console for the console.
com.amazonaws.<region>.signin for AWS Sign-In.
com.amazonaws.<region>.console-static for console-only APIs. This endpoint is required only if your VPC has no internet path.

Step 3: Configure private DNS for AWS Management Console Private Access

To use AWS Management Console Private Access, you must configure private DNS so that the console domains—.console.aws.amazon.com, .signin.aws.amazon.com, and the associated static-content domains—resolve to your interface endpoints’ network interfaces within your VPC.

For workloads inside your VPC: If the workloads in your VPC use the default Amazon Route 53 Resolver, no additional DNS configuration is required. When you create each interface endpoint, enable the private DNS name option (set PrivateDnsEnabled = true). The public console domains will then resolve automatically to the endpoint network interfaces inside your VPC. If you use a custom DNS resolver or a private hosted zone, you must configure it explicitly to map the console domains to the endpoint addresses. See Working with private hosted zones for more information. For the complete list of domains and detailed DNS configuration steps, see the AWS Management Console Private Access required endpoints documentation.
For workloads outside your VPC: For workloads that reach the endpoints from outside the VPC—such as corporate offices connecting over AWS Direct Connect or AWS Site-to-Site VPN—ensure that your corporate DNS resolver returns the endpoint addresses for these domains. See Simplify DNS management in a multi-account environment with Route 53 Resolver for more information.

Step 4: Verify private connectivity

Sign in to the console from a workload inside your VPC. The console should load normally. To confirm that traffic is routing through your VPC endpoints, look for the lock icon in the console navigation bar, shown in Figure 2.

Figure 2: Console Private Access

You can also verify in CloudTrail that recent ConsoleLogin events show the vpcEndpointId field populated with one of your endpoint IDs. Here’s an example CloudTrail ConsoleLogin event snippet showing the vpcEndpointId field:

{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "AROA3XFRBF23EXAMPLE:john.doe",
    "arn": "arn:aws:sts::123456789012:assumed-role/Admin/john.doe",
    "accountId": "123456789012"
  },
  "eventTime": "2026-07-08T19:15:32Z",
  "eventSource": "signin.amazonaws.com",
  "eventName": "ConsoleLogin",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "10.0.1.47",
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)...",
  "requestParameters": null,
  "responseElements": {
    "ConsoleLogin": "Success"
  },
  "additionalEventData": {
    "LoginTo": "https://console.aws.amazon.com/console/home",
    "MobileVersion": "No",
    "MFAUsed": "Yes",
    "vpcEndpointId": "vpce-0abc123def456789a"
  },
  "eventID": "a1b2c3d4-5678-90ab-cdef-EXAMPLE11111",
  "eventType": "AwsConsoleSignIn",
  "recipientAccountId": "123456789012"
}

If the AWS Management Console doesn’t load, work through the following checks.

Private DNS is enabled on each interface endpoint (or your custom resolver returns the endpoint addresses)

When Private DNS is enabled, AWS automatically creates the DNS entries that resolve the service domains (such as console.aws.amazon.com) to the private IP addresses of your VPC endpoints. Without it, your browser still routes to the public AWS endpoints, bypassing your private access setup entirely.

Confirm that each endpoint shows PrivateDnsEnabled:

aws ec2 describe-vpc-endpoints \
--filters "Name=vpc-endpoint-type,Values=Interface" \
--query "VpcEndpoints[].{Id:VpcEndpointId,Service:ServiceName,PrivateDns:PrivateDnsEnabled}" \
--output table

Then, from within your VPC, verify that the domains resolve to private addresses:

nslookup console.aws.amazon.com
nslookup signin.aws.amazon.com

# Should return a private IP (e.g., 10.x.x.x), not a public one

Each query should return a private IP address from your VPC CIDR range. If you use a custom DNS resolver instead of the Amazon-provided DNS, ensure your forwarding rules direct the AWS domain queries to the Route 53 Resolver inbound endpoints in your VPC.

The endpoint security groups allow HTTPS (TCP 443) from your workload subnets

Each VPC endpoint creates elastic network interfaces (ENIs) in your subnets, and these ENIs are governed by security groups. If those security groups don’t permit inbound HTTPS traffic from your workloads, the connection fails silently.

Identify the security groups attached to your endpoints:

aws ec2 describe-vpc-endpoints --vpc-endpoint-ids vpce-0abc123def456789a \
  --query "VpcEndpoints[].Groups[].GroupId" --output text

Then verify that each security group allows inbound TCP 443 from your workload subnets:

aws ec2 describe-security-groups --group-ids sg-xxxxxxxx \
  --query "SecurityGroups[].IpPermissions[?ToPort==\`443\`]" \
  --output json

For traffic from outside the VPC (Direct Connect or Site-to-Site VPN), corporate DNS returns the endpoint IPs and the route propagates correctly

If you access the console from an on-premises workstation connected over AWS Direct Connect or AWS Site-to-Site VPN, two additional conditions must be met.

Your corporate DNS must resolve the AWS domains to the VPC endpoint private IPs. From your on-premises machine, run:
nslookup console.aws.amazon.com

If this returns public AWS IPs, your corporate DNS isn’t forwarding queries through Route 53 Resolver. Configure conditional forwarding for the aws.amazon.com and amazonaws.com domains to your Resolver inbound endpoint IPs.
Second, network routes must propagate correctly. Ensure the route table associated with your endpoint subnets has propagated routes from your virtual private gateway (VGW) or transit gateway, so return traffic can reach your on-premises network. Verify this with:
```
aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=subnet-xxxxx" \
  --query "RouteTables[].PropagatingVgws"
```

A quick end-to-end validation: Run traceroute console.aws.amazon.com from your workstation and confirm the path uses private hops only—no traffic should traverse the public internet.

If the console loads but the lock icon is missing

If the console loads but the connection isn’t private (for example, the lock icon is missing), the browser is reaching the console over the public internet instead of through your VPC endpoints.

Run nslookup console.aws.amazon.com from a workload inside the VPC. The result should be a private IP from your VPC CIDR range. A public IP means DNS is bypassing the endpoint, which usually happens because Private DNS has not been enabled on the interface endpoint (set PrivateDnsEnabled = true).
For workloads outside the VPC, make sure your corporate DNS forwards the console domains into the VPC, for example, through an Amazon Route 53 Resolver inbound endpoint.

Step 5: Apply VPC endpoint policies

Attach an endpoint policy to the console and AWS Sign-In endpoints that limits access to identities in your organization. The static-content endpoint doesn’t support endpoint policies.

Begin with a permissive Allow * policy and confirm that traffic routes through the endpoints (you should see the vpcEndpointId field populated in CloudTrail console events). After confirming the routing, add restrictions to your VPC endpoint policy and observe the traffic.

A starter policy uses two condition keys: aws:PrincipalOrgID to restrict identities to your organization and aws:ResourceOrgID to restrict the resources the console can reach to your organization’s resources. The full reference, including additional condition keys and resource-restriction patterns, is in the AWS Management Console Private Access user guide.

For policies beyond the pilot, see Data perimeters on AWS. The data perimeter policy examples GitHub repository covers service-specific considerations for implementing data perimeters in your environment.

Step 6: Apply a Sign-In policy

Sign-In policies deny console authentication requests that don’t match your network or principal conditions. The policy is composed of a pre-authentication statement covering signin:Authenticate and a post-authentication statement covering signin:AuthorizeOAuth2Access and signin:CreateOAuth2Token. Include both statements.

For your pilot, deploy the policy as an RCP from your AWS Organizations management account. When enabled, the RCP applies to all accounts in your organization, so we recommend piloting in a dedicated test organization before rolling it out broadly. Activate enforcement by calling the signin:PutConsoleAuthorizationConfiguration API for the organization in the us-east-1 Region (AWS Sign-In replicates policies globally from there). Resource permission statements have no effect until console authorization is enabled.

Important: Configure at least one excluded principal as a break-glass path before you enable the RCP. The recommended principal is a dedicated IAM role.

Write the permission statements that define the network conditions:
Example – Restrict access to corporate VPC:

aws signin put-resource-permission-statement \
  --source-vpc vpc-0abc123def456789 \
  --requested-region us-west-2 \
  --excluded-principal "arn:aws:iam::123456789012:user/EmergencyAdmin" \
  --region us-east-1

Example – Restrict access to specific IP range:

aws signin put-resource-permission-statement \
  --source-ip "IP_ADDRESS" \
  --excluded-principal "arn:aws:iam::123456789012:role/BreakGlassRole" \
  --region us-east-1

Enable console authorization for the organization to start enforcing the policy.

aws signin put-console-authorization-configuration \
  --target-id <your-target-id> \
  --region us-east-1

Review the consolidated policy that’s now in effect:
```
aws signin get-resource-policy --region us-east-1
```

For policy examples, the AWS Command Line Interface (AWS CLI) reference, and the lockout-recovery procedure, see the sign-in RBP blog post and the Controlling console access with resource-based policies documentation. The same documentation also covers the per-account alternative, which uses an RBP attached to a single account instead of an organization-wide RCP.

Step 7: Add a service VPC endpoint

So far, the console shell loads, the lock icon appears, and your Sign-In policy lets approved identities through. If you sign in to a service console such as the AWS Key Management Service (AWS KMS) console, the page might fail to load resources or hang. The Private Access endpoints carry the console shell, not the service API calls that the console makes on your behalf. In a VPC without an internet gateway, those calls have nowhere to go.

Add a VPC endpoint for the service itself. For the pilot, create an AWS KMS interface endpoint (com.amazonaws.<region>.kms) in the same VPC, with Private DNS enabled. Open the AWS KMS console from inside the VPC and confirm the list of keys loads. Repeat for each service your users need on day one. Please note that a single service console often calls more than one AWS service API. If a console loads but parts of the page show errors or stay empty, the most common cause is a missing endpoint for one of the services it depends on.

The current list of services that support PrivateLink is in the AWS PrivateLink documentation. Service consoles whose services don’t support PrivateLink will not work in a no-internet VPC and need to be handled separately.

Step 8: Hide Regions and services you haven’t configured (optional)

Console links to a service or Region that you don’t have endpoints for will fail inside your VPC. To prevent users from navigating to broken pages, use User Experience Customization (UXC) to hide Regions and services that aren’t part of your Private Access deployment. UXC is configured at the account level and applies to navigation, search results, and service-selection drop-downs.

Step 9: Validate, then expand

After applying the endpoint policies and the Sign-In RCP to one pilot account:

Sign in from inside the corporate network. The session should succeed.
Sign in from outside the corporate network. The session should be denied at the Sign-In step, before reaching the console.
In CloudTrail, confirm ConsoleLogin events show vpcEndpointId populated for traffic from inside the network.
For unexpected denials, look in CloudTrail for ConsoleLogin events with the error message Authorization denied because of a resource-based policy or Authorization denied because of a resource control policy to identify which statement was responsible.

Considerations

A few items worth mentioning before you commit to this design:

AWS IAM Identity Center: IAM Identity Center sign-in support isn’t yet available through a VPC endpoint. Initial single sign-on (SSO) authentication must still transit over the internet.
Programmatic access: Sign-In RBPs and RCPs gate interactive console sign-in. AWS SDK and AWS CLI requests signed with SigV4 aren’t affected. This is also your recovery path: a principal with signin:DeleteConsoleAuthorizationConfiguration permission can disable enforcement programmatically if console authorization is misconfigured.
Apps integrated with AWS Sign-In: Sign-In policies also apply to Amazon Connect, Amazon WorkSpaces, Amazon QuickSight, AWS Health Dashboard, Amazon AppStream 2.0, and Amazon Lightsail when those applications use AWS Sign-In to authenticate.
AWS Management Console Private Access is available in all commercial AWS Regions but supports only a subset of AWS service consoles. See Supported AWS Regions, service consoles, and features in Private Access documentation for more information.
For services that aren’t supported, you can still navigate to other consoles, but will require internet connectivity for the unsupported service consoles and console-only APIs.
Costs: You pay regular AWS PrivateLink endpoint pricing and data processing for each endpoint and each Region you deploy in. The three Private Access endpoints (console, signin, and console-static) plus the service endpoints you already use are the relevant line items.

Conclusion

In this post, we showed you how to extend the AWS data perimeter framework to the AWS Management Console. You routed console traffic through VPC endpoints with AWS Management Console Private Access, restricted console sign-in by network and organization with Sign-In RBPs and RCPs, and configured the console to operate in a VPC without an internet gateway. The four control objectives that you already enforce for API traffic now also apply to the console.

To get started, see the AWS Management Console Private Access documentation. For deployment patterns and Region-by-Region considerations, see the AWS Management Console Private Access reference architectures. For background on the broader pattern, see Establishing a data perimeter on AWS.

If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, start a new thread on the IAM forum on AWS re:Post or contact AWS Support.

How to build a serverless mass email solution with Amazon SES

2026-08-28 Brad Watson

Post Syndicated from Brad Watson original https://aws.amazon.com/blogs/messaging-and-targeting/how-to-build-a-serverless-mass-email-solution-with-amazon-ses/

Sending mass email campaigns presents significant challenges for many organizations. Enterprises often spend millions annually on proprietary email systems that are inflexible and expensive to maintain. These legacy platforms can restrict sending capacity, offer limited control, and require costly licensing agreements. The challenges intensify when handling large-scale communications like automated notifications, bulk marketing campaigns, and system-generated alerts. These scenarios create reliability issues, scaling limitations, and rising costs that impact teams’ ability to communicate effectively with customers.

Recently, a large federal organization faced similar challenges, spending over a million dollars annually on their email campaigns. By building a custom email solution on AWS, they sent a 2 million email campaign for approximately $300. This cost includes Amazon Simple Email Service (Amazon SES) and other AWS services. This transformation cut costs while providing the scalability and flexibility they needed for their growing campaign needs.

This transformation succeeded because building a cloud-native serverless mass email solution offers several advantages:

Cost optimization.
- Pay only for email sent and actual compute resources used.
- Remove costs associated with managing email servers.
- Remove expensive licensing fees and maintenance overhead.
Scalability and reliability.
- Automatically handle varying email volumes without infrastructure changes.
- Support reliable delivery through built-in retry mechanisms and error handling.
- Perform consistently during peak sending periods.
Security and compliance.
- Secure access control through AWS Identity and Access Management (IAM) roles with least-privilege principles.
- Comprehensive audit trails for all email campaigns with detailed logging to support your reporting requirements.
- Detailed logging that customers can use for their compliance and reporting requirements.
- Data encryption in transit and at rest that you can configure.

In this post, we explore the architecture of a cloud-native serverless mass email solution that integrates Amazon SES with AWS Step Functions, Amazon API Gateway, and Amazon DynamoDB. You will learn how these services work together to process email campaigns at scale while minimizing cost. Let’s get started!

Solution overview

The serverless mass email solution consists of two main components: a user-friendly frontend interface and a scalable serverless backend. The frontend operates completely independently from the backend processing system, communicating through RESTful APIs from Amazon API Gateway. With this architecture, you can use the provided frontend interface as-is. Alternatively, you can integrate your own custom UI or existing applications while using the same backend email processing infrastructure.

The following diagram shows the complete architecture of the serverless mass email solution, including how the frontend and backend components connect through API Gateway to process email campaigns.

Complete serverless mass email architecture, with the frontend and backend connected through Amazon API Gateway

Figure 1: Complete architecture

Frontend architecture and user flow

The frontend of the solution prioritizes usability while providing email campaign capabilities. Here’s how the components work together:

Frontend architecture: web interface, Amazon Cognito authentication, and requests through API Gateway to AWS Lambda

Figure 2: Frontend architecture of the SES email application

Login – Users navigate to the web interface URL (hosted on Amazon Simple Storage Service (Amazon S3)) which prompts them to authenticate.
User authentication – Amazon Cognito handles authentication, providing secure user management and restricting access to authorized users.
User interface – After successful authentication, users are redirected to a graphical user interface (GUI) where they can design and save email templates and launch large-scale campaigns (refer to figures 3 and 4).
1. Templates.
  1. Amazon SES supports two types of templates: stored and inline. Stored templates live in SES, and you can reuse them across campaigns. With inline templates, you define the content and variables directly in the email sending request. Both approaches support dynamic personalization by replacing variables with recipient-specific data when the email is sent. For example, you can create a template that personalizes each email with the recipient’s name, custom offers, or any other dynamic content. For detailed information about template capabilities and personalization options, refer to the Amazon SES template documentation.

The following screenshots show the campaign interface, the template creation interface, and the campaign monitoring interface.

Email template creation interface of the mass email application

Figure 3: Email template creation interface

Mass email campaign interface of the application

Figure 4: Mass email campaign interface

Campaign monitoring interface showing the delivery status of a mass email campaign

Figure 5: Campaign monitoring interface

Request processing – Each user action triggers a secure request through Amazon API Gateway to AWS Lambda functions, which then coordinate with our backend processing system.

From the user’s perspective, the experience is similar to using any standard email platform, with the added capability of handling campaigns at scale. This interface helps marketing teams, customer success managers, and business operations staff create and launch email campaigns directly through their browser, without needing to understand complex email protocols.

Backend architecture

After a user initiates an email campaign, our backend orchestrates a series of steps to facilitate reliable, large-scale email delivery. Let’s follow how an email campaign flows through the system:

Figure 6: Backend architecture of the SES email application

As shown in the preceding figure, the backend processes email campaigns through the following steps:

Email campaign processor – When a user creates a new campaign through the GUI, a Lambda function processes the initial request, taking the user’s selected email template and campaign parameters. The function then triggers an AWS Step Functions workflow.
Workflow orchestration – The Step Functions workflow acts as the conductor and coordinates the entire email sending process. It initializes the campaign, sets up necessary configurations, and organizes the campaign into manageable batches.
Recipient processing – Before sending email, the Step Functions workflow retrieves recipient information, including the recipient’s name and email address, from DynamoDB and checks it for accurate delivery details.
Batch email processing – The Step Functions workflow begins organizing the email into manageable batches. The workflow queues these batches in Amazon Simple Queue Service (Amazon SQS), preparing them for processing.
Batch monitoring – As batches move through the system, Step Functions actively monitors their progress, tracking the status of each batch throughout the sending process.
Email sending – When SQS receives a message, it invokes a Lambda function that sends the email to Amazon SES for delivery. The function logs each delivery attempt in DynamoDB, with failed deliveries automatically returning to the SQS queue for retry attempts. It also records successful deliveries to support idempotency and prevent duplicate sends.
Record management – DynamoDB stores an audit trail that tracks both successful and failed delivery attempts, providing detailed logs to support reporting, campaign performance assessments, and compliance efforts.

Using these AWS services, the solution automatically scales from sending a few email to millions without manual intervention or infrastructure provisioning. You pay only for what you use, with no idle server costs. To demonstrate the cost-effectiveness of this architecture: sending 10,000 email costs approximately USD $4, including all AWS service charges. For current pricing details, refer to Amazon SES pricing.

To deploy this solution in your AWS account, refer to the source code on GitHub.

Conclusion

In this post, we explored the architecture of a scalable email sending solution using Amazon SES and other AWS serverless services. This architecture removes the complexity of traditional email infrastructure while providing capabilities for handling large-scale email campaigns. Whether you’re looking to modernize your existing email infrastructure or stand up a new solution, this serverless approach offers the ideal combination of streamlined design, scalability, and cost-effectiveness.

Additional resources

About the authors

Build your own continuous modernization pipeline with AWS Transform custom

2026-08-28 Janardhan Molumuri

Post Syndicated from Janardhan Molumuri original https://aws.amazon.com/blogs/devops/build-your-own-continuous-modernization-pipeline-with-aws-transform-custom/

Introduction

Development velocity has reached new heights with AI-driven development tools and practices. Organizations are generating code faster than ever before. But that speed carries risk. Researchers Anderson, Parker, and Tan warned in MIT Sloan Management Review, “Legacy systems tend to carry hidden debt; layering AI-generated code on top of them creates additional tangled dependencies.” The faster you generate code, the faster technical debt compounds — especially in brownfield environments where outdated frameworks, deprecated libraries, and undocumented services already carry years of accumulated risk.

As organizations accelerate their software development, manual or periodic processes to synchronize dependencies and update documentation no longer keep pace, and technical debt piles up faster than ever. Continuous modernization built into your pipeline enables you to maintain up-to-date dependencies and documentation across repositories on every commit, preventing future tech debt and improving AI agent accuracy and accountability.“

You can embed AI-powered code transformations directly into your CI/CD pipelines, turning modernization from a periodic project into an automated, ongoing practice. AWS gives you two ways to get there. AWS Transform – continuous modernization is the fully managed option, delivering continuous modernization automatically with no pipeline for you to build or maintain. The Do-It-Yourself (DIY) approach assembles the same practices yourself using AWS Transform custom and your existing CI/CD platform. Choose DIY when you need to fit modernization into a specific pipeline (GitHub Actions, AWS CodePipeline, Jenkins, GitLab CI, and so on), or want to customize the workflow with existing tools like Dependabot.

In this post, we cover the DIY approach on how to set up a continuous modernization pipeline using AWS Transform custom and demonstrate it in action.

The Do It Yourself (DIY) path – continuous modernization pipeline with AWS Transform custom

Sample application: instrumentShop

For this walkthrough, we use a dated Java application called instrumentShop (Figure 1) — a Java microservices application built with Spring Boot that simulates an online instrument shop to demonstrate four practices: automated dependency remediation, auto-documentation on every commit, scaling transformations across repositories, and continual learning.

Architecture overview

Figure 1: instrumentShop Java application architecture

The instrumentShop application is a Spring Boot microservices application with a Spring Gateway (v1.5.19) routing traffic from a single HTTP/8010 entry point to four REST services: Agents, Instruments, Consumers, and Products. A Thymeleaf client provides server-side rendering, PostgreSQL 13.1 handles persistence via JDBC, and Hystrix provides circuit-breaking for inter-service calls. A ShopTester utility generates HTTP traffic for testing.

This application is a strong candidate for continuous modernization:

Spring Boot 1.5.19 is years past end of life and carries known CVEs
Hystrix has been in maintenance mode since Netflix deprecated it in 2018
Cross-service coordination — dependency updates must propagate across multiple microservices
Transitive dependency risk — PostgreSQL JDBC drivers and other transitive dependencies accumulate security advisories over time

A typical workflow for the continuous modernization pipeline is shown below (Figure 2):

A developer pushes code to main — GitHub Actions triggers the auto-documentation workflow, generating updated architecture docs and technical debt reports.
Dependabot detects a vulnerable dependency — A PR opens automatically. GitHub Actions triggers the dependency remediation workflow, runs AWS Transform custom to remediate the code, validates with tests, and pushes the result back to the PR.
A platform team defines a new transformation (e.g., “Upgrade Spring Boot to the latest stable release “) — The scheduled GitHub Actions workflow runs the transformation weekly in non-interactive mode across all instrumentShop microservices and other repositories in the portfolio.
The agent learns — Knowledge items from each execution improve future runs, reducing manual intervention over time.

Figure 2: AWS Transform continuous code modernization workflow

Prerequisites

Before setting up the continuous modernization pipeline, ensure you have the following:
An active AWS account with permissions for AWS Transform custom
AWS Transform CLI installed and configured in your development environment
Authentication with AWS credentials configured locally and proper IAM permissions to call AWS Transform
Git installed for cloning sample repositories
GitHub Dependabot enabled on your repository for automated vulnerability detection

Continuous modernization through CI/CD in action

Continuous modernization shifts code transformation from a periodic project into an automated, pipeline-driven practice. Instead of scheduling a “modernization sprint” once a year, your CI/CD pipeline identifies and remediates technical debt on every commit, every dependency alert, and across every repository.

We implement this through four practices, each powered by AWS Transform custom running as a step in GitHub Actions workflows.

Note: This post uses GitHub Actions because the instrumentShop demo repository is built with it. The same AWS Transform CLI (atx) commands work with AWS CodePipeline, Jenkins, GitLab CI, CircleCI, or any CI/CD system that runs shell commands. Continuous modernization is a practice, not a tool choice.

Important: Every atx custom def exec invocation in this post uses the –trust-all-tools flag, which allows the agent to execute tools without interactive confirmation. This is required for non-interactive CI/CD execution. Review your organization’s security policies before enabling this flag in production pipelines.

1. Dependency analysis and remediation

GitHub Dependabot scans your repository for known vulnerabilities and generates alerts when a new vulnerability is added or your dependency graph changes—for example, when you push commits that update packages or versions. However, resolving these alerts requires more than bumping a version number. Upgrading a dependency can introduce breaking API changes, require code modifications, or demand configuration updates.

AWS Transform custom helps handle the code changes needed to resolve the alerts. It runs via a GitHub Actions workflow that triggers automatically to:

Fetch the list of latest Dependabot alerts
Run AWS Transform custom to analyze the alerts and apply code transformations
Run your build and test suite to validate the changes
Create a new pull request for each resolved alert

The workflow calls a shell script that invokes the AWS Transform CLI in headless mode with retry logic. Place this script at the root of your repository:

run_dependabot_alert_fixes.sh:

#!/usr/bin/env bash
set -euo pipefail

# -------------------------------------------------------------------
# run_dependabot_alert_fixes.sh
# Runs the Dependabot alert remediation transformation in headless mode.
# Retries up to MAX_RETRIES times on failure.
#
# Usage:
#   ./run_dependabot_alert_fixes.sh [-n <transformation-name>] [-p <path>] [-c <build-command>]
#
# Defaults:
#   -n  Remediate-Critical-GitHub-Dependabot-Alerts-Java-Maven
#   -p  .                   (current directory)
#   -c  mvn clean install   (Maven build)
# -------------------------------------------------------------------

TRANSFORMATION_NAME="Remediate-Critical-GitHub-Dependabot-Alerts-Java-Maven"
CODE_PATH="."
BUILD_CMD="mvn clean install"
MAX_RETRIES=3

while getopts "n:p:c:" opt; do
  case $opt in
    n) TRANSFORMATION_NAME="$OPTARG" ;;
    p) CODE_PATH="$OPTARG" ;;
    c) BUILD_CMD="$OPTARG" ;;
    *) echo "Usage: $0 [-n <transformation-name>] [-p <path>] [-c <build-command>]" && exit 1 ;;
  esac
done

echo "=== AWS Transform Custom ==="
echo "Transformation: $TRANSFORMATION_NAME"
echo "Code path:      $CODE_PATH"
echo "Build command:  $BUILD_CMD"
echo "============================"

attempt=1
while [ $attempt -le $MAX_RETRIES ]; do
  echo "--- Attempt $attempt of $MAX_RETRIES ---"

  if atx custom def exec \
    -n "$TRANSFORMATION_NAME" \
    -p "$CODE_PATH" \
    -c "$BUILD_CMD" \
    -x -t; then
    echo "=== Transformation completed successfully ==="
    exit 0
  fi

  echo "Attempt $attempt failed."
  attempt=$((attempt + 1))

  if [ $attempt -le $MAX_RETRIES ]; then
    echo "Retrying in 10 seconds..."
    sleep 10
  fi
done

echo "=== All $MAX_RETRIES attempts failed ==="
exit 1

This script accepts optional flags to override the transformation name (-n), code path (-p), and build command (-c). The -x flag enables non-interactive mode and -t enables --trust-all-tools, both required for CI/CD execution. On failure, it retries up to three times with a 10-second backoff.

Your CI/CD workflow must configure AWS credentials and install the AWS Transform CLI before invoking this script. With this setup, Dependabot alerts are reviewed continuously for any changes — not just a version bump, but the complete code adaptation required to make the upgrade work.

2. Auto documentation

Documentation is one of the most neglected aspects of modern software development. Documentation increases accuracy and acts as a contract between requirements and implementation. AWS Transform custom codebase analysis capability generates structured documentation covering architecture, technical debt, code metrics, and migration planning on every incremental update ensuring every Agent or human that modifies the codebase is working from a true “current state”.

By embedding this as a post-push step in your CI/CD pipeline, your documentation stays current automatically. The workflow triggers on every pull request to main, runs your build and test suite, then calls a shell script that invokes AWS Transform custom to generate documentation and commits it back to the PR branch.

Place this script at the root of your repository:

run_code_analysis.sh:

#!/usr/bin/env bash
set -euo pipefail

# -------------------------------------------------------------------
# run_code_analysis.sh
# Runs an AWS Transform custom transformation in headless mode.
# Retries up to MAX_RETRIES times on failure.
#
# Usage:
#   ./run_code_analysis.sh [-n <name>] [-p <path>] [-c <build-cmd>] [-U <pr-url>]
#
# Defaults:
#   -n  GitHub-PR-Context-Codebase-Analysis
#   -p  .                   (current directory)
#   -c  mvn clean install   (Maven build)
#   -U  (empty)             PR URL
# -------------------------------------------------------------------

TRANSFORMATION_NAME="GitHub-PR-Context-Codebase-Analysis"
CODE_PATH="."
BUILD_CMD="mvn clean install"
PR_URL=""
MAX_RETRIES=3

while getopts "n:p:c:U:" opt; do
  case $opt in
    n) TRANSFORMATION_NAME="$OPTARG" ;;
    p) CODE_PATH="$OPTARG" ;;
    c) BUILD_CMD="$OPTARG" ;;
    U) PR_URL="$OPTARG" ;;
    *) echo "Usage: $0 [-n <name>] [-p <path>] [-c <build-cmd>] [-U <pr-url>]" && exit 1 ;;
  esac
done

echo "=== AWS Transform Custom ==="
echo "Transformation: $TRANSFORMATION_NAME"
echo "Code path:      $CODE_PATH"
echo "Build command:  $BUILD_CMD"
echo "PR URL:         $PR_URL"
echo "============================"

attempt=1
while [ $attempt -le $MAX_RETRIES ]; do
  echo "--- Attempt $attempt of $MAX_RETRIES ---"

  if atx custom def exec \
    -n "$TRANSFORMATION_NAME" \
    -p "$CODE_PATH" \
    -c "$BUILD_CMD" \
    -g "additionalPlanContext=$PR_URL" \
    -x -t; then
    echo "=== Transformation completed successfully ==="
    exit 0
  fi

  echo "Attempt $attempt failed."
  attempt=$((attempt + 1))

  if [ $attempt -le $MAX_RETRIES ]; then
    echo "Retrying in 10 seconds..."
    sleep 10
  fi
done

echo "=== All $MAX_RETRIES attempts failed ==="
exit 1

This script accepts optional flags for the transformation name (-n), code path (-p), build command (-c), and PR URL (-U). Pass the PR URL to the agent via the -g flag as additionalPlanContext, giving it awareness of the pull request context when generating documentation. On failure, it retries up to three times with a 10-second backoff.

Your CI/CD workflow must configure AWS credentials and install the AWS Transform CLI before invoking this script. The workflow commits the generated documentation back to the PR branch automatically, keeping your architecture docs and technical debt reports current with every code change.

Every push now updates the documentation (Figures 3 and 4) — reducing knowledge silos and preserving institutional knowledge.

A GitHub pull request triggering the auto-documentation workflow.

Figure 3: PR triggering auto-documentation

Generated documentation output showing architecture and technical debt reports.

Figure 4 – Generated documentation output

3. Scale across repositories

For organizations with hundreds of microservices, transforming one repository at a time doesn’t scale. AWS Transform custom non-interactive mode combined with GitHub Actions matrix strategy allows you to orchestrate transformations across your entire portfolio in parallel. You can run them on demand or on a recurring schedule, so modernization runs as a continuous practice rather than a one-time project.

# .github/workflows/scale-modernization.yml
name: Scale Modernization
on:
  schedule:
    - cron: '0 6 * * 1'
  workflow_dispatch:

jobs:
  transform-repos:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        repo:
          - magnefique-studios/instrumentShop
          - magnefique-studios/orderService
          - magnefique-studios/paymentGateway
    steps:
      - name: Checkout ${{ matrix.repo }}
        uses: actions/checkout@v4
        with:
          repository: ${{ matrix.repo }}
          token: ${{ secrets.GH_PAT }}

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1

      - name: Install ATX CLI
        run: curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash

      - name: Run transformation
        run: |
          atx custom def exec \
            --transformation-name "spring-boot-3-upgrade" \
            --code-repository-path "." \
            --build-command "mvn clean install" \
            --non-interactive \
            --trust-all-tools

Tip: GitHub Actions matrix strategy runs each repository in parallel automatically — no separate orchestration layer needed. For larger portfolios, you can also wrap this in AWS Batch or AWS Fargate for large-scale parallel execution. The AWS Transform web console tracks progress across all repositories in a single view.

4. Continual learning

Each time AWS Transform custom completes a transformation, a memory agent scans the full execution trajectory and extracts lessons. Lessons include patterns that the agent learned, decisions that the agent made during planning, and feedback you provide during execution. AWS Transform custom automatically attaches these lessons to your transformation definition, which improves accuracy in subsequent runs.

AWS Transform custom applies lessons automatically, and each lesson belongs to a category that groups related lessons for review. You can browse and archive any lesson you do not want AWS Transform custom to apply to future runs.This keeps a human in the loop on what the agent “remembers” which matters when the same transformation runs across many repositories with different conventions.

In practice, this means your “Spring Boot 3 Upgrade” transformation gets sharper with each execution. The first repository surfaces the edge cases; once you review the resulting lessons and archive the ones that do not fit, subsequent runs handle those edge cases without intervention.

For production use, you can combine these practices into a single workflow file:

Note: The individual workflows shown in Practices 1–3 are presented separately for clarity. Combine them into a single workflow file as shown here, or keep them as separate workflow files depending on your team’s preference.

# .github/workflows/continuous-modernization.yml
name: Continuous Modernization
on:
  push:
    branches: [main]
  pull_request:
    types: [opened]
  schedule:
    - cron: '0 6 * * 1'

jobs:
  dependency-remediation:
    if: github.actor == 'dependabot[bot]'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.head_ref }}
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1
      - name: Install ATX CLI
        run: curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash
      - name: Remediate dependency changes
        run: |
          atx custom def exec \
            --transformation-name "dependency-remediation" \
            --code-repository-path "." \
            --build-command "mvn clean install" \
            --non-interactive \
            --trust-all-tools

  auto-documentation:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1
      - name: Install ATX CLI
        run: curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash
      - name: Generate documentation
        run: |
          atx custom def exec \
            --transformation-name "codebase-documentation" \
            --code-repository-path "." \
            --build-command "echo 'docs-only'" \
            --non-interactive \
            --trust-all-tools

  weekly-modernization:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1
      - name: Install ATX CLI
        run: curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash
      - name: Run modernization scan
        run: |
          atx custom def exec \
            --transformation-name "tech-debt-analysis" \
            --code-repository-path "." \
            --build-command "mvn clean install" \
            --non-interactive \
            --trust-all-tools

Conclusion

Continuous modernization moves code transformation out of periodic sprints and into your CI/CD pipeline. By combining GitHub Dependabot’s vulnerability detection with AWS Transform custom agent, orchestrated through GitHub Actions, you can:

Remediate dependency vulnerabilities automatically — beyond version bumps to full code adaptation
Keep documentation current with every commit, preserving institutional knowledge
Scale transformations across hundreds of repositories with consistent quality
Improve continuously as the agent accumulates knowledge items from each execution

The instrumentShop sample application demonstrates that even a moderately complex microservices architecture — with end-of-life Spring Boot versions, deprecated libraries like Hystrix, and multiple interconnected services — can be continuously modernized without dedicated modernization sprints.

Ready to get started? This post walked through the do-it-yourself path with AWS Transform custom. If you would rather have continuous modernization delivered as a fully managed service, explore AWS Transform continuous modernization. Either way, visit the AWS Transform documentation to start your continuous modernization journey.

Creating and testing an End User Messaging RCS agent with AWS CLI

2026-08-27 Bruno Giorgini

Post Syndicated from Bruno Giorgini original https://aws.amazon.com/blogs/messaging-and-targeting/creating-and-testing-an-end-user-messaging-rcs-agent-with-aws-cli/

A step-by-step walkthrough for setting up a Rich Communication Services (RCS) test agent, from brand assets to verified inbound messaging.

If you’re still sending plain SMS, you’re leaving a significant experience gap on the table. SMS gives you 160 characters of unformatted text, no branding, and zero confirmation that your message was even read. Rich Communication Services (RCS) changes that entirely. It delivers branded carousels, read receipts, typing indicators, high-resolution images, and verified sender identity, all through the native messaging app your customers already use. No app download required, no new account to create.

Compared to over-the-top (OTT) platforms like WhatsApp or iMessage for Business, RCS doesn’t fragment your audience. It works on an Android’s default messaging app with RCS enabled or an iPhone on iOS 18 or later, which means you reach users where they already are. You are not limited to the ones who happen to have a specific app installed. And compared to building a custom in-app messaging experience, RCS requires no SDK, no UI work, and no convincing users to enable notifications.

With AWS End User Messaging, standing up an RCS agent is surprisingly fast. You configure your brand assets, submit a registration, and within minutes you have a test agent sending branded messages through production APIs. This is real infrastructure, not a sandbox. That means you can prototype, validate your integration, and show stakeholders a working demo before committing to a full build.

This post walks through the entire process of creating an RCS test agent using only the AWS Command Line Interface (AWS CLI). Using the CLI means every step is a repeatable, scriptable command. Need to spin up another agent in a different account or Region? Run the same script and you’re done in minutes. By the end, you will have a working agent that can send branded messages to verified testers and receive inbound messages with automatic responses.

What you will build

In this walkthrough, you will:

Create an RCS agent and configure its brand identity (logo, banner, accent color).
Submit a test registration for automated approval.
Add a verified tester device.
Send your first branded RCS message.
Configure and verify inbound messaging with an automatic keyword response.

Prerequisites

Before you begin, confirm you have:

An AWS account with access to AWS End User Messaging (Amazon Pinpoint SMS and Voice v2 API)
AWS CLI v2.35.12 or later installed and configured with credentials that have pinpoint-sms-voice-v2:* permissions. Version 2.35.12 adds the send-rcs-message command, which you will need for rich media messages (rich cards, carousels, and suggestion chips) beyond this walkthrough. For production deployments, scope the IAM policy down to only the specific actions your application requires. The pinpoint-sms-voice-v2:* scope is convenient for testing but broader than necessary.
rsvg-convert for generating brand asset images from SVG (install with brew install librsvg on macOS)
A test phone that supports RCS messaging.

Verify your setup:

# Confirm AWS credentials are working
aws sts get-caller-identity
# Verify EUM access
aws pinpoint-sms-voice-v2 describe-spend-limits --region us-east-1
# Confirm rsvg-convert is installed
which rsvg-convert

If you use a named AWS CLI profile, append --profile <your-profile> to every AWS command in this walkthrough.

Step 1: Create the RCS agent

The first step is to create an empty RCS agent container. The agent’s display name and branding come from the registration you will configure in Step 2.

aws pinpoint-sms-voice-v2 create-rcs-agent \
  --region us-east-1

Expected output:

{
  "RcsAgentArn": "arn:aws:sms-voice:us-east-1:123456789012:rcs-agent/rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4",
  "RcsAgentId": "rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4",
  "Status": "CREATED",
  "DeletionProtectionEnabled": false,
  "CreatedTimestamp": "2026-07-15T10:00:01.000000-07:00"
}

Save the RcsAgentId and RcsAgentArn values. You will use them throughout this walkthrough.

Next, enable deletion protection to prevent accidental removal. This is especially important once carrier approvals are in place, since re-creating an agent requires a new registration and approval cycle:

aws pinpoint-sms-voice-v2 update-rcs-agent \
  --rcs-agent-id rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 \
  --deletion-protection-enabled \
  --region us-east-1

Step 2: Generate brand assets

Your RCS agent needs a logo (224×224 px, must be under 50 KB as PNG) and a banner (1440×448 px, must be under 200 KB as PNG). Both must be JPEG or PNG format. You can use your own designs as long as they meet these dimension and size requirements. In this example, we generate them as SVGs and convert to PNG.

Create the logo SVG

Create a file named brand-assets/logo.svg:

<svg xmlns="http://www.w3.org/2000/svg" width="224" height="224" viewBox="0 0 224 224"><defs><linearGradient id="bg" x1="0%" y1="0%" x2="100%" y2="100%"><stop offset="0%" style="stop-color:#0D47A1"/><stop offset="100%" style="stop-color:#1565C0"/></linearGradient></defs><rect width="224" height="224" rx="40" fill="url(#bg)"/><g transform="translate(112,90)"><path d="M-48,-36 L48,-36 C54,-36 58,-32 58,-26 L58,16 C58,22 54,26 48,26             L10,26 L0,42 L-10,26 L-48,26 C-54,26 -58,22 -58,16 L-58,-26             C-58,-32 -54,-36 -48,-36 Z" fill="white" opacity="0.95"/><path d="M-20,-12 C-14,-20 14,-20 20,-12" stroke="#0D47A1" stroke-width="4" fill="none" stroke-linecap="round"/><path d="M-14,-2 C-9,-8 9,-8 14,-2" stroke="#0D47A1" stroke-width="4" fill="none" stroke-linecap="round"/><circle cx="0" cy="6" r="4" fill="#0D47A1"/></g><text x="112" y="168" text-anchor="middle" font-family="Arial, Helvetica, sans-serif" font-size="16" font-weight="bold" fill="white">AWS EUM</text><text x="112" y="188" text-anchor="middle" font-family="Arial, Helvetica, sans-serif" font-size="12" fill="white" opacity="0.85">DEMO</text></svg>

Create a file named brand-assets/banner.svg:

<svg xmlns="http://www.w3.org/2000/svg" width="1440" height="448" viewBox="0 0 1440 448"><defs><linearGradient id="bannerBg" x1="0%" y1="0%" x2="100%" y2="100%"><stop offset="0%" style="stop-color:#0D47A1"/><stop offset="50%" style="stop-color:#1565C0"/><stop offset="100%" style="stop-color:#0D47A1"/></linearGradient></defs><rect width="1440" height="448" fill="url(#bannerBg)"/><circle cx="200" cy="224" r="300" fill="white" opacity="0.03"/><circle cx="1300" cy="100" r="250" fill="white" opacity="0.04"/><text x="720" y="190" text-anchor="middle" font-family="Arial, Helvetica, sans-serif" font-size="56" font-weight="bold" fill="white">    AWS End User Messaging  </text><text x="720" y="250" text-anchor="middle" font-family="Arial, Helvetica, sans-serif" font-size="48" font-weight="bold" fill="white" opacity="0.9">    Demo  </text><text x="720" y="320" text-anchor="middle" font-family="Arial, Helvetica, sans-serif" font-size="24" fill="white" opacity="0.7">    Rich messaging experiences, powered by AWS  </text></svg>

Convert to PNG

rsvg-convert -w 224 -h 224 brand-assets/logo.svg -o brand-assets/logo.png
rsvg-convert -w 1440 -h 448 brand-assets/banner.svg -o brand-assets/banner.png

Verify the file sizes. The logo must be under 50 KB and the banner under 200 KB:

ls -la brand-assets/*.png
# logo.png   ~9 KB
# banner.png ~79 KB

Step 3: Create and configure the registration

RCS agents require a registration that contains all brand details. For testing, use the TEST_RCS_LAUNCH_REGISTRATION type.

Create the registration

aws pinpoint-sms-voice-v2 create-registration \
  --registration-type TEST_RCS_LAUNCH_REGISTRATION \
  --region us-east-1

Expected output:

{
  "RegistrationId": "registration-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4",
  "RegistrationType": "TEST_RCS_LAUNCH_REGISTRATION",
  "RegistrationStatus": "CREATED",
  "CurrentVersionNumber": 1
}

Save the RegistrationId.

Link the registration to your agent

aws pinpoint-sms-voice-v2 create-registration-association \
  --registration-id registration-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 \
  --resource-id rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 \
  --region us-east-1

Upload brand assets

Upload the logo and banner as registration attachments. Note that --attachment-body and --attachment-url cannot be used together. Use --attachment-body with the fileb:// prefix:

# Upload logo
aws pinpoint-sms-voice-v2 create-registration-attachment \
  --attachment-body fileb://brand-assets/logo.png \
  --region us-east-1
# Save: RegistrationAttachmentId (e.g., attachment-1111aaaa2222bbbb3333cccc4444dddd)
# Upload banner
aws pinpoint-sms-voice-v2 create-registration-attachment \
  --attachment-body fileb://brand-assets/banner.png \
  --region us-east-1
# Save: RegistrationAttachmentId (e.g., attachment-5555eeee6666ffff7777aaaa8888bbbb)

Set registration fields

The registration has 23 fields. Each field has a specific type that determines which CLI parameter to use:

Field type	CLI parameter	Example
TEXT	`--text-value`	`--text-value "My Brand"`
SELECT	`--select-choices`	`--select-choices "MULTI_USE"`
ATTACHMENT	`--registration-attachment-id`	`--registration-attachment-id "attachment-abc123"`

Do not use --field-values. That parameter does not exist in this CLI.

Set all the TEXT fields:

REG_ID="registration-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4"
REGION="us-east-1"

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.brandName" \
  --text-value "AWS End User Messaging Demo" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.senderDisplayName" \
  --text-value "AWS End User Messaging Demo" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.agentDescription" \
  --text-value "Experience the power of rich messaging with AWS End User Messaging" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.accentColor" \
  --text-value "#0D47A1" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.contactPhoneNumber" \
  --text-value "+12065550100" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.contactPhoneLabel" \
  --text-value "Call Us" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.contactEmailAddress" \
  --text-value "[email protected]" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.contactEmailLabel" \
  --text-value "Email Us" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.contactWebsite" \
  --text-value "https://www.example.com" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.contactWebsiteLabel" \
  --text-value "Visit Website" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.privacyPolicyUrl" \
  --text-value "https://www.example.com/privacy" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.privacyPolicyLabel" \
  --text-value "Privacy Policy" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.termsAndConditionsUrl" \
  --text-value "https://www.example.com/terms" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.termsAndConditionsLabel" \
  --text-value "Terms and Conditions" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.serviceName" \
  --text-value "AWS End User Messaging Demo RCS Agent" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.monthlyRcsVolume" \
  --text-value "1000" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "complianceKeywords.helpResponse" \
  --text-value "Reply STOP to opt out. For help, contact [email protected]" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "complianceKeywords.stopResponse" \
  --text-value "You have been unsubscribed. No more messages will be sent." \
  --region $REGION

Set the SELECT fields. These use --select-choices instead of --text-value:

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.useCase" \
  --select-choices "MULTI_USE" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.billingCategory" \
  --select-choices "CONVERSATIONAL" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.averageMonthlyRcsFrequency" \
  --select-choices "10" \
  --region $REGION

Set the ATTACHMENT fields. These use --registration-attachment-id:

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.logoImage" \
  --registration-attachment-id "attachment-1111aaaa2222bbbb3333cccc4444dddd" \
  --region $REGION

aws pinpoint-sms-voice-v2 put-registration-field-value \
  --registration-id $REG_ID \
  --field-path "agentDetails.bannerImage" \
  --registration-attachment-id "attachment-5555eeee6666ffff7777aaaa8888bbbb" \
  --region $REGION

A note on accent color

The accent color must meet a 4.5:1 contrast ratio against white. This is the WCAG AA accessibility standard, enforced to make sure the text is readable for users with visual impairments. Colors with an HSL lightness value above ~45% will typically fail this threshold and be rejected with ACCENT_COLOR_CONTRAST_INSUFFICIENT. Safe choices include #0D47A1 (blue), #1B5E20 (green), #BF360C (orange), #B71C1C (red), and #4A148C (purple). If you are using a custom brand color, verify it passes before submitting using the WebAIM Contrast Checker.

Submit the registration

aws pinpoint-sms-voice-v2 submit-registration-version \
  --registration-id $REG_ID \
  --region $REGION

Expected output:

{
  "RegistrationId": "registration-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4",
  "VersionNumber": 1,
  "RegistrationVersionStatus": "SUBMITTED"
}

Step 4: Wait for approval

Poll the registration and agent status. Test registrations typically complete within a few minutes.

# Check registration status
aws pinpoint-sms-voice-v2 describe-registrations \
  --registration-ids $REG_ID \
  --query 'Registrations[0].{Status:RegistrationStatus,Version:CurrentVersionNumber}' \
  --region $REGION

# Check agent status
aws pinpoint-sms-voice-v2 describe-rcs-agents \
  --query "RcsAgents[?RcsAgentId=='rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4'].{Status:Status,TestingStatus:TestingAgent.Status}" \
  --region $REGION

You will see the status progress through these stages:

Registration status	Agent status	Testing status	Meaning
SUBMITTED	PENDING	PENDING	Under review
REVIEWING	PENDING	PENDING	Automated checks in progress
COMPLETE	TESTING	ACTIVE	Ready to use

Wait until TestingAgent.Status shows ACTIVE before proceeding.

NOTE: If the registration returns REQUIRES_UPDATES, run describe-registration-field-values to find fields with a DeniedReason. Create a new registration version with create-registration-version, re-populate all 23 fields (new versions do not inherit values), fix the issue, and re-submit.

Step 5: Add a verified tester

Wait at least 120 seconds after agent creation before adding testers. Then register your test device:

aws pinpoint-sms-voice-v2 create-verified-destination-number \
  --destination-phone-number +12065550199 \
  --rcs-agent-id rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 \
  --region $REGION

You will receive a tester invitation on your phone within 2 to 20 minutes from “RBM Tester Management.” On iPhone, check the Unknown Senders folder. Tap “Make me a tester” to accept.

After accepting, verify the status:

aws pinpoint-sms-voice-v2 describe-verified-destination-numbers \
  --filters Name=rcs-agent-id,Values=rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 \
  --region $REGION \
  --query 'VerifiedDestinationNumbers[].{Phone:DestinationPhoneNumber,Status:Status}'

Expected output once accepted:

[
  {
    "Phone": "+12065550199",
    "Status": "VERIFIED"
  }
]

Step 6: Send your first RCS message

Before sending, check for potential blockers.

Check the protect configuration

Verify that the US is not blocked in your account’s default protect configuration:

# List protect configurations
aws pinpoint-sms-voice-v2 describe-protect-configurations --region $REGION

# Check US status on the default (account-default) protect configuration
aws pinpoint-sms-voice-v2 get-protect-configuration-country-rule-set \
  --protect-configuration-id <your-protect-config-id> \
  --number-capability SMS \
  --query 'CountryRuleSet.US' \
  --region $REGION

If the US status is BLOCK, update it to ALLOW:

aws pinpoint-sms-voice-v2 update-protect-configuration-country-rule-set \
  --protect-configuration-id <your-protect-config-id> \
  --country-rule-set-updates '{"US":{"ProtectStatus":"ALLOW"}}' \
  --number-capability SMS \
  --region $REGION

Check the opt-out list

aws pinpoint-sms-voice-v2 describe-opted-out-numbers \
  --opt-out-list-name Default \
  --region $REGION

If your test number appears in the list, remove it:

aws pinpoint-sms-voice-v2 delete-opted-out-number \
  --opt-out-list-name Default \
  --opted-out-number +12065550199 \
  --region $REGION

Now, send the test message:

aws pinpoint-sms-voice-v2 send-text-message \
  --destination-phone-number +12065550199 \
  --origination-identity rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 \
  --message-body "Hello from AWS End User Messaging Demo! This is your first RCS test message." \
  --message-type TRANSACTIONAL \
  --region $REGION

Expected output:

{"MessageId": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"}

Check your phone. You should see a branded message from your agent with the logo and accent color you configured. On iPhone, check the Unknown Senders folder.

Step 7: Configure and test inbound messaging

With inbound messaging, your agent can respond to messages that testers send back. Configure an automatic keyword response, then verify it end to end.

Set up an automatic keyword response

The put-keyword API configures an automatic reply when someone sends a specific keyword to your agent. With it, you can verify inbound messaging without writing any backend code:

aws pinpoint-sms-voice-v2 put-keyword \
  --keyword RCSINBOUNDTESTING \
  --keyword-action AUTOMATIC_RESPONSE \
  --keyword-message "Inbound test successful! Your message was received." \
  --origination-identity rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 \
  --region $REGION

Test inbound messaging

While the previous steps used the CLI exclusively, the inbound testing deep link is most easily accessed through the console. Navigate to your agent and use the Testing tab to generate the deep link:

Open the AWS End User Messaging console: https://console.aws.amazon.com/sms-voice/home?region=[REGION]#/rcs-agents.
Select your agent and choose the Testing tab.
Choose Inbound deep link.
Enter RCSINBOUNDTESTING in the message body field.
Choose Generate link.
Scan the QR code with your test phone. The message is pre-filled.
Send the message.

You should receive the automatic response: “Inbound test successful! Your message was received.”

Clean up

To avoid unexpected charges, remove the resources created during this walkthrough when you are finished testing. You must delete resources in the following order. Attempting to delete the agent before its registration results in a ConflictException: RESOURCE_NOT_EMPTY error.

# 1. Remove the keyword
aws pinpoint-sms-voice-v2 delete-keyword \
  --keyword RCSINBOUNDTESTING \
  --origination-identity rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 \
  --region $REGION

# 2. Remove verified tester
aws pinpoint-sms-voice-v2 delete-verified-destination-number \
  --verified-destination-number-id <your-verified-number-id> \
  --region $REGION

# 3. Disable deletion protection
aws pinpoint-sms-voice-v2 update-rcs-agent \
  --rcs-agent-id rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 \
  --no-deletion-protection-enabled \
  --region $REGION

# 4. Delete the registration
aws pinpoint-sms-voice-v2 delete-registration \
  --registration-id registration-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 \
  --region $REGION

# 5. Delete the agent
aws pinpoint-sms-voice-v2 delete-rcs-agent \
  --rcs-agent-id rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4 \
  --region $REGION

If you modified the protect configuration (changed US from BLOCK to ALLOW), revert it to its original state if your account does not need US messaging enabled.

Summary

You now have a working RCS test agent that can send and receive branded messages. Here is a recap of the resources created:

Resource	Value
Agent ID	`rcs-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4`
Registration ID	`registration-a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4`
Region	`us-east-1`
Console	`https://us-east-1.console.aws.amazon.com/sms-voice/home?region=us-east-1#/rcs-agents`

Registration field reference

For reference, here is the complete list of registration fields and their types:

Field	Type	Requirement
`agentDetails.brandName`	TEXT	Required
`agentDetails.serviceName`	TEXT	Required
`agentDetails.senderDisplayName`	TEXT	Required
`agentDetails.useCase`	SELECT	Required
`agentDetails.agentDescription`	TEXT	Required
`agentDetails.bannerImage`	ATTACHMENT	Required
`agentDetails.logoImage`	ATTACHMENT	Required
`agentDetails.accentColor`	TEXT	Required
`agentDetails.contactPhoneNumber`	TEXT	Conditional
`agentDetails.contactPhoneLabel`	TEXT	Conditional
`agentDetails.contactEmailAddress`	TEXT	Conditional
`agentDetails.contactEmailLabel`	TEXT	Conditional
`agentDetails.contactWebsite`	TEXT	Conditional
`agentDetails.contactWebsiteLabel`	TEXT	Conditional
`agentDetails.privacyPolicyUrl`	TEXT	Required
`agentDetails.privacyPolicyLabel`	TEXT	Optional
`agentDetails.termsAndConditionsUrl`	TEXT	Required
`agentDetails.termsAndConditionsLabel`	TEXT	Optional
`agentDetails.averageMonthlyRcsFrequency`	SELECT	Required
`agentDetails.billingCategory`	SELECT	Required
`agentDetails.monthlyRcsVolume`	TEXT	Required
`complianceKeywords.helpResponse`	TEXT	Conditional
`complianceKeywords.stopResponse`	TEXT	Conditional

Troubleshooting

Error	Resolution
`ACCENT_COLOR_CONTRAST_INSUFFICIENT`	Use a darker accent color with 4.5:1 contrast ratio against white. Create a new registration version and re-populate all fields.
`DESTINATION_COUNTRY_BLOCKED_BY_PROTECT_CONFIGURATION`	Update the protect configuration to set the US to `ALLOW` for SMS capability.
`DESTINATION_PHONE_NUMBER_OPTED_OUT`	Remove the number from the Default opt-out list with `delete-opted-out-number`.
Registration `REQUIRES_UPDATES`	Run `describe-registration-field-values` to find fields with `DeniedReason`. Create a new version, re-populate all 23 fields, fix the issue, and re-submit.
No tester invitation received	Wait up to 20 minutes. Check the Unknown Senders folder on iPhone. Verify the agent status is ACTIVE.
Message delivered as SMS instead of RCS	Confirm the agent is ACTIVE, the device supports RCS, and you used the correct origination identity.

Next steps

With your test agent running, you can explore richer message types such as cards and carousels, set up event destinations for programmatic inbound message handling, or add more verified testers. For production use, submit a full launch registration instead of a test registration.

For an overview of the business case for RCS and implementation strategy, see Upgrade business messaging with RCS on AWS. For sample code and scripts that automate this walkthrough, see the sample-rcs-agent-setup-and-send-messages repository on GitHub. For more information, see the AWS End User Messaging service page and the RCS documentation.

About the authors

Detecting multi-stage attacks on AWS: A guide to cross-service signal correlation

2026-08-26 Nisha Kashyap

Post Syndicated from Nisha Kashyap original https://aws.amazon.com/blogs/security/detecting-multi-stage-attacks-on-aws-a-guide-to-cross-service-signal-correlation/

A single alert from one security service tells you something happened. Read that signal alongside activity from other services and your own business context, and you will know whether what happened is part of a multi-stage attack.

Consider a short sequence. An identity calls GetCallerIdentity from a source address it hasn’t previously used. Within minutes, that same identity runs a burst of List and Describe calls across several services, and some of them fail with AccessDenied. Soon after, a large volume of data leaves your environment toward a domain that was registered last week. Amazon GuardDuty might already flag pieces of this, such as the reconnaissance from an unfamiliar source, through finding types like Recon:IAMUser/* or Discovery:S3/*. What you gain from correlating the pieces yourself is a single view of the sequence, tied to your own business context, so you can act on the whole rather than triaging findings one at a time.

This post is for security engineers and security operations teams who run Amazon Web Services (AWS) detection services and want to catch patterns specific to their environment. You will see how AWS detection and your business context fit together, and how to build correlations that use that context. The examples run in Amazon CloudWatch Logs Insights so you can try them today, and the closing section describes how to grow them into an automated pipeline. The walkthrough later in this post lists the prerequisites for these queries.

Start with AWS detection services

Begin with the AWS detection services. They cover the threats common across customers, and everything in this post is built on them.

Amazon GuardDuty analyzes activity across AWS CloudTrail, Amazon Virtual Private Cloud (Amazon VPC) Flow Logs, DNS query logs, Amazon Simple Storage Service (Amazon S3) data events, and more and it produces high confidence findings out of the box.
Amazon Detective (a security investigation service that visualizes relationships between resources and findings) lets you investigate findings with graph-based context across services and accounts.
AWS Security Hub aggregates security findings into a single prioritized dashboard.
Amazon Security Lake centralizes security data into a purpose-built data lake, stored in the Open Cybersecurity Schema Framework (OCSF) format for long-term analysis.
Amazon Inspector (a vulnerability management service that scans workloads for software vulnerabilities) correlates vulnerability data with workload context.

Turn these on and tune them before you build anything custom. Tuning means adjusting sensitivity to reduce false positives for your environment, choosing which data sources each service monitors, and suppressing findings for known-good patterns.

GuardDuty correlates multi-stage attacks for you

Before you build anything by hand, see what GuardDuty already does for you. Amazon GuardDuty Extended Threat Detection correlates signals across multiple data sources including AWS CloudTrail, Amazon S3 data events, runtime monitoring, Amazon Elastic Kubernetes Service (Amazon EKS) audit logs, and more, then raises a single critical severity attack sequence finding when it spots a multi-stage pattern. It recognizes sequences such as credential compromise followed by data exfiltration, maps them to MITRE ATT&CK tactics, and attaches a timeline and remediation guidance. If you have GuardDuty enabled today, then GuardDuty Extended Threat Detection is already enabled by default and needs no queries from you. For details on how GuardDuty charges apply, see Amazon GuardDuty pricing.

The credential compromise sequence in the opening example is the kind of universal pattern GuardDuty Extended Threat Detection is built to catch, so rely on it for those. Attack sequence findings show up in the GuardDuty console next to your other findings, and they route to Security Hub and your response workflows the same way.

GuardDuty handles the threats that look the same in every account. What it doesn’t have is the context that makes a given action suspicious in your account. That’s what you provide.

Add your business context

Business context is what only you know about your environment: which buckets hold sensitive data, which principals have a reason to touch which resources, which role chains your policy permits, and when your production change windows open. GuardDuty Extended Threat Detection learns from patterns common across customers, but it can’t answer these environment-specific questions. Express them as correlations and you add a detection layer tuned to your environment. Each of the following four patterns turns one of these facts into a query.

Run these queries in the AWS Management Console for CloudWatch by choosing Logs, then Logs Insights, using the CloudWatch Logs Insights query language. Most read CloudTrail events from a CloudWatch Logs log group that your trail delivers to. If your trail writes only to Amazon S3, add CloudWatch Logs delivery on the trail, or run equivalent queries in Amazon Athena (a serverless query service for analyzing data in Amazon S3 using SQL).

Note: The queries and code in this post use placeholder values. Replace them with your own before running: your-sensitive-bucket (your S3 bucket name), your-key-id (your AWS KMS key ID), region (your AWS Region, such as us-east-1), account-id (your 12-digit AWS account ID), and aws-cloudtrail-logs-my-trail (your CloudTrail log group name).

A note on multi-account environments. In AWS Organizations, an organization trail delivers every account’s events to one log group, so these queries work as-is but return cross-account results. Filter by recipientAccountId for account-scoped views. Without an organization trail, run queries per account or use Amazon Security Lake as a central query surface.

The attack chain mapped to AWS services

Multi-stage attacks move through five phases, and each phase leaves a signal in a different service. These signals surface across three log sources: CloudTrail, which records API activity in your account; Amazon VPC Flow Logs, which capture network connection metadata; and Amazon Route 53 Resolver query logs, which record DNS queries from your VPCs.

Initial access – Stolen credentials reach your environment. CloudTrail records GetCallerIdentity, GetSessionToken, or AssumeRole from an unfamiliar source.
Discovery – The threat actor enumerates with List, Describe, and Get calls, often triggering AccessDenied responses.
Privilege escalation – The threat actor chains roles or edits policies. CloudTrail records AssumeRole sequences, PutRolePolicy, or CreateAccessKey.
Lateral movement – The threat actor moves across accounts or AWS Regions, assuming roles and creating resources in unfamiliar places.
Exfiltration – Data leaves through GetObject calls at scale, large outbound transfers in VPC Flow Logs, and DNS queries in Route 53 Resolver query logs to recently registered domains.

Figure 1 shows the five attack phases mapped to the AWS log source that records each one.

Figure 1: Attack chain mapped to AWS services

GuardDuty Extended Threat Detection watches this chain for universal patterns. The four patterns that follow add the dimension you supply: your business context.

Pattern one: Sensitive data access by an unexpected principal

Your data classification and access norms drive this detection. One bucket holds customer records, another holds public web assets, and you know which principals have a reason to read the customer records, which are sensitive. Encode that knowledge and an ordinary looking read turns into something worth chasing.

Three signals converge here. CloudTrail shows GetObject at volume on a bucket you’ve classified as sensitive. The principal isn’t on your list of expected readers for that bucket. And VPC Flow Logs show a large outbound transfer from the same source in the same window, while DNS query logs show a recently registered destination domain, which together increase your confidence that there’s a potential threat.

CloudTrail management events don’t record GetObject. You must turn on CloudTrail data events for the buckets you care about to capture GetObject. Many teams miss GetObject because data events weren’t enabled on the relevant buckets.

This query shows bulk reads on a sensitive bucket, grouped by principal. Run it in CloudWatch Logs Insights with your CloudTrail log group selected.

fields @timestamp, userIdentity.arn, requestParameters.bucketName
| filter eventSource = "s3.amazonaws.com" and eventName = "GetObject"
| filter requestParameters.bucketName = "your-sensitive-bucket"
| stats count(*) as objectReads,
        count_distinct(requestParameters.key) as distinctObjects
        by userIdentity.arn, bin(10m)
| filter objectReads > 100
| sort objectReads desc

The threshold of 100 is a placeholder. Run the query over a week of normal activity, find the ninety-fifth percentile read count for that bucket, and set the threshold above it. Then check each principal the query returns against your expected reader list. A principal that isn’t on the list, reading at volume, is the result to investigate.

To corroborate, look for a matching outbound transfer. Switch the log group selector to your VPC Flow Logs log group and run this.

fields @timestamp, srcAddr, dstAddr, bytes
| filter action = "ACCEPT"
# exclude RFC 1918 private ranges so only external destinations remain
| filter dstAddr not like /^10\./
        and dstAddr not like /^192\.168\./
        and dstAddr not like /^172\.(1[6-9]|2[0-9]|3[0-1])\./
| stats sum(bytes) as totalBytes by srcAddr, dstAddr, bin(10m)
| filter totalBytes > 1000000000
| sort totalBytes desc

The Amazon S3 query returns a principal, and the Flow Logs query works on IP addresses, so you translate one into the other. The worked example later in this post covers that translation in full.

Picture an analytics role that reads a reporting bucket all day. One afternoon, it reads a thousand objects from your customer records bucket instead. GuardDuty stays quiet, because an authenticated role making valid GetObject calls isn’t suspicious anywhere else. Your query flags it, because that role isn’t on the expected reader list for that bucket. The classification you applied is what turns silence into a signal.

Figure 2 shows a bulk read from a sensitive bucket in CloudTrail, a large outbound transfer in VPC Flow Logs, and a young domain resolution in Route 53 Resolver logs.

Figure 2: Three signals converging within a single time window to indicate exfiltration

Pattern two: A role chain that crosses your access policy

Picture a deployment that assumes one role to build, then a second to release. For one principal, that two-hop AssumeRole chain is routine; for a different principal it’s a policy violation. This pattern relies on your trust topology—the chains your organization permits—so put that knowledge in the query.

This pattern needs three conditions:

CloudTrail shows several AssumeRole calls from the same source inside a short window
The chain ends in a sensitive action such as CreateAccessKey, PutRolePolicy, or AttachUserPolicy
The starting identity isn’t one your policy expects to run that chain

In CloudWatch Logs Insights, select your CloudTrail log group and run this query, which surfaces chains of two or more hops.

fields @timestamp, userIdentity.arn, requestParameters.roleArn, sourceIPAddress
| filter eventName = "AssumeRole"
| stats count(*) as assumeCount,
        count_distinct(requestParameters.roleArn) as rolesAssumed
        by sourceIPAddress, bin(5m)
| filter assumeCount >= 2 and rolesAssumed >= 2
| sort assumeCount desc

Two hops is the minimum for a chain; raise the count if your environment chains roles often. Your deployment pipeline probably assumes several roles an hour, as do AWS service principals such as AWS Security Hub. Exclude the identities you expect to see assuming multiple roles, including your pipeline role and known AWS service principals. What’s left is the set to investigate, such as a person assuming several roles at an odd hour and ending in a new access key. Treat that distinction as data: list the identities and actions you consider normal, and review the chains that fall outside the list.

Pattern three: An encryption key used outside its owning workload

Resource ownership is the signal here. A given AWS Key Management Service (AWS KMS) key creates and controls the encryption keys for a workload, and a single key should serve a single workload, such as a payments service. A Decrypt call against it is a valid, authorized API action, so nothing about the call itself looks wrong. The ownership rule you set is what makes another principal’s use of the key worth a second look.

This pattern applies only to customer-managed keys scoped to one workload. It doesn’t apply to AWS-managed keys (alias/aws/*) or to customer-managed keys intentionally shared across services. Confirm single-workload intent from the key policy’s Principal block before deploying this rule.

Two conditions indicate misuse:

CloudTrail shows Decrypt or GenerateDataKey calls on a key that’s tied to one workload
The calling principal isn’t the role that owns that workload

Against your CloudTrail log group, run this query to list the principals that called a specific key.

fields @timestamp, userIdentity.arn, eventName
| filter eventSource = "kms.amazonaws.com"
| filter eventName in ["Decrypt", "GenerateDataKey", "Encrypt"]
| filter resources.0.ARN = "arn:aws:kms:region:account-id:key/your-key-id"
| stats count(*) as keyUses by userIdentity.arn, eventName
| sort keyUses desc

Compare what comes back against the one workload role you expect. A principal you don’t recognize on that key is the signal. Because key misuse is an early move in data theft, this correlation catches activity that only your ownership knowledge can flag.

Consider a key that wraps your payments database. The payments service role calls it in normal operation, and nothing else should. If a developer role or a freshly created role runs Decrypt against it, the call succeeds and reads as ordinary in isolation. The reason it matters is the ownership rule you hold in your head and now state in this query.

Pattern four: A privileged action outside your change window

Start with the query, then read what it means.

fields @timestamp, userIdentity.arn, eventName, sourceIPAddress
| filter eventName in ["PutRolePolicy", "AttachRolePolicy",
        "CreateAccessKey", "AuthorizeSecurityGroupIngress", "PutBucketPolicy"]
| stats count(*) as sensitiveChanges by userIdentity.arn, eventName, sourceIPAddress
| sort sensitiveChanges desc

Run it against your CloudTrail log group, scoped to your off-hours window when you schedule it, so it returns only activity outside the change window. Your change process defines what normal looks like here: production security and identity changes flow through a pipeline during defined hours, run by a known actor. A console-driven policy change at 2:00 AM, made by a person rather than the pipeline, doesn’t fit those expectations. The signal is a sensitive change such as PutRolePolicy or AuthorizeSecurityGroupIngress, made outside the window, by a person rather than your pipeline role.

Exclude the actors you expect, such as your deployment pipeline role, your patch automation role, and AWS service principals like AWS CloudFormation and AWS Systems Manager. What remains is privileged change made outside your process, which is both what an attacker does to establish persistence and what your own change discipline says shouldn’t happen.

Your pipeline might open security group rules during a deployment every weekday afternoon. A person opening a security group rule at midnight on a weekend is the same API call carrying a very different meaning. The schedule and the actor, both facts you define, are what separate the two.

Build your first correlation rule

The following walkthrough uses pattern one as a complete example. The other three patterns follow the same design with their own queries.

Prerequisites

These prerequisites feed the queries in this walkthrough. Confirm each one before you start:

A CloudTrail trail logging management events to a CloudWatch Logs log group
CloudTrail data events enabled for your sensitive S3 buckets
GuardDuty enabled, with its protection plans and Extended Threat Detection
VPC Flow Logs on for your production VPCs
Amazon Route 53 Resolver query logging on

CloudTrail, GuardDuty, VPC Flow Logs, and Route 53 Resolver query logging provide the raw signals that your correlations connect. Without them, the queries in this post return empty results.

Step 1: Record the bucket and its expected readers

Choose one sensitive bucket to monitor, and write down the principals allowed to read it. Store the list where your automation can reach it, such as a configuration file in version control or an Amazon DynamoDB table (a managed NoSQL database).

{
  "customer-records-prod": [
    "arn:aws:iam::123456789012:role/AnalyticsPipeline",
    "arn:aws:iam::123456789012:role/ComplianceAudit"
  ],
  "financial-data-archive": [
    "arn:aws:iam::123456789012:role/FinanceReporting"
  ]
}

This example hardcodes the list for simplicity. In production, load it from a DynamoDB table or Parameter Store so you can update it without redeploying.

Step 2: Baseline before you set a threshold

Run the pattern one query over one week of normal activity. Find the 95th percentile read count for the bucket and use a value greater than that as your alert threshold. This step keeps legitimate high-volume access from generating false positives later.

Set the THRESHOLD_READS environment variable to this value when you configure the function in Step 5.

Step 3: Run the access query

In the CloudWatch console:

Choose Logs, then choose Logs Insights.
In the Select log group(s) dropdown, select your CloudTrail log group.
Set the time range to 3h (the last three hours).
In the query editor, paste the pattern one query.
Replace your-sensitive-bucket with your bucket name.
Choose Run query.
Review the principals in the results table.
Compare each principal against your expected reader list from step 1, and flag any that are not on it.

Each result includes a principal that step 4 translates into an IP address.

Step 4: Correlate with network activity

CloudTrail logs actions by AWS Identity and Access Management (IAM) principal, while VPC Flow Logs record traffic by IP address. To connect the two signals, translate the principal into its address.

For a role attached to an Amazon Elastic Compute Cloud (Amazon EC2) instance, the userIdentity.principalId field includes the instance ID after the colon, in the form AROAEXAMPLE:i-1234567890abcdef0. Copy the instance ID and look up its private IP address.

aws ec2 describe-instances \
  --instance-ids i-1234567890abcdef0 \
  --query "Reservations[0].Instances[0].PrivateIpAddress" \
  --output text

Other compute types differ. A VPC-connected AWS Lambda function sends traffic through elastic network interfaces in your subnets, so correlate on those interface addresses. An Amazon Elastic Container Service (Amazon ECS) task records its network interface in task metadata. For a plain assumed-role session with no instance behind it, the sourceIPAddress field in CloudTrail already holds the caller’s address, so you correlate on it directly.

Run the Flow Logs query from pattern one, filtering srcAddr to that address within 10 minutes of the Amazon S3 read timestamp. A match places the same source behind both the sensitive read and a large external transfer in one window. CloudTrail events reach CloudWatch Logs 5–15 minutes after the API call, so correlate on eventTime rather than query time. Query a wider lookback than your correlation window: for example, look back 30 to 60 minutes but correlate on a 10-minute eventTime window. Steps 3 and 4 are manual validation; step 5 automates them.

Figure 2 shows DNS resolution as a third corroborating signal. This walkthrough implements the CloudTrail and VPC Flow Logs correlation. To add DNS, apply the same run_query() pattern against your Route 53 Resolver query log group.

Step 5: Automate the check

Move the query into a Lambda function (serverless compute that runs your code without a server to manage), send results to a notification channel, and schedule regular runs. Work through the following sub-procedures.

To create the notification channel

Open the Amazon Simple Notification Service (Amazon SNS) console. Amazon SNS is a managed messaging service that delivers notifications to subscribers.
In the navigation pane, choose Topics.
Choose Create topic.
For Type, select Standard.
For Name, enter security-correlation-alerts.
Choose Create topic.
Note the topic Amazon Resource Name (ARN) at the top of the topic details page. You will use it in the function.
Choose Create subscription.
For Protocol, select Email.
For Endpoint, enter your email address or incident management endpoint.
Choose Create subscription, then confirm the subscription from the email AWS sends.

To create the EventBridge Scheduler execution role

The schedule needs a role that lets it invoke your function, and its trust policy needs conditions that pin the role to the schedule you own. Without those conditions, another account with access to the scheduler service could theoretically call this role; a class of misuse known as the confused deputy problem.

1. Create a trust policy file named scheduler-trust-policy.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "scheduler.amazonaws.com" },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "ACCOUNT-ID"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws:scheduler:REGION:ACCOUNT-ID:schedule/*/s3-access-correlation-hourly"
        }
      }
    }
  ]
}

2. Create the role, then attach permission to invoke the function. Scope Resource to the specific function ARN so this role can’t invoke anything else.

aws iam create-role \
  --role-name EventBridgeSchedulerRole \
  --assume-role-policy-document file://scheduler-trust-policy.json

aws iam put-role-policy \
  --role-name EventBridgeSchedulerRole \
  --policy-name LambdaInvokePolicy \
  --policy-document '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Effect": "Allow",
        "Action": "lambda:InvokeFunction",
        "Resource": "arn:aws:lambda:REGION:ACCOUNT-ID:function:CorrelationFunction"
      }
    ]
  }'

When you create the function, Lambda automatically creates an execution role. You will attach the permissions this function needs to that role in a later step.

To deploy the correlation function

Open the Lambda console.
Choose Create function.
For Function name, enter CorrelationFunction.
For Runtime, select the latest Python runtime.
Choose Create function.
On the Code tab, replace the default code with the following function, then choose Deploy.

import os
import time
import logging
import boto3
from botocore.exceptions import ClientError

logger = logging.getLogger()
logger.setLevel(logging.INFO)

logs = boto3.client("logs")
sns = boto3.client("sns")
ec2 = boto3.client("ec2")

CLOUDTRAIL_LOG_GROUP = os.environ["CLOUDTRAIL_LOG_GROUP"]
FLOWLOGS_LOG_GROUP = os.environ["FLOWLOGS_LOG_GROUP"]
SNS_TOPIC = os.environ["SNS_TOPIC_ARN"]
BUCKET = os.environ["SENSITIVE_BUCKET"]
THRESHOLD = int(os.environ.get("THRESHOLD_READS", "100"))

# Expected readers per bucket
EXPECTED_READERS = {
    "customer-records-prod": [
        "arn:aws:iam::123456789012:role/AnalyticsPipeline",
        "arn:aws:iam::123456789012:role/ComplianceAudit",
    ],
}


def run_query(log_group, query, start, end):
    """Start a Logs Insights query and wait for it to finish."""
    started = logs.start_query(
        logGroupName=log_group,
        startTime=start,
        endTime=end,
        queryString=query,
    )
    query_id = started["queryId"]
    while True:
        outcome = logs.get_query_results(queryId=query_id)
        if outcome["status"] in ("Complete", "Failed", "Cancelled"):
            break
        time.sleep(1)
    if outcome["status"] != "Complete":
        raise RuntimeError(f"Query did not complete: {outcome['status']}")
    return [{f["field"]: f["value"] for f in row} for row in outcome["results"]]


def private_ip_for_principal(principal_id):
    """Resolve an EC2 instance role principalId to its private IP."""
    if ":" not in principal_id:
        return None
    instance_id = principal_id.split(":", 1)[1]
    if not instance_id.startswith("i-"):
        return None
    reservations = ec2.describe_instances(InstanceIds=[instance_id])
    for reservation in reservations["Reservations"]:
        for instance in reservation["Instances"]:
            return instance.get("PrivateIpAddress")
    return None


def egress_bytes(src_addr, start, end):
    """Sum external egress bytes for one source address."""
    query = f"""
    fields srcAddr, dstAddr, bytes
    | filter action = "ACCEPT" and srcAddr = "{src_addr}"
    | filter dstAddr not like /^10\\./
            and dstAddr not like /^192\\.168\\./
            and dstAddr not like /^172\\.(1[6-9]|2[0-9]|3[0-1])\\./
    | stats sum(bytes) as totalBytes
    """
    rows = run_query(FLOWLOGS_LOG_GROUP, query, start, end)
    if rows and rows[0].get("totalBytes"):
        return int(rows[0]["totalBytes"])
    return 0


def lambda_handler(event, context):
    try:
        # 1-hour lookback absorbs CloudTrail's 5-15 min delivery latency;
        # correlation happens on eventTime via 10-min bins in the query below.
        end = int(time.time())
        start = end - 3600  # 1 hour lookback
        allowed = EXPECTED_READERS.get(BUCKET, [])

        access_query = f"""
        fields userIdentity.arn, userIdentity.principalId
        | filter eventSource = "s3.amazonaws.com" and eventName = "GetObject"
        | filter requestParameters.bucketName = "{BUCKET}"
        | stats count(*) as objectReads
                by userIdentity.arn, userIdentity.principalId, bin(10m)
        | filter objectReads > {THRESHOLD}
        """

        for row in run_query(CLOUDTRAIL_LOG_GROUP, access_query, start, end):
            principal = row.get("userIdentity.arn")
            if not principal or principal in allowed:
                continue

            message = (
                f"Principal {principal} read {row.get('objectReads')} "
                f"objects from {BUCKET}."
            )

            ip = private_ip_for_principal(row.get("userIdentity.principalId", ""))
            if ip and egress_bytes(ip, start, end) > 1_000_000_000:
                message += (
                    f" The same source ({ip}) also sent a large volume of "
                    f"data to external destinations in the same window."
                )

            sns.publish(
                TopicArn=SNS_TOPIC,
                Subject="Unexpected S3 access detected",
                Message=message,
            )
    except ClientError as error:
        logger.error(f"AWS API error: {error}")
        raise
    except Exception as error:
        logger.error(f"Unexpected error: {error}")
        raise
    finally:
        logger.info("Correlation check completed")

On the Configuration tab, choose General configuration, then choose Edit. Set Timeout to 5 minutes (300 seconds). CloudWatch Logs Insights queries run asynchronously and can take 30 to 60 seconds against large log groups. Choose Save.
On the Configuration tab, choose Environment variables, then choose Edit, and add CLOUDTRAIL_LOG_GROUP, FLOWLOGS_LOG_GROUP, SNS_TOPIC_ARN, SENSITIVE_BUCKET, and THRESHOLD_READS.
On the Configuration tab, choose Permissions, open the execution role, and attach the following least-privilege policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["logs:StartQuery", "logs:GetQueryResults"],
      "Resource": [
        "arn:aws:logs:REGION:ACCOUNT-ID:log-group:aws-cloudtrail-logs-my-trail:*",
        "arn:aws:logs:REGION:ACCOUNT-ID:log-group:vpc-flow-logs:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "ec2:DescribeInstances",
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:REGION:ACCOUNT-ID:security-correlation-alerts"
    }
  ]
}

Replace REGION, ACCOUNT-ID, and the log-group names with your values. The ec2:DescribeInstances action doesn’t support resource-level permissions, so Resource: "*" is required for that statement; the other statements are scoped to specific ARNs.

To schedule automated runs

Amazon EventBridge (a serverless event bus that connects applications using events) runs targets on a schedule. Create one from the command line, using the role you made earlier.

aws scheduler create-schedule \
  --name s3-access-correlation-hourly \
  --schedule-expression "rate(1 hour)" \
  --target "Arn=arn:aws:lambda:REGION:ACCOUNT-ID:function:CorrelationFunction,RoleArn=arn:aws:iam::ACCOUNT-ID:role/EventBridgeSchedulerRole" \
  --flexible-time-window "Mode=OFF"

Step 6: Add enrichment context (optional)

Enrichment cuts triage time by adding an independent signal, but it isn’t required for the correlation to work. This step adds costs. You pay your geolocation provider for API calls, and the additional Lambda execution time increases your Lambda charges. To add IP geolocation, sign up for a geolocation API, add this function to the code, and call it where the handler resolves an IP.

import urllib.request
import json

def geo_context(ip_address):
    """Enrich an IP address with geolocation data from your provider."""
    try:
        url = f"https://your-geolocation-api.example/json/{ip_address}"
        with urllib.request.urlopen(url, timeout=5) as response:
            data = json.load(response)
        return {
            "country": data.get("country_name"),
            "city": data.get("city"),
            "org": data.get("org"),
        }
    except Exception as error:
        logger.warning(f"Geolocation lookup failed for {ip_address}: {error}")
        return None

Inside the handler’s loop, after you resolve ip, append the location to the alert.

            if ip:
                geo = geo_context(ip)
                if geo:
                    message += (
                        f" Source location: {geo['city']}, "
                        f"{geo['country']} ({geo['org']})."
                    )

Step 7: Scale to additional patterns and accounts

As your library grows, move the logic into automated pipelines with EventBridge, Lambda, and AWS Step Functions (a serverless orchestration service that coordinates multiple services into workflows), and surface correlations next to findings in Security Hub. For cross-service correlation at scale, CloudWatch unified data and telemetry capabilities can convert security and compliance data into the OCSF format and let you query sources such as CloudTrail, VPC Flow Logs, and DNS logs from one interface. Security Lake with Athena is a strong option for long-term analysis. Choose the endpoint that fits your retention and query needs.

Figure 3 shows a correlation pipeline built on AWS services including EventBridge, Lambda, Step Functions, and AWS Security Hub. The pipeline runs from data sources through scheduled queries and enrichment to automated response and centralized visibility.

Figure 3: A correlation pipeline built on AWS services

Conclusion

You now have four correlation patterns that layer your business context on top of GuardDuty Extended Threat Detection to catch attacks specific to your environment. A few principles carry across every correlation you build.

Identity is your primary correlation key: Track the same principal across services.
Time windows matter, but they depend on the attack: Events minutes apart are usually related for fast, automated sequences; the ten-minute bins here work for that pattern. Slow or manual reconnaissance can stretch across hours or days, so widen the window when the pattern is deliberate rather than automated.
Context is what you add: Your data classification, access norms, resource ownership, and change windows are signals you bring to detection.
Start with one rule: A single well-tuned correlation catches more significant activity than a wall of uncorrelated alerts.

GuardDuty Extended Threat Detection handles the multi-stage patterns common across customers. The correlations in this post add the layer that only your business context can supply. Start with one pattern this week, validate it against your own traffic, and add the next pattern after the first proves reliable.

Have you built correlation rules for patterns not covered here? Share your experience in the Comments section below.

AI-driven software delivery with Kiro, AWS DevOps Agent and Bluebox by Dynatrace

2026-08-26 Philipp Ushiromiya

Post Syndicated from Philipp Ushiromiya original https://aws.amazon.com/blogs/devops/ai-driven-software-delivery-with-kiro-aws-devops-agent-and-bluebox-by-dynatrace/

This post was co-written with Michael Stephan, Senior Principal Product Manager, and Christian Kreuzberger, Principal Software Engineer, at Dynatrace.

AI-driven software delivery changes how code gets written, but not what production demands of it. A generated change still has to fit the traffic your service receives, the dependencies it calls, and the capacity limits it runs within. Without that context, you validate the change after it ships, which adds rework and deployment risk.

Kiro turns intent into specifications, code, and pull requests. AWS DevOps Agent investigates incidents and proposes mitigations. Bluebox by Dynatrace supplies the runtime topology, dependency, and traffic data that both draw on, so each change and each investigation is grounded in how the system behaves rather than how it’s expected to behave. In this post, we will follow a travel-booking example from feature design through post-deployment remediation. You’ll see how telemetry from Bluebox shapes a change in Kiro, how AWS DevOps Agent investigates an incident, and where human review and existing CI/CD controls remain in the process.

What are Kiro and AWS DevOps Agent?

Kiro is an agentic development environment that applies AI across the software development lifecycle. Its spec-driven workflow organizes a feature request into requirements, design, and implementation tasks before generating any code.

AWS DevOps Agent is a frontier agent for software delivery and operations across AWS, multicloud, and on-premises environments. It investigates incidents, identifies likely root causes, and recommends mitigations. Its release management capability (Preview) reviews code for release readiness and runs release tests before deployment.

Bluebox by Dynatrace: Helps agents ship the code you trust to production

To close the loop between code generation and production context, Kiro and AWS DevOps Agent rely on real-time production intelligence. This is where Bluebox by Dynatrace fits in. Bluebox provides the observability foundation that detects problems, measures their impact, and surfaces the runtime application topology, service dependencies, and actual traffic patterns that make AI-generated code and autonomous investigations truly production-aware.

Without production telemetry, AI-generated code operates in a vacuum – it cannot know that an endpoint handles 40:1 read-to-write ratios, that a service dependency has specific latency characteristics, or how API traffic fluctuates throughout the day. Bluebox grounds actions taken by Kiro and AWS DevOps Agent in how the system actually behaves, not in assumptions about how it should behave.

How the closed loop works

The combination of Kiro, AWS DevOps Agent, and Bluebox creates a continuous cycle from development through production and back:

Production-aware code generation: Before code is written, Kiro retrieves runtime context from Bluebox – service topology, traffic patterns, and resource utilization. Kiro’s spec-driven workflow translates this context into requirements and generates code that aligns with real production conditions from the first commit.
Confident code review: Kiro generates pull requests with production evidence attached. The release management capability in AWS DevOps Agent reviews the change for dependency impacts, drifts from internal standards, and production readiness – running autonomous tests in isolated environments.
Continuous monitoring: After deployment, Dynatrace continuously monitors application behavior. When an anomaly occurs, Bluebox detects it and surfaces full production context.
Autonomous investigation: Bluebox triggers AWS DevOps Agent with the relevant observability and topology data. AWS DevOps Agent performs a deep investigation, correlating telemetry, logs, infrastructure changes, and deployment history to pinpoint the root cause.
Automated remediation: AWS DevOps Agent generates the mitigation plan from the observability and runtime data that Bluebox provides. Bluebox adds that plan to the investigation report and files it as a GitHub issue. Kiro then proposes a production-aware fix as a pull request for your review, completing the loop.

Figure 1: Bluebox supports the closed loop from feature build to operations.

Next, we walk through a concrete example of this workflow in action.

Walkthrough

We follow a travel-booking application through two connected scenarios: shipping a new feature with production context, then responding to a production incident after it deploys.

Building a production-aware feature

Consider a team enhancing a travel booking application to improve customer experience. You begin by describing a new feature in Kiro, such as updating how products are displayed or adjusting backend logic to support new capabilities. In this case, we are using Kiro IDE.

Figure 2. A feature request in Kiro, with the project’s steering documents loaded for context.

Kiro’s spec-driven workflow expands this request into structured requirements before writing code. You connect Kiro to the Bluebox CLI to retrieve the full production context from Dynatrace: service dependencies, runtime topology, and observed traffic. The following figure shows how Kiro queries current load data for the flight-search path, including the ratio of Amazon DynamoDB reads to writes. Kiro composes and runs the CLI command on your behalf, so you don’t have to type it or set environment variables by hand. The command and its output stay visible in the session, so you can approve it before it runs and check what was retrieved before acting on it. In this case, the command queries the Bluebox API for the requested metrics. The output returns read and write counts per second for the DynamoDB table behind flight search, along with the services calling it.

Figure 3. Kiro runs the Bluebox CLI, then reads the codebase with production context before proposing changes.

The telemetry shows the flight-search endpoint is read-heavy. Users repeatedly query the same routes, at roughly 40 reads for every write against the DynamoDB table. Repeated identical reads are what a cache absorbs, so Kiro proposes an Amazon ElastiCache layer in front of the table, sized to the active working set derived from the observed request distribution. Without the read-to-write ratio, the same request could have produced a larger provisioned table or an added read replica, neither of which addresses repeated identical queries.

Kiro generates the code that implements the change and opens a pull request in GitHub for review. Nothing reaches production until a reviewer approves and merges it. The pull request carries the code changes and the Bluebox telemetry that justified them, so reviewers assess the decision against the same telemetry Kiro retrieved.

Figure 4. Kiro pushes a feature branch and opens a pull request in GitHub.

After review and approval through standard processes, a reviewer merges the pull request, and the existing CI/CD pipeline deploys the change.

Figure 5. The pull request is reviewed and merged through the standard GitHub workflow.

Responding to a production incident

With the feature live, Dynatrace continues monitoring the application. A marketing promotion then drives traffic above the observed baseline, and failed requests start to appear. The loop now runs from operations back to development.

Figure 6. Dynatrace detects a spike in failed requests, surfacing the production incident.

Bluebox collects the relevant observability and topology data, runs an initial root-cause analysis, then opens an autonomous investigation in AWS DevOps Agent. The AWS DevOps Agent multi-agent reasoning architecture decomposes the investigation across specialized capabilities that each examine one class of evidence: telemetry, logs, infrastructure configuration, and recent deployment activity.

Figure 7. Bluebox delegates an autonomous investigation to AWS DevOps Agent.

AWS DevOps Agent locates the cause in the DynamoDB table rather than the new cache. The table’s billing mode had been changed to PROVISIONED, with 5 read capacity units (RCU) and 5 write capacity units (WCU) and no auto scaling. The ElastiCache layer absorbs repeated reads, but cache misses and all writes still reach DynamoDB, and at promotion traffic that residual load exceeds 5 RCU and 5 WCU. AWS DevOps Agent produces a mitigation plan with specific remediation steps. This plan and the full investigation context from Bluebox, is documented as a GitHub issue.

Figure 8. GitHub issue is created with results from Bluebox and AWS DevOps Agent.

Kiro proposes a production-aware fix as a new pull request – including the root-cause analysis, supporting telemetry, and recommended configuration changes.

Figure 9. The Kiro coding session works on the GitHub issue and creates a remediation Pull Request.

The fix is reviewed, merged, and deployed like any other change. Dynatrace then confirms that error rates and response times return to baseline, which closes the loop.

Conclusion

In this post, we showed how Kiro, AWS DevOps Agent, and Bluebox by Dynatrace connect production telemetry with feature development and incident remediation. The travel-booking example keeps human review and existing CI/CD controls in the process while passing operational context from production back to development.

To get started pick one application and define a measurable outcome, such as investigation time, change-failure rate, or pull-request review time. Then:

Download Kiro and start building with spec-driven development
Enable AWS DevOps Agent for autonomous incident investigation and remediation
Get started with Bluebox by Dynatrace to complete the loop with production intelligence

Setting up an RCS agent with an AI coding assistant and AWS End User Messaging

2026-08-25 Bruno Giorgini

Post Syndicated from Bruno Giorgini original https://aws.amazon.com/blogs/messaging-and-targeting/setting-up-an-rcs-agent-with-an-ai-coding-assistant-and-aws-end-user-messaging/

Clone a repo, open it in your AI coding assistant, type “go,” and walk away with a working RCS agent.

Creating an RCS agent on AWS End User Messaging normally means juggling 23 registration fields, three different CLI parameter types, brand asset requirements, and a multi-step approval process. An AI coding assistant can handle all of that for you. With AWS End User Messaging, you can create RCS agents that send and receive rich messages complete with your brand’s logo, colors, and verified identity.

Setting up an RCS agent involves creating an agent container, uploading brand assets, configuring a 23-field registration, submitting for approval, adding verified testers, and testing both outbound and inbound messaging. Each field has a specific type (TEXT, SELECT, or ATTACHMENT) that requires a different CLI parameter, and getting any of them wrong means starting over.

We built an open-source sample repository that encodes all of this knowledge into an AGENTS.md file. When you open the repo in an AI coding assistant like Kiro, Cursor, or Windsurf, the assistant reads the instructions and walks you through the entire setup interactively. You provide a brand name and your phone number. The AI handles everything else.

How it works

The repository aws-samples/sample-rcs-agent-setup-and-send-messages contains:

AGENTS.md — A structured instruction file that AI coding assistants read automatically. It contains the complete RCS agent setup workflow: credential checks, brand asset generation, registration field configuration, tester management, and message testing.
brand-assets/ — Template SVG files for the agent logo (224×224 px) and banner (1440×448 px), ready to be customized and converted to PNG.
.kiro/steering/rcs-agent-setup.md — A Kiro-specific steering file with the same instructions, using the inclusion: always frontmatter so Kiro loads it automatically.

The AGENTS.md file is the key. It defines six skills that the AI assistant executes in sequence:

Create RCS agent — Creates the agent container, generates brand assets (logo and banner SVGs), converts them to PNG, creates a test registration, sets all 23 fields with the correct parameter types, and submits for approval.
Add verified testers — Registers test phone numbers and guides you through accepting the tester invitation.
Send a test message — Checks for blockers (protect configuration, opt-out lists) and sends your first branded RCS message.
Set up inbound keyword — Configures an automatic response keyword so you can test inbound messaging without writing backend code.
Verify inbound messaging — Walks you through the console deep link flow to confirm two-way messaging works.
Delete an RCS agent — Removes an agent cleanly by disabling deletion protection, deleting the associated registration, then deleting the agent itself.

Prerequisites

Before you start, you need:

An AWS account with access to AWS End User Messaging.
AWS Command Line Interface (AWS CLI) v2.35.12 or later installed and configured with credentials that have pinpoint-sms-voice-v2:* permissions.
An AI coding assistant that reads AGENTS.md files (Kiro, Cursor, Windsurf, or similar).
librsvg for SVG to PNG conversion (brew install librsvg on macOS).
A test phone that supports RCS messaging.

Getting started

Follow these steps to go from zero to a working RCS agent. The entire process takes about five minutes.

Step 1: Clone the repository

git clone https://github.com/aws-samples/sample-rcs-agent-setup-and-send-messages.git
cd sample-rcs-agent-setup-and-send-messages

Step 2: Open in your AI coding assistant

Open the cloned directory in your preferred AI coding assistant. The assistant will automatically detect the AGENTS.md file (or .kiro/steering/rcs-agent-setup.md if you are using Kiro).

Step 3: Type “go”

In the chat panel, type go. The AI assistant will:

Check your AWS credentials — It runs aws sts get-caller-identity and asks how you authenticate if credentials are not configured. It supports named profiles, SSO, IAM user credentials, and environment variables.
Verify EUM access — It confirms your account can use AWS End User Messaging.
Check tooling — It verifies rsvg-convert is installed for brand asset generation.
Ask for your preference — Quick mode (provide a brand name) or interactive mode (you specify every detail).

Step 4: Provide a brand name

In quick mode, you provide a brand name and the AI generates everything else: a description, an accessible accent color, contact information with placeholder values, privacy and terms URLs, and custom SVG brand assets with your brand name and colors.

In interactive mode, the AI asks for each detail one section at a time: brand name, accent color, logo description, banner description, contact information, and policy URLs.

Step 5: Watch it work

The AI assistant executes every AWS CLI command in sequence:

Creates the RCS agent container.
Enables deletion protection.
Creates a test registration and links it to the agent.
Generates and converts brand asset SVGs to PNG.
Uploads the logo and banner as registration attachments.
Sets all 23 registration fields using the correct parameter type for each (TEXT, SELECT, or ATTACHMENT).
Submits the registration and polls for approval.
Reports when the agent is active.

Step 6: Add a tester and send a message

Once the agent is approved, the AI asks for your test phone number, registers it as a verified tester, and waits for you to accept the invitation. After verification, it checks for blockers (protect configuration and opt-out lists), then sends your first branded RCS message.

Step 7: Test inbound messaging

The AI configures an automatic keyword response and walks you through the console deep link flow to verify two-way messaging. When you send RCSINBOUNDTESTING to your agent, you receive an automatic reply confirming inbound messaging works.

What the AI handles for you

The AGENTS.md file encodes several non-obvious behaviors that would otherwise require trial and error:

Challenge	How the repo handles it
`create-rcs-agent` takes no `--display-name` parameter	The brand name comes from the registration, not the agent creation call. The instructions reflect this.
Three different field parameter types	The instructions include a field reference table mapping each of the 23 fields to its correct CLI parameter: `--text-value`, `--select-choices`, or `--registration-attachment-id`.
`--field-values` does not exist	The instructions explicitly warn against this non-existent parameter and use the correct alternatives.
`--attachment-body` and `--attachment-url` conflict	The instructions use `--attachment-body` only.
Accent color contrast requirements	The instructions include pre-validated color choices with 4.5:1 contrast ratio against white.
Field paths differ from what you might expect	The correct paths are `agentDetails.logoImage` and `agentDetails.bannerImage`, not `logoAttachmentId` or `bannerAttachmentId`.
New registration versions do not inherit field values	The troubleshooting section warns that all 23 fields must be re-populated when creating a new version.

Customizing the repo

You can modify the AGENTS.md file to fit your workflow:

Change default values — Update placeholder contact information, privacy URLs, or terms URLs to match your organization.
Add custom brand assets — Replace the template SVGs in brand-assets/ with your own designs. Keep the logo at 224×224 px and the banner at 1440×448 px.
Extend the skills — Add new skills for richer message types (cards, carousels), event destinations for programmatic inbound handling, or integration with other AWS services.

Cleanup

To remove the resources created during testing:

# 1. Disable deletion protection
aws pinpoint-sms-voice-v2 update-rcs-agent \
  --rcs-agent-id <your-agent-id> \
  --no-deletion-protection-enabled \
  --region us-east-1

# 2. Delete the associated registration (required before deleting the agent)
aws pinpoint-sms-voice-v2 delete-registration \
  --registration-id <your-registration-id> \
  --region us-east-1

# 3. Delete the agent
aws pinpoint-sms-voice-v2 delete-rcs-agent \
  --rcs-agent-id <your-agent-id> \
  --region us-east-1

Note: You must delete the registration before the agent. Skipping this step results in a ConflictException: RESOURCE_NOT_EMPTY error.

Conclusion

The aws-samples/sample-rcs-agent-setup-and-send-messages repository turns a multi-step, error-prone CLI workflow into a guided conversation. Clone the repo, open it in your AI coding assistant, type “go,” and you have a working RCS agent that can send and receive branded messages to verified testers.

The AGENTS.md pattern is reusable. Any complex AWS workflow with non-obvious API behavior can be encoded the same way: document the correct commands, parameter types, and pitfalls in a structured file, and let the AI assistant execute it interactively.

For a detailed manual walkthrough of the same process, see Creating and testing an RCS agent with AWS End User Messaging. For an overview of the business case for RCS, see Upgrade business messaging with RCS on AWS. For more information, see the AWS End User Messaging service page and the RCS documentation.

About the author

Enable cross-cloud analytics with Amazon S3 Tables and Google BigQuery, Part 1: IAM-based access control

2026-08-25 Lakshmi Nair

Post Syndicated from Lakshmi Nair original https://aws.amazon.com/blogs/big-data/enable-cross-cloud-analytics-with-amazon-s3-tables-and-google-bigquery-part-1-iam-based-access-control/

Organizations running analytics workloads across multiple clouds often hit the same friction: the data lives on one cloud, but the engine querying it lives on another. Copying data across the boundary creates a second dataset that must be kept in sync, adding cost, latency, and reconciliation overhead. In this post, we address a specific instance of that pattern: your Google BigQuery users need to work with data that lives in Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), on AWS. The ideal outcome is a single, governed dataset that serves teams in both clouds without a standing replication pipeline between them.

With Amazon S3 Tables, you get managed Apache Iceberg tables with built-in compaction, snapshot management, and an integration with the AWS Glue Data Catalog. Because S3 Tables stores data in the open Iceberg format, supported external engines can read it directly if the right access path exists.

This two-part blog series demonstrates how you can connect Google BigQuery to Amazon S3 Tables using the cross-cloud lakehouse with AWS Glue. We cover two access control approaches:

AWS Identity and Access Management (IAM): You can define a single policy that uses IAM permissions to set up access to both table metadata and data.
AWS Lake Formation: You can use temporary vended credentials for data access, with metadata access managed by Lake Formation permissions.

This post focuses on the IAM-based approach. Part 2 covers the Lake Formation approach for organizations that need credential-vended access across multiple engines.

By the end, you will have BigQuery querying Iceberg tables stored on S3 Tables without data copy or duplication, providing live access to Iceberg data.

Cross-cloud analytics scenarios

There are several scenarios where organizations benefit from cross-cloud querying capabilities. Here are some of the common patterns this architecture addresses:

Schema evolution across cloud boundaries

When source schemas change frequently, streaming pipelines writing to BigQuery-managed store require coordinated DDL changes on the BigQuery table and downstream views. Teams often work around this challenge by storing payloads as untyped columns and parsing them later.

With Iceberg on S3 Tables, schema evolution is tracked in table metadata. When the writing engine adds a new column, BigQuery’s Lakehouse refresh picks up the updated schema automatically on the next sync cycle.

Multi-cloud analytics without data duplication

A company has its production data environment on AWS (data lakes, warehouses, streaming) but acquired a business unit that runs analytics exclusively on BigQuery. In-place querying from BigQuery keeps your data in Amazon S3 Tables, so you pay for one copy, work from live data, and avoid the operational overhead of a synchronized second store.

Cost optimization for infrequently queried datasets

An organization has hundreds of datasets on AWS, but only a fraction is queried daily from BigQuery. Replicating all of them to Google Cloud Storage drives unnecessary storage and transfer costs. With Lakehouse catalog federation, you keep your data on S3 Tables. BigQuery reads data only when queried, so you pay per query rather than per-copy storage.

Decoupled compute across engines

Data team wants storage on AWS with the flexibility for multiple engines to read the same data: BigQuery and Amazon Redshift for data warehousing use cases, Amazon Athena for interactive ad-hoc querying, Amazon SageMaker AI for machine learning (ML). With Apache Iceberg’s open format, you can use one storage layer, many compute engines, no data copies between them.

Solution overview

You use the AWS Glue Iceberg REST Catalog (IRC) as the bridge between BigQuery and S3 Tables. BigQuery’s cross-cloud Lakehouse creates a federated catalog that syncs metadata from the Glue IRC, then uses the synced metadata to read Iceberg data files directly.

Figure 1: Architecture diagram showing BigQuery connecting to Amazon S3 Tables through the AWS Glue Iceberg REST Catalog

The key components in this architecture:

Amazon S3 Tables: With Amazon S3 Tables, you get a fully managed Apache Iceberg table experience in Amazon S3, optimized for analytics workloads. You can register table metadata in the AWS Glue Data Catalog for discovery and governance.
AWS Glue Data Catalog: With AWS Glue Data Catalog, you can access the federated s3tablescatalog catalog that maps S3 Tables resources (table buckets, namespaces, tables) into a catalog hierarchy from supported analytics engines. The standard Iceberg REST endpoint of Glue Data Catalog serves table metadata to external engines. BigQuery connects through this endpoint.
Google Cross-Cloud Lakehouse: With Google Cross-Cloud Lakehouse, you can connect BigQuery to external Iceberg catalogs. It assumes an AWS IAM role using OpenID Connect (OIDC), calls the Glue Iceberg REST endpoint, and syncs metadata on a configurable refresh interval.

Prerequisites

Before you begin, you need:

An AWS account with Amazon S3 Tables available in your AWS Region.
A Google Cloud project with billing enabled and the BigLake API activated.
AWS Command Line Interface (AWS CLI) and gcloud CLI installed and configured.
An S3 table bucket with at least one namespace and table containing data.

Setting up Amazon S3 Tables

If you already have S3 Tables with data, skip to the next section. Otherwise, create a table bucket, namespace, and populate a table.

Create a table bucket and namespace

Use the AWS CLI to create resources as follows:

# Create a Table bucket
aws s3tables create-table-bucket \
    --name <TABLE_BUCKET_NAME> \
    --region <REGION>

# Create a Namespace (Database)
aws s3tables create-namespace \
    --table-bucket-arn "arn:aws:s3tables:<REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET_NAME>" \
    --namespace <NAMESPACE> \
    --region <REGION>

Integrating S3 Tables with the Glue Data Catalog

For BigQuery to access S3 Tables, the tables must be discoverable through the Glue Data Catalog. S3 Tables integrates with Glue through a federated catalog called s3tablescatalog.

Set up S3 Tables integration with the Glue Data Catalog using IAM mode

Open the Amazon S3 console:

In the navigation pane, choose Table buckets.
Choose Enable integration, and then choose Enable integration again to confirm.

This creates the s3tablescatalog federated catalog in Glue, where access is controlled entirely by IAM policies on the calling role. This is a one-time setup per account and Region. After you enable it, the analytics integration applies to all table buckets in your account.

The Enable integration option on the table buckets page of the Amazon S3 console

Figure 2: Enabling the S3 Tables integration in the Amazon S3 console

Alternatively, create the catalog using the AWS CLI:

aws glue create-catalog --region <REGION> --cli-input-json '{
  "Name": "s3tablescatalog",
  "CatalogInput": {
    "FederatedCatalog": {
      "Identifier": "arn:aws:s3tables:<REGION>:<AWS_ACCOUNT_ID>:bucket/*",
      "ConnectionName": "aws:s3tables"
    },
    "CreateDatabaseDefaultPermissions": [
      { "Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"] }
    ],
    "CreateTableDefaultPermissions": [
      { "Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"] }
    ]
  }
}'

Create a table and insert data

Now, to create the table and insert data, open the Amazon Athena console. In the query editor, select s3tablescatalog/<TABLE_BUCKET_NAME> as your data source and <NAMESPACE> as the database. Then run the following SQL statements one by one:

CREATE TABLE `<NAMESPACE>`.orders (
    order_id STRING,
    customer_id STRING,
    amount BIGINT,
    order_date DATE,
    region STRING
)
TBLPROPERTIES ('table_type' = 'iceberg');

INSERT INTO orders
VALUES
    ('ORD-001', 'C100', 4500, DATE '2024-06-01', 'EMEA'),
    ('ORD-002', 'C200', 8900, DATE '2024-06-01', 'EMEA'),
    ('ORD-003', 'C100', 3200, DATE '2024-06-02', 'NAMER'),
    ('ORD-004', 'C300', 12000, DATE '2024-06-02', 'NAMER'),
    ('ORD-005', 'C400', 6700, DATE '2024-06-03', 'APJ'),
    ('ORD-006', 'C200', 4100, DATE '2024-06-03', 'APJ'),
    ('ORD-007', 'C500', 9500, DATE '2024-06-04', 'EMEA'),
    ('ORD-008', 'C100', 2800, DATE '2024-06-04', 'LATAM'),
    ('ORD-009', 'C600', 15000, DATE '2024-06-05', 'NAMER'),
    ('ORD-010', 'C300', 7200, DATE '2024-06-05', 'LATAM');

Configuring cross-cloud access

BigQuery assumes an AWS IAM role via OIDC federation to access the Glue IRC. This section walks through creating the role, OIDC provider, and permissions.

Create the OIDC identity provider

Register Google as an OIDC identity provider in your AWS account. This allows AWS to validate tokens issued by Google’s identity service:

aws iam create-open-id-connect-provider \
    --url https://accounts.google.com \
    --client-id-list accounts.google.com \
    --thumbprint-list 08745487e891c19e3078c1f2a07e452950ef36f6

The –thumbprint-list parameter is optional. When omitted, IAM automatically retrieves the thumbprint from the OIDC provider’s certificate. See AWS documentation for details.

Create the cross-cloud IAM role

Login into AWS Console, and create the role with a placeholder trust policy. You will update it with the actual BigLake service account ID after you create the federated catalog in Google Cloud.

aws iam create-role \
    --role-name bigquery-cross-cloud-role \
    --max-session-duration 43200 \
    --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["PLACEHOLDER"],
            "accounts.google.com:aud": ["PLACEHOLDER"]
          }
        }
      }]
    }'

The --max-session-duration 43200 allows sessions up to 12 hours, which is needed for long-running BigQuery queries.

Attach permissions

The permissions policy differs based on your access control approach. For the IAM-based approach, attach the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueRead",
      "Effect": "Allow",
      "Action": [
        "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases",
        "glue:GetTable", "glue:GetTables", "glue:GetPartition", "glue:GetPartitions"
      ],
      "Resource": [
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog/<TABLE_BUCKET>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:database/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>/*"
      ]
    },
    {
      "Sid": "S3TablesRead",
      "Effect": "Allow",
      "Action": [
        "s3tables:GetTableBucket", "s3tables:ListTableBuckets",
        "s3tables:ListNamespaces", "s3tables:GetNamespace",
        "s3tables:ListTables", "s3tables:GetTable",
        "s3tables:GetTableMetadataLocation", "s3tables:GetTableData"
      ],
      "Resource": [
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>",
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>/*"
      ]
    },
    {
      "Sid": "S3TablesListBuckets",
      "Effect": "Allow",
      "Action": ["s3tables:ListTableBuckets"],
      "Resource": "*"
    }
  ]
}

Connecting BigQuery to S3 Tables

With the AWS side configured, create the federated catalog in Google Cloud that connects BigQuery to the Glue IRC.

Create the federated catalog

Authenticate to Google Cloud using gcloud auth login, or use Cloud Shell, which is pre-authenticated. Verify that the BigLake API is enabled:

gcloud services enable biglake.googleapis.com --project="<GCP_PROJECT_ID>"

For IAM mode:

gcloud alpha biglake iceberg catalogs create <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --catalog-type=federated \
    --federated-catalog-type=glue \
    --glue-aws-region=<AWS_REGION> \
    --glue-aws-role-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role \
    --glue-warehouse=<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET> \
    --primary-location=<GCP_REGION>

The --glue-warehouse parameter uses the format <AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>. This tells the Glue IRC to scope requests to your specific S3 Tables bucket within the federated catalog hierarchy.

The --primary-location refers to the Google Cloud region where the federated catalog metadata is stored. Use the AWS to Google Cloud region mapping to find the corresponding GCP region for your AWS Region. For example, AWS us-east-1 maps to GCP us-east4.

Retrieve the BigLake service account ID

After catalog creation, Google provisions a dedicated service account for your federated catalog. Retrieve its numeric ID:

BIGLAKE_SA_ID=$(gcloud alpha biglake iceberg catalogs describe <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --format="value(biglake-service-account-id)")
echo $BIGLAKE_SA_ID

Update the AWS trust policy

Back on AWS, replace the placeholder in the IAM role’s trust policy with the actual service account ID:

aws iam update-assume-role-policy \
    --role-name bigquery-cross-cloud-role \
    --policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["<BIGLAKE_SA_ID>"],
            "accounts.google.com:aud": ["<BIGLAKE_SA_ID>"]
          }
        }
      }]
    }'

Register the service account ID in the OIDC provider’s audience list. Without this step, AWS rejects the token because the aud claim doesn’t match any registered client:

aws iam add-client-id-to-open-id-connect-provider \
    --open-id-connect-provider-arn "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com" \
    --client-id "<BIGLAKE_SA_ID>"

Set up metadata sync

Wait 3–5 minutes for IAM changes to propagate globally, then set up background refresh:

gcloud alpha biglake iceberg catalogs update <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --refresh-interval=300s

The --refresh-interval (300 seconds in this example) determines how often BigQuery syncs metadata from the Glue IRC. New tables and schema changes appear in BigQuery within this interval.

Querying from BigQuery

After the catalog refresh completes, BigQuery automatically creates external datasets corresponding to the synced namespaces. No manual CREATE SCHEMA is required.

Verify the sync:

gcloud alpha biglake iceberg namespaces list \
    --catalog="<FEDERATED_CATALOG_NAME>" \
    --project="<GCP_PROJECT_ID>"

Run a query in BigQuery:

SELECT * FROM `<GCP_PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders` LIMIT 1000

Sample Query Output:

SELECT
    customer_id,
    COUNT(*) as order_count,
    SUM(amount) as total_spend
FROM `<PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders`
GROUP BY customer_id
ORDER BY total_spend DESC

BigQuery query results showing order count and total spend per customer from the Amazon S3 Tables data

Figure 3: BigQuery query results returned directly from the Amazon S3 Tables data

BigQuery reads the Iceberg metadata to identify which Parquet data files contain relevant data. It also applies partition pruning where applicable, and fetches only the necessary files from S3 Tables managed storage.

Schema evolution

When new columns are added to an Iceberg table on the AWS side (through Spark, Athena, or the Glue IRC), the schema change is captured in Iceberg’s metadata. On the next Lakehouse refresh cycle, BigQuery picks up the new columns automatically. No DDL changes are needed in BigQuery.

Metadata freshness

The s3tablescatalog in Glue is a federated catalog that resolves table metadata live from the S3 Tables service on each request. When a streaming job commits new data to an S3 Table, the latest metadata is immediately available through the AWS Glue IRC. BigQuery sees the update on its next refresh cycle (as configured by --refresh-interval).

OIDC identity federation

The trust relationship between Google Cloud and AWS uses OpenID Connect. When BigQuery Lakehouse needs to access your data, it presents a signed JWT token containing:

iss: accounts.google.com (the issuer)
sub: The BigLake service account ID (identifies which catalog is making the request)
aud: The same service account ID (the intended audience)

AWS validates this token against the registered OIDC provider and trust policy conditions before issuing temporary credentials. Each federated catalog receives a unique service account ID, providing per-catalog isolation and auditability through AWS CloudTrail.

Network path

By default, traffic between BigQuery and AWS travels over the public internet. For workloads requiring private connectivity, Google Cloud supports Cross-Cloud Interconnect or Partner Interconnect. This helps routing queries over a dedicated network path. Refer to the Google Cloud documentation for private interconnect configuration.

Clean up

To avoid ongoing charges, remove the resources created in this walkthrough.

On AWS:

# Delete the table (if created for this walkthrough)
aws s3tables delete-table \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace analytics --name orders --region <AWS_REGION>

# Delete namespace and table bucket
aws s3tables delete-namespace \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace <NAMESPACE> --region <AWS_REGION>

aws s3tables delete-table-bucket --name <TABLE_BUCKET> --region <AWS_REGION>

# Delete IAM role and OIDC provider (if no longer needed)
aws iam delete-role --role-name bigquery-cross-cloud-role

On Google Cloud:

gcloud alpha biglake iceberg catalogs delete <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" --location=<GCP_REGION>

Conclusion

This post demonstrated how to query Amazon S3 Tables from Google BigQuery using the open Apache Iceberg format and the AWS Glue Iceberg REST Catalog as the metadata bridge. Using Apache Iceberg’s open format, you can write data once on AWS and read it from supported engines that speak Iceberg, including BigQuery. We used IAM-based access control to govern access to both Glue Data Catalog metadata and the underlying Amazon S3 Tables data. This is the simpler configuration path with fewer components. In Part 2, we walk through configuring AWS Lake Formation to vend temporary, scoped credentials to BigQuery for data access.

To get started with this pattern in your environment:

Review the Amazon S3 Tables documentation for table bucket setup.
Follow the Google Cloud cross-cloud Lakehouse setup guide for the BigQuery federation configuration.
Explore the AWS Glue Iceberg REST Catalog APIs for programmatic access patterns.

About the authors

Enable cross-cloud analytics with Amazon S3 Tables and Google BigQuery, Part 2: access control with Lake Formation

2026-08-25 Lakshmi Nair

Post Syndicated from Lakshmi Nair original https://aws.amazon.com/blogs/big-data/enable-cross-cloud-analytics-with-amazon-s3-tables-and-google-bigquery-part-2-access-control-with-lake-formation/

In Part 1, we showed how to connect Google BigQuery to Amazon Simple Storage Service (Amazon S3) Tables, a capability of Amazon S3, using access control based on AWS Identity and Access Management (IAM). A single IAM policy governs both table metadata and data access. We also walked through common cross-cloud analytics scenarios where this pattern adds value. This post covers the approach using AWS Lake Formation. Instead of relying solely on IAM policies for data access, Lake Formation manages fine-grained permissions and vends temporary, scoped credentials to the requesting engine. This is a better fit when multiple engines need different levels of access to the same tables, or when you want to manage grants centrally without touching IAM policies every time a new consumer comes along.

Solution overview

Figure 1: Architecture diagram showing BigQuery connecting to Amazon S3 Tables through the AWS Glue Iceberg REST Catalog

The key components in this architecture:

Amazon S3 Tables: With Amazon S3 Tables, data is stored in table buckets, specifically designed for storing tables in the Apache Iceberg format. Table metadata is registered on AWS Glue Data Catalog for discovery and governance.
AWS Glue Data Catalog: With AWS Glue Data Catalog, you can access the federated s3tablescatalog catalog that maps S3 Tables resources (table buckets, namespaces, tables) into a catalog hierarchy from supported analytics engines. The standard Iceberg REST endpoint of Glue Data Catalog serves table metadata to external engines. BigQuery connects through this endpoint.
AWS Lake Formation: With AWS Lake Formation, you define access permissions at the catalog, database, and table level. Instead of granting broad IAM permissions for data access, Lake Formation evaluates permissions at query time and issues short-lived credentials limited to the resources the caller is authorized to read.
Google Cross-Cloud Lakehouse: With Google Cross-Cloud Lakehouse, you can connect BigQuery to external Iceberg catalogs. It assumes an IAM role using OpenID Connect (OIDC), calls the AWS Glue Iceberg REST endpoint, and syncs metadata on a configurable refresh interval.

Prerequisites

Before you begin, you need:

An AWS account with Amazon S3 Tables available in your AWS Region.
A Google Cloud project with billing enabled and the BigLake API activated.
AWS Command Line Interface (AWS CLI) and gcloud CLI installed and configured.
An S3 table bucket with at least one namespace and table containing data.

Setting up Amazon S3 Tables

If you already have S3 Tables with data, skip to the next section. Otherwise, create a table bucket, namespace, and populate a table.

Create a table bucket and namespace

Use AWS CLI to create resources as follows:

# Create a Table bucket
aws s3tables create-table-bucket \
    --name <TABLE_BUCKET_NAME> \
    --region <REGION>

# Create a Namespace (Database)
aws s3tables create-namespace \
    --table-bucket-arn "arn:aws:s3tables:<REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET_NAME>" \
    --namespace <NAMESPACE> \
    --region <REGION>

Set up S3 Tables integration with the Glue Data Catalog using Lake Formation mode

Lake Formation needs its own service role to interact with S3 Tables on your behalf. This is the role Lake Formation assumes internally when it reads or writes data on behalf of authorized callers.

Create a Lake Formation service IAM role named LakeFormationS3TablesServiceRole with the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LakeFormationPermissionsForS3ListTableBucket",
      "Effect": "Allow",
      "Action": ["s3tables:ListTableBuckets"],
      "Resource": ["*"]
    },
    {
      "Sid": "LakeFormationDataAccessPermissionsForS3TableBucket",
      "Effect": "Allow",
      "Action": [
        "s3tables:CreateTableBucket", "s3tables:GetTableBucket",
        "s3tables:CreateNamespace", "s3tables:GetNamespace",
        "s3tables:ListNamespaces", "s3tables:DeleteNamespace",
        "s3tables:DeleteTableBucket", "s3tables:CreateTable",
        "s3tables:DeleteTable", "s3tables:GetTable",
        "s3tables:ListTables", "s3tables:RenameTable",
        "s3tables:UpdateTableMetadataLocation", "s3tables:GetTableMetadataLocation",
        "s3tables:GetTableData", "s3tables:PutTableData"
      ],
      "Resource": ["arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/*"]
    }
  ]
}

Attach the following trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LakeFormationDataAccessPolicy",
      "Effect": "Allow",
      "Principal": { "Service": "lakeformation.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:SetContext", "sts:SetSourceIdentity"],
      "Condition": { "StringEquals": { "aws:SourceAccount": "<AWS_ACCOUNT_ID>" } }
    }
  ]
}

In the Lake Formation console, in the navigation pane, choose Catalogs, and then choose Enable S3 Table Integration.

The Enable S3 Table Integration option on the Catalogs page of the Lake Formation console

Figure 2: Enabling the S3 Tables integration in the Lake Formation console

Choose the role you created earlier when prompted for an IAM role, and select Allow external engines to access data in Amazon S3 locations with full table access.

S3 Tables integration performs the following:

Registers the S3 Tables data location with Lake Formation.
Creates the s3tablescatalog federated catalog in Glue.

Important: Before enabling the integration, verify your Lake Formation data lake settings have empty default permissions to prevent IAMAllowedPrincipals from being auto-granted on the catalog:

aws lakeformation put-data-lake-settings \
    --data-lake-settings '{"DataLakeAdmins":[{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ADMIN_ROLE>"}],"CreateDatabaseDefaultPermissions":[],"CreateTableDefaultPermissions":[]}' \
    --region <AWS_REGION>

The S3 Tables integration dialog in Lake Formation with full table access selected

Figure 3: Selecting full table access for external engines during S3 Tables integration

When you select this option, you allow external engines to access data in Amazon S3 locations with full table access, and Lake Formation grants full table-level access to external engines. Column-level and row-level filtering are not enforced for external engine connections. Access is granted at the whole-table level.

Verify the integration by confirming the catalog in Lake Formation console.

Create a table and insert data

CREATE TABLE `<NAMESPACE>`.orders (
    order_id STRING,
    customer_id STRING,
    amount BIGINT,
    order_date DATE,
    region STRING
)
TBLPROPERTIES ('table_type' = 'iceberg');

INSERT INTO orders
VALUES
    ('ORD-001', 'C100', 4500, DATE '2024-06-01', 'EMEA'),
    ('ORD-002', 'C200', 8900, DATE '2024-06-01', 'EMEA'),
    ('ORD-003', 'C100', 3200, DATE '2024-06-02', 'NAMER'),
    ('ORD-004', 'C300', 12000, DATE '2024-06-02', 'NAMER'),
    ('ORD-005', 'C400', 6700, DATE '2024-06-03', 'APJ'),
    ('ORD-006', 'C200', 4100, DATE '2024-06-03', 'APJ'),
    ('ORD-007', 'C500', 9500, DATE '2024-06-04', 'EMEA'),
    ('ORD-008', 'C100', 2800, DATE '2024-06-04', 'LATAM'),
    ('ORD-009', 'C600', 15000, DATE '2024-06-05', 'NAMER'),
    ('ORD-010', 'C300', 7200, DATE '2024-06-05', 'LATAM');

Configuring cross-cloud access

BigQuery assumes an AWS IAM role using OIDC federation to access the AWS Glue IRC. This section walks through creating the role, OIDC provider, and permissions.

Create the OIDC identity provider

Register Google as an OIDC identity provider in your AWS account. This allows AWS to validate tokens issued by Google’s identity service:

aws iam create-open-id-connect-provider \
    --url https://accounts.google.com \
    --client-id-list accounts.google.com \
    --thumbprint-list 08745487e891c19e3078c1f2a07e452950ef36f6

The –thumbprint-list parameter is optional. When omitted, IAM automatically retrieves the thumbprint from the OIDC provider’s certificate. See AWS documentation for details.

Create the cross-cloud IAM role on AWS

Sign in to the AWS Management Console. Create the role with a placeholder trust policy. You will update it with the actual BigLake service account ID after you create the federated catalog in Google Cloud.

aws iam create-role \
    --role-name bigquery-cross-cloud-role \
    --max-session-duration 43200 \
    --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["PLACEHOLDER"],
            "accounts.google.com:aud": ["PLACEHOLDER"]
          }
        }
      }]
    }'

The --max-session-duration 43200 allows sessions up to 12 hours, which is needed for long-running BigQuery queries.

Attach permissions

The permissions policy differs based on your access control approach. For the Lake Formation approach, attach the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueRead",
      "Effect": "Allow",
      "Action": [
        "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases",
        "glue:GetTable", "glue:GetTables", "glue:GetPartition", "glue:GetPartitions"
      ],
      "Resource": [
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog/<TABLE_BUCKET>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:database/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>/*"
      ]
    },
    {
      "Sid": "S3TablesRead",
      "Effect": "Allow",
      "Action": [
        "s3tables:GetTableBucket", "s3tables:ListTableBuckets",
        "s3tables:ListNamespaces", "s3tables:GetNamespace",
        "s3tables:ListTables", "s3tables:GetTable",
        "s3tables:GetTableMetadataLocation", "s3tables:GetTableData"
      ],
      "Resource": [
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>",
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>/*"
      ]
    },
    {
      "Sid": "LakeFormationCredentialVending",
      "Effect": "Allow",
      "Action": ["lakeformation:GetDataAccess"],
      "Resource": "*"
    }
  ]
}

Grant Lake Formation permissions

Lake Formation permissions work as a layered grant model: you grant access at each level of the catalog hierarchy, from catalog down to table. The cross-cloud role needs DESCRIBE on the catalog and database so it can discover what exists, and SELECT plus DESCRIBE on the table so it can read the actual data. Without grants at every level, Lake Formation denies access even if the IAM policy allows it.

If using Lake Formation, grant the bigquery-cross-cloud-role access to your tables:

Grant catalog permission: DESCRIBE.
Grant database permission: DESCRIBE.
Grant table permission: SELECT, DESCRIBE.

Grant Lake Formation permissions on the cross-cloud role (one-time).

aws lakeformation grant-permissions     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role"}'     --resource '{"Catalog":{"Id":"<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>"}}'     --permissions '["DESCRIBE"]'     --region <AWS_REGION>

aws lakeformation grant-permissions     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role"}'     --resource '{"Database":{"CatalogId":"<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>","Name":"<NAMESPACE>"}}'     --permissions '["DESCRIBE"]'     --region <AWS_REGION>

aws lakeformation grant-permissions     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role"}'     --resource '{"Table":{"CatalogId":"<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>","DatabaseName":"<NAMESPACE>","Name":"orders"}}'     --permissions '["SELECT","DESCRIBE"]'     --region <AWS_REGION>

Before granting Lake Formation permissions, revoke the default IAMAllowedPrincipals access. By default, Lake Formation grants IAMAllowedPrincipals full access to all databases and tables, so you first need to revoke this to enforce fine grain access. IAMAllowedPrincipals provides backward compatibility when you start using Lake Formation permissions to secure the Data Catalog resources that were earlier protected by IAM policies for AWS Glue.

Set up Lake Formation for external engines

For table metadata to sync from Glue to BigLake/BigQuery, the following Lake Formation settings are required. You might notice that a similar setting also appeared during the S3 Table integration setup. The first one registers the data location and enables external access at the catalog level, while this one enables the Lake Formation credential vending mechanism at the account level for all external engines. For a clean cross-cloud setup, we recommend that you enable both.

In the Lake Formation console, choose Administration, then Application integration settings, and then select Allow external engines to access data in Amazon S3 locations with full table access.

Application integration settings in the Lake Formation console with external-engine access enabled

Figure 4: Enabling external-engine access in Lake Formation application integration settings

Connecting BigQuery to S3 Tables

With the AWS side configured, create the federated catalog in Google Cloud that connects BigQuery to the AWS Glue IRC.

Create the federated catalog

Authenticate to Google Cloud using gcloud auth login, or use Cloud Shell, which is pre-authenticated. Verify the BigLake API is enabled:

gcloud services enable biglake.googleapis.com --project="<GCP_PROJECT_ID>"

For Lake Formation mode (with credential vending):

gcloud alpha biglake iceberg catalogs create <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --catalog-type=federated \
    --federated-catalog-type=glue \
    --glue-aws-region=<AWS_REGION> \
    --glue-aws-role-arn=arn:aws:iam::<ACCOUNT_ID>:role/bigquery-cross-cloud-role \
    --glue-warehouse=<ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET> \
    --primary-location=<GCP_REGION> \
    --credential-mode=vended-credentials

The --glue-warehouse parameter uses the format <AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>. This tells the AWS Glue IRC to scope requests to your specific S3 Tables bucket within the federated catalog hierarchy.

The --credential-mode=vended-credentials flag (Lake Formation mode) instructs BigQuery Lakehouse to request scoped temporary credentials from Lake Formation rather than using the role’s IAM permissions directly for data access.

Retrieve the BigLake service account ID

After catalog creation, Google provisions a dedicated service account for your federated catalog. Retrieve its numeric ID:

BIGLAKE_SA_ID=$(gcloud alpha biglake iceberg catalogs describe <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --format="value(biglake-service-account-id)")
echo $BIGLAKE_SA_ID

Update the AWS trust policy

Back on AWS, replace the placeholder in the IAM role’s trust policy with the actual service account ID:

aws iam update-assume-role-policy \
    --role-name bigquery-cross-cloud-role \
    --policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["<BIGLAKE_SA_ID>"],
            "accounts.google.com:aud": ["<BIGLAKE_SA_ID>"]
          }
        }
      }]
    }'

Register the service account ID in the OIDC provider’s audience list. Without this step, AWS rejects the token because the aud claim doesn’t match any registered client:

aws iam add-client-id-to-open-id-connect-provider \
    --open-id-connect-provider-arn "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com" \
    --client-id "<BIGLAKE_SA_ID>"

Set up metadata sync

Wait 3–5 minutes for IAM changes to propagate globally, then set up background refresh:

gcloud alpha biglake iceberg catalogs update <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --refresh-interval=300s

The --refresh-interval (300 seconds in this example) determines how often BigQuery syncs metadata from the AWS Glue IRC. New tables and schema changes appear in BigQuery within this interval.

Querying from BigQuery

After the catalog refresh completes, BigQuery automatically creates external datasets corresponding to the synced namespaces. No manual CREATE SCHEMA is required.

Verify the sync:

gcloud alpha biglake iceberg namespaces list \
    --catalog="<FEDERATED_CATALOG_NAME>" \
    --project="<GCP_PROJECT_ID>"

Run a query in BigQuery:

SELECT * FROM `<GCP_PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders` LIMIT 1000

Sample Query Output:

SELECT
    customer_id,
    COUNT(*) as order_count,
    SUM(amount) as total_spend
FROM `<GCP_PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders`
GROUP BY customer_id
ORDER BY total_spend DESC

Figure 5: BigQuery query results returned through Lake Formation credential vending

Schema evolution

When new columns are added to an Iceberg table on the AWS side (through Spark, Athena, or the AWS Glue IRC), the schema change is captured in Iceberg’s metadata. On the next Lakehouse refresh cycle, BigQuery picks up the new columns automatically. No DDL changes are needed in BigQuery.

Metadata freshness

The s3tablescatalog catalog in AWS Glue is a federated catalog that resolves table metadata live from the S3 Tables service on each request. When a streaming job commits new data to an S3 Table, the latest metadata is immediately available through the AWS Glue IRC. BigQuery sees the update on its next refresh cycle (as configured by --refresh-interval).

OIDC identity federation

The trust relationship between Google Cloud and AWS uses OpenID Connect. When BigQuery Lakehouse needs to access your data, it presents a signed JWT token containing:

iss: accounts.google.com (the issuer)
sub: The BigLake service account ID (identifies which catalog is making the request)
aud: The same service account ID (the intended audience)

Network path

Clean up

To avoid ongoing charges, remove the resources created in this walkthrough.

On AWS:

# Delete the table (if created for this walkthrough)
aws s3tables delete-table \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace analytics --name orders --region <AWS_REGION>

# Delete namespace and table bucket
aws s3tables delete-namespace \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace <NAMESPACE> --region <AWS_REGION>

aws s3tables delete-table-bucket --name <TABLE_BUCKET> --region <AWS_REGION>

# Delete IAM role and OIDC provider (if no longer needed)
aws iam delete-role --role-name bigquery-cross-cloud-role

On Google Cloud:

gcloud alpha biglake iceberg catalogs delete <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" --location=<GCP_REGION>

Conclusion

This post demonstrated how to query Amazon S3 Tables from Google BigQuery using AWS Lake Formation credential vending, where Lake Formation manages the permissions and issues temporary, scoped credentials for data access. With the open Iceberg format, you can write data once on AWS and read it from supported engines that speak Iceberg, including BigQuery.

Together with the IAM approach covered in Part 1, two access control modes provide flexibility: IAM for teams who want a straightforward setup and Lake Formation for organizations with complex governance requirements where multiple engines need centrally managed access to the same data.

To get started with this pattern in your environment:

Review the Amazon S3 Tables documentation for table bucket setup.
Follow the Google Cloud cross-cloud Lakehouse setup guide for the BigQuery federation configuration.
Explore the AWS Glue Iceberg REST Catalog APIs for programmatic access patterns.

About the authors

Track SageMaker Unified Studio project costs with custom tags and AWS CUR

2026-08-20 Nisha Gambhir

Post Syndicated from Nisha Gambhir original https://aws.amazon.com/blogs/big-data/track-sagemaker-unified-studio-project-costs-with-custom-tags-and-aws-cur/

Organizations running machine learning (ML), analytics, and generative AI workloads on Amazon SageMaker Unified Studio domains and projects face a common cost governance challenge. System tags (AmazonDataZoneDomainId and AmazonDataZoneProject) are automatically propagated to all underlying project resources. However, custom tags such as CostCenter, Team, or Environment are not propagated to dynamic resources created through the Studio UI. This creates a gap when you need to report project costs grouped by custom tags.

In this post, we walk through a serverless solution that bridges this gap by enriching AWS Cost and Usage Report (CUR) data with custom project tags. By the end of this post, you can build an Amazon Quick Sight dashboard to filter and analyze Amazon SageMaker Unified Studio project costs by any custom tag dimension that you define. This gives your team the visibility to make informed spending decisions.

Solution overview

The solution consists of three automated subsystems:

Event-driven tag lookup management – An Amazon EventBridge rule captures Amazon DataZone project lifecycle events (Create, Update, Delete) and triggers an AWS Lambda function. The function maintains an Amazon DynamoDB lookup table that maps each project’s DomainId and ProjectId to its custom tags.
CUR enrichment pipeline – An AWS Glue extract, transform, and load (ETL) job reads CUR 2.0 Parquet data from Amazon Simple Storage Service (Amazon S3). The job joins each billing line item with the DynamoDB lookup table using the system tags (DomainId, ProjectId), appends the custom tag values as new columns, and writes the enriched data back to Amazon S3.
Cost visualization – An Amazon Quick Sight dashboard backed by a custom SQL dataset over Amazon Athena provides interactive cost and consumption analytics filtered by custom tags.

Architecture

The following diagram shows the end-to-end architecture:

Figure 1: SageMaker Unified Studio project custom tag cost reporting

The workflow is as follows:

An Amazon SageMaker Unified Studio administrator creates or updates a project with custom tags.
AWS CloudTrail captures the API call.
Amazon EventBridge matches the event.
The Lambda orchestrator writes the tag mapping to DynamoDB.
Separately, AWS Data Exports delivers CUR data to Amazon S3.
The AWS Glue ETL job enriches CUR line items with custom tags from DynamoDB.
The AWS Glue Crawler catalogs the enriched data.
Amazon Quick Sight visualizes costs by custom tags.

Prerequisites

Before deploying this solution, you need:

An Amazon SageMaker Unified Studio domain (you create projects after deployment).
AWS Cloud Development Kit (AWS CDK) CLI installed.
Python 3.12+.
Amazon Quick Sight Enterprise edition enabled in your account.
An AWS Identity and Access Management (IAM) user or role with permissions to deploy AWS CloudFormation stacks.

Step 1: Configure custom tags on your project profile

You configure custom tags on project profiles through the Amazon DataZone API. First, enable custom tags on your project profile:

aws datazone update-project-profile \
  --domain-identifier $DOMAIN_ID \
  --identifier $PROJECT_PROFILE_ID \
  --region $REGION \
  --allow-custom-project-resource-tags \
  --project-resource-tags '[
  {"key": "CostCenter", "value": "default", "isValueEditable": true},
  {"key": "Team", "value": "default", "isValueEditable": true},
  {"key": "Environment", "value": "default", "isValueEditable": true}
]'

When creating or updating a project, set the tag values:

aws datazone update-project \
  --domain-identifier $DOMAIN_ID \
  --identifier $PROJECT_ID \
  --project-profile-version latest \
  --region $REGION \
  --resource-tags '{"CostCenter": "CC-100", "Team": "ML-Platform", "Environment": "Production"}'

Important: The AmazonSageMakerProvisioning-<domainAccountId> role needs an inline policy that permits your custom tag keys. Without this, project environment deployment fails.

The following is the inline policy that’s used for the custom tags shared in this post:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCustomTagKeys",
      "Effect": "Allow",
      "Action": [
        "sagemaker:AddTags",
        "sagemaker:DeleteTags",
        "cloudformation:TagResource",
        "cloudformation:CreateStack",
        "cloudformation:UpdateStack"
      ],
      "Resource": "*",
      "Condition": {
        "ForAnyValue:StringLike": {
          "aws:TagKeys": [
            "AmazonDataZone*",
            "CostCenter",
            "Team",
            "Environment"
          ]
        }
      }
    }
  ]
}

Step 2: Activate cost allocation tags

Activate the SageMaker Unified Studio system tags as cost allocation tags so they appear in CUR data:

aws ce update-cost-allocation-tags-status \
  --cost-allocation-tags-status '[
  {"TagKey": "AWSDataZoneProject", "Status": "Active"},
  {"TagKey": "AmazonDataZoneDomainId", "Status": "Active"}
]'

These tags take up to 24 hours to start appearing in CUR reports after activation.

Step 3: Deploy the infrastructure

The solution is packaged as a CDK application. Clone the GitHub repository and deploy:

# Install dependencies
pip install -r requirements.txt

# Bootstrap CDK (first time only)
cdk bootstrap aws://$ACCOUNT_ID/$REGION

# Deploy
cdk deploy

This creates the following resources:

DynamoDB table (smus-project-tag-lookup) – stores project-to-tag mappings.
Lambda function (smus-orchestrator) – processes project lifecycle events.
Amazon EventBridge rule – matches Amazon DataZone CreateProject/UpdateProject/DeleteProject events.
S3 buckets – for raw CUR and enriched CUR data.
AWS Glue ETL job (smus-cur-enrichment) – enriches CUR with custom tags.
AWS Glue Crawler – catalogs enriched data.
Amazon Simple Notification Service (Amazon SNS) topic – pipeline failure alerts.

Note: The solution uses serverless components (Lambda, DynamoDB on-demand, AWS Glue, Amazon Quick Sight), so you only pay for what you use. The primary cost drivers are AWS Glue ETL job execution time and Amazon Quick Sight SPICE storage.

Step 4: Configure CUR delivery

Create a CUR 2.0 export through AWS Data Exports that delivers Parquet files to the CUR S3 bucket created by the stack. The following screenshots show the complete configuration process in the AWS Billing and Cost Management console.

To create the export, follow these steps:

Go to AWS Billing and Cost Management and then choose Data Exports.
Choose Create in the upper right corner of the Exports and dashboards page. The Data Exports console shows any existing exports, their status, export type, data table, and last refresh date.
On the Create export page, under Export details, select Standard data export and enter an export name. Under Data table content settings, select CUR 2.0.
Under Data table configurations, set Time granularity to Hourly. The configuration page also lets you choose additional export content options such as including resource IDs, split cost allocation data, caller identity allocation data, and capacity reservation columns.
Under Data export delivery options, set Compression type and file format to Parquet. Under Data export storage settings, configure the S3 bucket to: smus-cur-report-{account-id}-{region} and set the S3 path prefix as needed. Choose Create to finish.

Data Exports console listing existing exports with status, type, and last refresh date

Figure 2: Data Exports page listing existing exports

Create export page with Standard data export selected and CUR 2.0 chosen

Figure 3: Create export page with Standard data export and CUR 2.0 selected

Figure 4: Data table configurations with time granularity set to Hourly

Data export delivery options with Parquet format and the S3 storage destination configured

Figure 5: Data export delivery options with Parquet format and S3 storage settings

Step 5: How the event-driven tag capture works

When a project is created or updated in Amazon SageMaker Unified Studio (through the Studio UI or API), the following happens automatically:

CloudTrail logs the Amazon DataZone API call.
Amazon EventBridge matches the event.
Amazon EventBridge invokes the Lambda function.
The Lambda extracts custom tags from the CloudTrail event payload.
The Lambda writes a record to DynamoDB with the DomainId, ProjectId, and all custom tag key-value pairs.

The Lambda function reads tags directly from the responseElements.resourceTags field of the CloudTrail event rather than making a separate GetProject API call. This avoids a race condition where GetProject might return empty tags while the project is in the UPDATING state.

def _extract_tags_from_event(detail):
    tags = {}
    response_elements = detail.get("responseElements") or {}
    for tag_entry in response_elements.get("resourceTags", []):
        if isinstance(tag_entry, dict) and "key" in tag_entry:
            tags[tag_entry["key"]] = tag_entry["value"]
    request_params = detail.get("requestParameters") or {}
    req_tags = request_params.get("resourceTags", {})
    if isinstance(req_tags, dict):
        tags.update(req_tags)
    return tags

Step 6: How the CUR enrichment works

The AWS Glue ETL job runs on a schedule (after each CUR delivery):

Reads CUR Parquet files from the CUR S3 bucket.
Reads all records from the DynamoDB lookup table.
Performs a left outer join on DomainId and ProjectId.
Appends custom tag columns (CostCenter, Team, Environment, and so on) to each CUR line item.
Writes enriched Parquet to the enriched S3 bucket.

Line items without a matching project in the lookup table retain all original columns with NULL custom tag values. No data is dropped.

joined_df = cur_df.join(
    lookup_df,
    on=(
        (cur_df[DOMAIN_COL] == lookup_df["domainId"])
        & (cur_df[PROJECT_COL] == lookup_df["projectId"])
    ),
    how="left_outer",
)

Step 7: Set up the Amazon Quick Sight dashboard

After the first ETL run and crawler execution, set up the Amazon Quick Sight dashboard:

python scripts/setup_quicksight.py \
  --account-id $ACCOUNT_ID \
  --region $REGION \
  --quicksight-user $QUICKSIGHT_USER_ARN

This creates a dashboard with five visuals:

Cost by Custom Tag (CostCenter) – horizontal bar chart.
Cost by Project – horizontal bar chart.
Daily Cost Trend – line chart.
Cost by Service per Project – stacked bar chart.
Usage by Project & Service – summary table.

And six interactive list filters: Domain, Project, CostCenter, Team, Environment, Service.

The custom SQL includes a CASE statement for service categorization:

SELECT
  line_item_usage_start_date,
  line_item_product_code,
  line_item_usage_amount,
  line_item_unblended_cost,
  resource_tags_user_amazondatazone_domain_id AS domain_id,
  resource_tags_user_amazondatazone_project AS project_id,
  costcenter, team, environment,
  CASE
    WHEN line_item_product_code = 'AmazonSageMaker' THEN 'SageMaker'
    WHEN line_item_product_code = 'AmazonS3' THEN 'S3'
    WHEN line_item_product_code = 'AWSGlue' THEN 'Glue'
    ELSE line_item_product_code
  END AS service_category
FROM "smus_cost_reporting"."enriched_cur"
WHERE line_item_unblended_cost > 0

Step 8: Verifying the solution

After deploying the infrastructure and setting up the dashboard, verify that each component of the pipeline is functioning correctly.

8.1 Verify Amazon EventBridge is capturing project events

Open the Amazon EventBridge console.
In the navigation pane, choose Rules.
Select the rule created by the CDK stack (for example, SmusCostReporting-ProjectTagRule).
Choose the Monitoring tab.
Confirm that the invocations are being recorded in the metrics.

Create or update an Amazon SageMaker Unified Studio project with custom tags using the following command:

aws datazone update-project \
  --domain-identifier <domain-id> \
  --identifier <project-id> \
  --custom-tags CostCenter=Engineering Team=DataPlatform Environment=Production

Within a few seconds, the Amazon EventBridge rule should show a new invocation in its metrics.

8.2 Verify DynamoDB schema and tag mappings

The DynamoDB lookup table uses a simple key schema:

Attribute	Type	Role
domainId	String	Partition Key
projectId	String	Sort Key
CostCenter	String	Custom tag
Team	String	Custom tag
Environment	String	Custom tag

Custom tags are stored as dynamic attributes. Any tag key set on a project becomes a column in the table.

8.2.1 Verify DynamoDB table contains tag mappings

Open the DynamoDB console.
Navigate to the table created by the stack (for example, SmusCostReporting-ProjectTagsTable).
Choose Explore table items.

Scan for your project with the following keys:

Partition key (domainId): <your-domain-id>
Sort key (projectId): <your-project-id>

Confirm the item contains the expected custom tag attributes (CostCenter, Team, Environment) with the values you assigned.

Alternatively, use the AWS CLI:

aws dynamodb get-item \
  --table-name SmusCostReporting-ProjectTagsTable \
  --key '{"domainId": {"S": "<domain-id>"}, "projectId": {"S": "<project-id>"}}'

8.3 Verify the AWS Glue ETL job enriches CUR data

Wait for the next CUR delivery (hourly if configured as described in Step 4).
Wait for the subsequent AWS Glue job execution.
Open the AWS Glue console.
In the navigation pane, choose ETL Jobs.
Confirm the job completed successfully (status: Succeeded).

Query the enriched data in Amazon Athena to confirm custom tag columns are populated:

SELECT
  line_item_usage_start_date,
  line_item_product_code,
  line_item_unblended_cost,
  costcenter,
  team,
  environment
FROM "smus_cost_reporting"."enriched_cur"
WHERE costcenter IS NOT NULL
LIMIT 10;

You should see rows with your custom tag values populated in the costcenter, team, and environment columns.

8.4 Verify the Amazon Quick Sight dashboard displays enriched data

Open the Amazon Quick Sight console and navigate to the dashboard created by the setup script.
Confirm that:
- The Cost by Custom Tag (CostCenter) bar chart displays cost data grouped by your CostCenter values.
- The list filters for CostCenter, Team, and Environment contain selectable values.
- Selecting a filter value correctly narrows the displayed data.
If the dashboard shows no data, verify that:
- The AWS Glue Crawler has run after the ETL job (check the crawler’s last run status in the AWS Glue console).
- The SPICE dataset has been refreshed. In the Amazon Quick Sight console, navigate to Datasets, select the dataset, and then choose Refresh now.

Figure 6 shows the Amazon Quick Sight dashboard with two side-by-side horizontal bar charts: Cost by Cost Center and Cost by Project. Domain Name and Project Name list filters appear at the top.

Amazon Quick Sight dashboard with Cost by Cost Center and Cost by Project bar charts and Domain and Project filters

Figure 6: Amazon Quick Sight dashboard showing cost data by custom tags, including Cost by Cost Center and Cost by Project bar charts with Domain Name and Project Name filters

Note: The first end-to-end cycle can take up to 48 hours depending on CUR delivery timing. After the initial cycle completes, subsequent updates will flow automatically on the configured schedule.

Operational considerations

Monitoring: The Amazon SNS topic smus-cost-reporting-alerts receives notifications when the AWS Glue ETL job fails or the Lambda orchestrator encounters repeated errors. Subscribe an email address or Slack webhook to stay informed. For instructions on how to create a subscription, see Subscribing to an Amazon SNS topic.

Cost: The solution uses serverless components (Lambda, DynamoDB on-demand, AWS Glue, Amazon Quick Sight, SPICE) so you only pay for what you use. The primary cost drivers are AWS Glue ETL job execution time and Amazon Quick Sight SPICE storage.

Scaling: The DynamoDB table uses on-demand capacity and can scale to accommodate your projects. You can scale the AWS Glue ETL job by increasing the number of workers for larger CUR datasets. For more information, see Managing throughput capacity automatically with DynamoDB auto scaling.

New tag keys: When you add new custom tag keys to projects, the ETL automatically picks them up as new columns. The AWS Glue Crawler’s UPDATE_IN_DATABASE policy adds new columns to the catalog table without manual intervention.

Cleanup

Warning: The following cleanup steps will permanently delete all CUR data, project tag mappings, and Amazon Quick Sight dashboards.

To remove all resources:

# Delete Amazon Quick Sight resources
python scripts/setup_quicksight.py --account-id $ACCOUNT_ID --region $REGION --quicksight-user $QS_USER --clean

# Delete CDK stack
cdk destroy

Go to AWS Billing and Cost Management, and then choose Data Exports and delete the CUR 2.0 export created in Step 4.

Deactivate the cost allocation tags that were activated in Step 2:

aws ce update-cost-allocation-tags-status \
  --cost-allocation-tags-status '[
  {"TagKey": "AWSDataZoneProject", "Status": "Inactive"},
  {"TagKey": "AmazonDataZoneDomainId", "Status": "Inactive"}
]'

Conclusion

In this post, we showed how to build an end-to-end cost reporting solution for Amazon SageMaker Unified Studio projects using custom tags. This solution combines tag capture driven by Amazon EventBridge, CUR enrichment through AWS Glue ETL, and visualization in Amazon Quick Sight. With it, organizations can track and attribute costs by CostCenter, Team, Environment, or any custom dimension. This works even for resources created through the Studio UI that don’t receive custom tag propagation.

This solution serves as an extension to the custom tag propagation feature and reports cost for all project resources. The architecture is fully serverless, automated, and can be deployed to any AWS account using the provided CDK application.

To start building your custom tag cost reporting pipeline, visit the GitHub repository. To learn more about the underlying services, visit the Amazon SageMaker Unified Studio service page. For a related approach to custom tag governance, see Use Amazon SageMaker custom tags for project resource governance and cost tracking

References

About the authors

Secure SageMaker Unified Studio access with SAML and conditional policies

2026-08-19 Manos Samatas

Post Syndicated from Manos Samatas original https://aws.amazon.com/blogs/big-data/secure-sagemaker-unified-studio-access-with-saml-and-conditional-policies/

Amazon SageMaker Unified Studio is a single data and AI development environment that brings together data preparation, analytics, and machine learning (ML) development in one place. By unifying these workflows, it saves teams from managing multiple tools and makes it straightforward for data scientists, analysts, and developers to build, train, and deploy ML models while collaborating. In Amazon SageMaker Unified Studio, a domain is the organizing entity for connecting your assets, users, and their projects. With Amazon SageMaker unified domains, you have the flexibility to reflect the data and analytics needs of your organizational structure. You can create a single unified domain for your enterprise or multiple domains for different business units.

Some enterprises, especially those in regulated industries, might require limiting access to trusted networks (such as VPN CIDRs) or to managed devices that meet compliance standards through device attestation.

In this post, we demonstrate how to integrate SageMaker Unified Studio as a custom SAML application and apply conditional access policies for enforcing device compliance, IP-based restrictions, or multi-factor authentication (MFA). For this post, we use Okta as the identity provider (IdP).

Solution overview

This solution demonstrates how to integrate Amazon SageMaker Unified Studio (SMUS) with external SAML identity providers such as Okta. The integration enforces enterprise security controls, including trusted network access, device compliance, and multi-factor authentication. With this integration, organizations in regulated industries can maintain strict access controls while providing single sign-on for their data science and AI development teams. By using SAML 2.0 federation with conditional access policies, you can help make sure that only authenticated users on compliant devices from trusted networks gain access. This access applies to your SageMaker Unified Studio domains and the associated data and AI workloads.

SAML authentication flow from a corporate device through the identity provider and AWS STS to Amazon SageMaker Unified Studio

Authentication flow for accessing SageMaker Unified Studio through SAML

The architecture diagram illustrates the secure authentication flow for accessing SageMaker Unified Studio through SAML integration:

Users typically initiate access from corporate-managed devices through VPN or trusted network connections.
The IdP authenticates the user and evaluates conditional access policies defined by your organization. Based on these policies, it checks for trusted devices, approved source IP ranges, and MFA completion. If any policy fails, the login is rejected. Otherwise, authentication proceeds.
Upon successful authentication and policy validation, the IdP generates a digitally signed SAML assertion containing user attributes and group memberships, securely delivering it to the user’s browser through HTTP POST binding.
The client browser automatically posts the SAML assertion to the AWS Security Token Service (AWS STS) sign-in endpoint. There, the AWS IAM Identity Provider validates the trust relationship with your corporate IdP through pre-configured SAML federation settings.
AWS STS validates the SAML assertion signature and authenticity. It then maps the user attributes to a specifically configured IAM role with SageMaker Unified Studio permissions, including the datazone:GetIamPortalLoginUrl permission required for domain access.
AWS STS confirms successful role assumption and generates temporary AWS credentials with a defined session duration. It then issues an HTTP redirect that returns the browser to the SageMaker Unified Studio domain with authenticated session tokens.
Users gain access to the unified environment for data preparation, analytics, and machine learning development. All activities are governed by the assumed IAM role permissions and logged for comprehensive audit trails.

Walkthrough

In this walkthrough, you create a SAML application in Okta, connect it to AWS, and configure a SageMaker Unified Studio domain to use it for authentication.

Prerequisites

Before you get started, make sure you have the following:

Familiarity with Amazon SageMaker Unified Studio.
A basic understanding of SAML 2.0.
AWS Identity and Access Management (IAM) permissions to create a domain in Amazon SageMaker Unified Studio.
Access to your SAML IdP (such as Okta or Entra ID) to create and configure a SAML application.

Step 1: Create an application in Okta

The first step is to set up a new SAML application in Okta that manages authentication for SMUS.

In Okta, go to Applications → Create App Integration, and choose SAML 2.0.
Provide an App name.
Set the Single sign-on URL to https://signin.aws.amazon.com/saml.
Set Name ID format to Persistent.
Set the Audience URI (SP Entity ID) to https://signin.aws.amazon.com/saml.
Choose Next, and finish creating the application.
Once created, copy the Metadata URL and Sign On URL. You need these in later steps.

Step 2: Create an identity provider in IAM

Now, let’s connect Okta to AWS by creating an IAM identity provider. This allows AWS to trust authentication responses from Okta.

Open the IAM console.
Go to Identity providers → Add provider.
Select SAML as the provider type.
Provide a Provider name.
In Okta, go to your application’s Sign On tab, choose Identity Provider metadata, and save the XML file. Upload it here.
Choose Add provider.
Copy the ARN of this provider. You need it when you create the role.

Step 3: Create an IAM role for Okta

Next, create an IAM role that Okta can assume. This role defines what access users have when they sign in through Okta.

In IAM, go to Roles → Create role.
Use the following trust policy (replace both instances of “{Replace with Identity provider ARN}” with the ARN you copied in Step 2):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "{Replace with Identity provider ARN}"
            },
            "Action": "sts:AssumeRoleWithSAML",
            "Condition": {
                "StringEquals": {
                    "SAML:aud": "https://signin.aws.amazon.com/saml"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "{Replace with Identity provider ARN}"
            },
            "Action": "sts:TagSession",
            "Condition": {
                "StringLike": {
                    "aws:RequestTag/Email": "*"
                }
            }
        }
    ]
}

Attach a permission policy. For example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "datazone:GetIamPortalLoginUrl",
            "Resource": "arn:aws:datazone:<REGION>:<ACCOUNT-ID>:domain/<DOMAIN-ID>"
        }
    ]
}

Replace <REGION>, <ACCOUNT-ID>, and <DOMAIN-ID> with the corresponding values from your SageMaker Unified Studio domain ARN (arn:aws:sagemaker:<REGION>:<ACCOUNT-ID>:domain/<DOMAIN-ID>). You can find the domain ARN in the SageMaker console under Domains.

Step 4: Configure SAML assertions

To make sure AWS understands who is signing in, configure the SAML assertions in Okta.

Open your application in Okta.
Go to General → SAML Settings → Edit.
Choose Next until you reach Attribute Statements.
Add the following mappings:
- https://aws.amazon.com/SAML/Attributes/PrincipalTag:Email → user.email.
- https://aws.amazon.com/SAML/Attributes/Role → {IAMROLEARN,IdentityProviderARN}.
- https://aws.amazon.com/SAML/Attributes/RoleSessionName → user.email.

Step 5: Create an SMUS domain

Finally, let’s set up the SMUS domain and tie it all together.

Note: Creating a SageMaker Unified Studio domain incurs charges. For pricing details, see the Amazon SageMaker pricing page.

Open the Amazon SageMaker console.
Choose Create domain.
Choose Manual setup (this allows for SAML integration).
Enter a domain name, then choose Create.
In Configure SSO user access, select SAML, then choose Next.
Set the IdP SSO URL to the Sign On URL from Step 1.
Select Do not require assignments. (Access is instead managed by your IdP team through Okta or Entra.)
Choose Next, then choose Save.

To verify the integration works, open your SMUS domain and choose Sign in with SSO. You are redirected to Okta, and conditional access policies such as VPN, device attestation, or MFA apply automatically.

Open your SMUS domain URL in a browser.
Choose Sign in with SSO.
Confirm that you are redirected to Okta for authentication.
Sign in with your Okta credentials.
Verify that you are redirected back to the SMUS domain with access to your projects.

Step 6: Assign users to the Okta application

Before users can authenticate through Okta to access SMUS, you must assign them to the application.

In Okta, navigate to your SAML application.
Go to the Assignments tab.
Choose Assign, and select Assign to People or Assign to Groups.
Select the users or groups who need access to SMUS.
Choose Save and Go Back, then choose Done.

Step 7: Apply conditional access policies

Up to Step 5, we configured SMUS with an external SAML IdP. At this point, anyone assigned to the new application in your IdP can sign in and access the SMUS domain.

This is where conditional access policies come into play. Based on your organization’s governance model, you can add policies in your IdP to further control how and when users gain access. For example:

Restricting access to specific corporate IP address ranges (for example, only through VPN).
Enforcing device compliance so that only managed or secure devices can connect.
Adding MFA requirements for sensitive actions.
Applying device attestation to help assess whether the endpoint conforms to security baselines.

Most major IdPs, including Okta and Entra ID, support conditional access. You can find more details in their documentation:

These policies allow you to enforce the right level of protection, from something as simple as requiring users to connect through corporate networks to something as advanced as verifying device attestation across your fleet.

Clean up

To avoid incurring ongoing charges, delete the resources you created during this walkthrough:

Delete the Amazon SageMaker Unified Studio domain from the SageMaker console.
Delete the IAM role you created for Okta.
Delete the IAM identity provider.
Delete the SAML application in Okta.

Important: Deleting the SMUS domain permanently removes all projects, assets, and data within it. Back up any important work before proceeding.

Conclusion

By integrating SMUS with an external IdP through SAML, you can help enforce modern access controls based on your organization’s security requirements. This post walked through how to configure SMUS with a custom SAML application and pointed you toward resources for setting up conditional access policies.

With conditional access in place, you can decide, based on your organization’s needs, whether access should be limited to trusted users on trusted networks, trusted devices, or both. This approach can help provide a more secure and compliant login experience that aligns SMUS access with your company’s broader identity and security strategy.