Understanding and optimizing materialized views in Amazon Redshift

Post Syndicated from Nick Barclay original https://aws.amazon.com/blogs/big-data/understanding-and-optimizing-materialized-views-in-amazon-redshift/

Analytical queries often repeat the same expensive joins and aggregations, and materialized views (MVs) in Amazon Redshift optimize that pattern. An MV pre-computes and stores a query’s result, so the expensive work happens at refresh time instead of on every read. A query that would otherwise scan and aggregate large tables becomes a fast lookup of the stored result.

That speed has a running cost. Every MV consumes storage, and every refresh consumes compute, whether or not a query ever reads the result. A well-used MV repays that cost many times over. An MV that refreshes constantly but is not read costs more than it saves. So does one that fully recomputes a hundred million rows to serve a handful of queries.

As MVs become more popular in your organization, it can be challenging to keep track of which MVs are worth the maintenance costs. This post introduces a diagnostic view, v_mv_health, that reports each MV’s refresh eligibility, staleness, refresh cost, dependencies, and query activity. A companion stored procedure, sp_populate_mv_definitions(), creates and fills a table, mv_definitions, holding the one signal a view cannot compute. It reads each MV’s definition and flags the SQL constructs that block incremental refresh. Join the two for a single result set that covers these signals. Once deployed, the view and procedure give you the data to make MV health part of your regular operational review.

Solution overview

The following diagram shows how the two components fit together.

Diagram of the v_mv_health view and sp_populate_mv_definitions procedure reading Amazon Redshift system views and joining into the MV health dataset

Figure 1: How the v_mv_health view and sp_populate_mv_definitions procedure combine into the MV health dataset

Both components read from Amazon Redshift system views. The v_mv_health view reads them directly each time you query it. The sp_populate_mv_definitions() procedure reads each MV’s definition, flags constructs that block incremental refresh, and writes the results to the mv_definitions table each time it runs. Joining v_mv_health with mv_definitions produces the MV health dataset, a single result set with one row per MV.

Prerequisites

The diagnostic view and procedure require:

  • An Amazon Redshift provisioned cluster or an Amazon Redshift Serverless workgroup with one or more MVs.
  • sys:superuser role, or the sys:monitor role, so the view can read activity in the cluster or workgroup. Without it, the refresh and query activity the view reports covers only your own sessions. Grant it with GRANT ROLE sys:monitor TO <username>; (for an IAM identity, use the IAMR:role-name form). For the full list of system-defined roles, refer to Amazon Redshift system-defined roles.
  • For a non-superuser (such as a sys:monitor user), USAGE on each schema that contains MVs, because SVV_MV_INFO only returns MVs in schemas the user can access. Grant it with GRANT USAGE ON SCHEMA <schema> TO <username>;.

The diagnostic view

Amazon Redshift already exposes the signals, but each lives in a different system view:

The v_mv_health view joins them into a single row per MV: state, staleness, autorefresh interval, refresh cost, query activity, dependency depth, and whether parent MVs in a chain are stale. The definition for the v_mv_health view is available in the amazon-redshift-utils GitHub repository.

The stored procedure

One signal the view can’t provide is the MV’s definition text. The sp_populate_mv_definitions() procedure iterates through each MV and extracts its definition into the mv_definitions table, created automatically if it doesn’t exist. It then scans each definition for SQL constructs covered in Refresh eligibility. The definition for the sp_populate_mv_definitions() procedure is available in the amazon-redshift-udfs GitHub repository.

The procedure uses a two-phase approach: it stages the MV object IDs into a temporary table first, then walks that local table one row at a time. It executes the pg_get_viewdef() function, which operates on one view at a time. Results are collected in a second temporary table and then replace the contents of mv_definitions in a single transaction, so each run reflects the current state of your MVs. Amazon Redshift drops the temporary tables automatically at the end of the session, so no manual cleanup is needed.

Run the procedure, then join the v_mv_health view to the mv_definitions table it populated:

CALL sp_populate_mv_definitions();

SELECT
    v.*,
    d.incremental_blocker_hint
FROM v_mv_health v
LEFT JOIN mv_definitions d
ON v.mv_schema = d.mv_schema
AND v.mv_name = d.mv_name;

Both objects use the same SQL on provisioned clusters and serverless workgroups. They report the MVs in their own database, so create them in each database that contains MVs.

Note: The view uses a 7-day lookback window because SYS_MV_REFRESH_HISTORY, SYS_QUERY_DETAIL, and SYS_QUERY_HISTORY retain 7 days of data by default. Amazon Redshift can retain system table data beyond 7 days by automatically replicating it to Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3). Long-term retention supports SYS_QUERY_HISTORY and SYS_QUERY_DETAIL, so you can extend the query-activity signals by pointing the view’s queries on them at the retained tables. At the time of writing, SYS_MV_REFRESH_HISTORY is not among the retained system tables, so the refresh-cost columns remain limited to the 7-day in-cluster window unless you capture that history yourself.

What the output tells you

The joined output returns one row per MV:

Column What it means
mv_database Database containing the MV
mv_schema Schema containing the MV
mv_name MV name
state 1 (incremental), 0 (full_recompute), 101-105 (broken). Refer to SVV_MV_INFO
state_label incremental, full_recompute, or broken with its cause (for example, broken:column_renamed)
is_stale Whether MV data is out of date relative to base tables; NULL for MVs on external tables other than Iceberg.
is_autorefresh Whether automatic refresh is turned on
autorefresh_interval_sec Autorefresh interval in seconds; NULL when autorefresh is off
is_default_autorefresh_interval Whether the MV uses the cluster or workgroup default interval rather than its own. false when autorefresh is off
is_autorewrite_eligible Whether the optimizer can transparently rewrite queries to use this MV
refresh_count Refresh attempts in the last 7 days, including failed refreshes and refreshes that found the MV already up to date
query_count Queries that read the MV in the last 7 days, including queries served through automatic query rewriting. Refreshes are excluded; NULL if the MV’s storage table can’t be found in the catalog
recompute_to_query_ratio Full-recompute refreshes (not incremental) ÷ queries. High values indicate refresh cost may exceed benefit; NULL when query_count is 0 or NULL
avg_refresh_sec Average refresh duration in seconds
last_refresh_status Status message from the most recent refresh, such as incremental, full recompute, or already up to date
mv_rows Row count of the MV, for gauging refresh cost against usage
chain_depth Depth in a dependency chain (0 = no MV dependencies)
mv_dependencies Parent MVs the MV depends on; NULL if it has no parent MVs
stale_ancestors Parent MVs in the chain that are currently stale; NULL if no parent MV is stale
pct_of_total_query_time Refresh time as a percentage of total query time on the cluster or workgroup
incremental_blocker_hint SQL constructs preventing incremental refresh, evaluated only for state 0 MVs (none otherwise, unidentified when no known construct matches). From mv_definitions

Refresh eligibility

Amazon Redshift refreshes an MV in one of two ways. In an incremental refresh, Amazon Redshift identifies the changes to the data in the base tables since the last refresh and updates the data in the MV. In a full recompute, it reruns the MV definition, replacing the data in the MV. Amazon Redshift automatically chooses the method based on the query that defines the MV.

Each MV in Amazon Redshift falls into one of three states:

  • Full recompute (state 0): The entire MV is rebuilt from scratch on every refresh.
  • Incremental (state 1): Only changed rows are processed on refresh.
  • Broken (state 101-105): The MV cannot refresh at all and must be dropped and recreated.

The state column tells you which category each MV falls into. An MV’s state depends on its definition, not its size. Its size (mv_rows) determines how much that state costs. An incremental refresh on a 100-million-row table might process 50 changed rows, while a full recompute scans all 100 million regardless of how few changed.

Note: A state 1 MV can still occasionally fully recompute, for example after a VACUUM with the FORCE option on a base table while the MV is out of date, or when the MV hasn’t been refreshed for an extended period. These show up as full recomputes in recompute_to_query_ratio even though the state is 1. If you need VACUUM FORCE, run it straight after refreshing the dependent MVs.

Certain SQL constructs block incremental refresh. The incremental_blocker_hint column identifies which constructs are preventing incremental eligibility. If any of the following appear in the MV definition, the MV cannot refresh incrementally and falls back to a full recompute on every refresh:

  • Distinct: SELECT DISTINCT, COUNT(DISTINCT ...).
  • Outer joins: LEFT, RIGHT, or FULL OUTER JOIN.
  • Window functions: anything with an OVER() clause (ROW_NUMBER, RANK, SUM, NTILE, and so on).
  • Subqueries: in the WHERE clause or SELECT list (subqueries in the FROM clause are fine).
  • Mutable functions: GETDATE(), SYSDATE, RANDOM(), CURRENT_DATE.
  • Set operations: UNION, INTERSECT, EXCEPT, MINUS.
  • Ordered and holistic aggregates: MEDIAN, PERCENTILE_CONT, PERCENTILE_DISC, LISTAGG, APPROXIMATE COUNT, APPROXIMATE PERCENTILE.

An MV may trip more than one of these at once, and incremental_blocker_hint lists each one it finds.

Note: incremental_blocker_hint is a heuristic text scan of the MV definition, not a semantic parse, and is evaluated only for MVs in state 0. The other MVs show none. It matches whole-word SQL keywords, so it won’t trip on names like distinct_count, but a keyword inside a string literal (for example, WHERE status = 'union') can still produce a false positive. Treat it as guidance and verify by inspecting the definition.

OUTER JOIN is a frequent blocker. Outer joins (FULL, LEFT, and RIGHT) are a legitimate and important SQL pattern, but they prevent incremental refresh in MVs. If you need incremental eligibility, you have options: if the join always produces a match (no NULLs in the result), rewriting as INNER JOIN restores incremental refresh with no change to output. If the outer join is genuinely needed, consider handling the unmatched rows upstream. For example, pre-joining in an ETL step, or using a regular view for the outer join while keeping the MV incremental on the inner-joined subset.

For the full list of incremental refresh limitations, refer to Refreshing a materialized view in the Amazon Redshift documentation.

Refresh cost

Every refresh consumes compute resources. The pct_of_total_query_time column shows what share of total query time is spent refreshing each MV, and avg_refresh_sec identifies individual MVs with unusually long refresh times. There is no universal threshold, because what counts as expensive depends on the cluster. Start with the MVs that rank highest on pct_of_total_query_time: those in the highest decile of refresh cost, and those whose cost looks large relative to the queries they serve.

The net-negative MV. An MV that refreshes frequently but is never queried consumes resources without delivering value: refresh_count > 0 with query_count = 0. If is_autorefresh = true and query_count = 0, consider dropping the MV. If it’s still needed occasionally, lengthen its autorefresh interval or switch it to manual refresh on demand.

Refresh-to-query ratio. Even when queried, the refresh cost may exceed the benefit. If an MV fully recomputes many times for every query that reads it, the caching isn’t paying for itself. The recompute_to_query_ratio column flags this. Consider lengthening the autorefresh interval or scheduling refreshes to match how often the MV is read, or dropping the MV entirely.

Note: Amazon Redshift supports creating an MV on a consumer warehouse over data-shared base tables or directly refreshing an MV from a remote warehouse. This isolates the compute and minimizes cluster disruption, whether the refresh is full or incremental.

Dependencies and staleness

Amazon Redshift supports MVs that reference other MVs, known as nested MVs. Nested MVs can be refreshed together, in dependency order, using REFRESH ... CASCADE (refer to Harnessing the power of nested materialized views and exploring cascading refresh). The chain_depth column shows how deep each MV sits in a dependency tree, and mv_dependencies lists its parent MVs.

Staleness propagates one hop at a time. The is_stale column reports whether an MV’s data is behind its base tables. When base table data changes, the immediate dependent MV is marked is_stale = true. Downstream MVs remain is_stale = false until their direct parent is refreshed. At that point they become stale in turn. For deeper chains, is_stale = false on a downstream MV indicates freshness relative to its immediate parent, not the original base table. The stale_ancestors column walks the full chain and reports which parent MVs are currently stale.

CASCADE refresh solves propagation but requires explicit invocation. REFRESH MATERIALIZED VIEW mv_name CASCADE refreshes the entire chain in topological order (parent MVs first). It is not automatic and autorefresh does not cascade, so run it at your required freshness interval, for example on a schedule with Amazon EventBridge and the Amazon Redshift Data API, or as the final step of the ETL job that loads the base tables. Refer to the Autorefresh section for details on when AUTO REFRESH YES is supported.

Note: CASCADE is supported only for chains built on local tables, data-shared tables, local MVs, and streaming MVs. It doesn’t extend to MVs on Amazon Redshift Spectrum sources or to MVs shared from another warehouse, which must be refreshed individually in dependency order. Each additional level in a dependency chain adds refresh latency and widens the window where downstream MVs serve data that hasn’t caught up. Keep chains shallow, and weigh that lag against the benefit the chain provides.

Refreshing materialized views on external tables

MVs can cache results from external tables stored in Amazon S3, so each query reads the stored result instead of re-scanning the same Amazon S3 files. Refresh behavior depends on two factors:

  1. Data storage format: a standard file type (Parquet, CSV, Avro) or an open table format (Apache Iceberg, Hudi, Delta Lake). When new files appear in Amazon S3, MVs on standard file types and Iceberg refresh incrementally, whether or not they aggregate. Hudi and Delta Lake don’t support incremental refresh.
  2. Aggregation: whether the MV aggregates (GROUP BY, SUM, COUNT). When a file is deleted or overwritten, a non-aggregating MV stays incremental, but an aggregating MV must fully recompute, because aggregates can’t be undone without re-reading the remaining data. The last_refresh_status column reports this as “A full recompute was successfully completed as a data file was deleted.”

Apache Iceberg supports incremental refresh in more cases than the other formats. Iceberg’s metadata layer tracks row-level changes (inserts, deletes, updates, compaction), so aggregating MVs stay incremental even when rows are removed. Iceberg is the only data lake format that accepts AUTO REFRESH YES. For MVs over data lake sources where incremental refresh matters, Iceberg is the strongest fit (refer to materialized views on external data lake tables). For a deeper treatment of how incremental refresh works across data lake table formats, refer to Incremental refresh for Amazon Redshift materialized views on data lake tables.

Note: An incrementally refreshable materialized view on Iceberg tables may fall back to a full recompute if that is expected to perform better, or if it joins multiple base tables and more than one of them has been updated since the last refresh.

The following table summarizes the refresh method each source format uses for file additions and deletions, and whether it supports autorefresh:

Source format On file addition On file deletion, non-aggregating MV On file deletion, aggregating MV Autorefresh
Apache Iceberg Incremental Incremental Incremental Supported
Parquet / CSV / Avro Incremental Incremental Full recompute Not supported
Hudi / Delta Lake Full recompute Full recompute Full recompute Not supported

Note: On Iceberg tables, an aggregating MV falls back to a full recompute if a snapshot it needs has been expired. Expire snapshots only after refreshing the MVs that depend on the table, and if you use AWS Glue Data Catalog snapshot retention, keep the retention period longer than the interval between MV refreshes. Refresh after compaction stays incremental but processes more data, so plan for a longer refresh.

On Amazon Redshift RG instances and Serverless, external table queries run through an integrated query engine rather than Amazon Redshift Spectrum, which can reduce MV refresh times on Iceberg data. No configuration change is required.

Autorefresh

Autorefresh keeps MVs current without manual scheduling. The is_autorefresh column shows which MVs have it turned on, and autorefresh_interval_sec shows how often Amazon Redshift checks each one for staleness. An autorefresh runs only when the MV is stale, with the same priority as other user queries. On provisioned clusters, autorefresh queries run in your workload management (WLM) queues alongside your workload and appear in STL_QUERY. On Amazon Redshift Serverless, they consume Redshift Processing Units (RPUs) like any other query. On both, SYS_MV_REFRESH_HISTORY is where you see autorefresh activity, because it doesn’t appear in SYS_QUERY_HISTORY.

Autorefresh on high-churn tables generates constant refresh activity. Some base tables change frequently through streaming ingestion, AWS Database Migration Service (AWS DMS) replication, or zero-ETL. On those tables the MV is almost always stale, so autorefresh runs at every interval (5 minutes by default). The refresh_count column quantifies this. The worst case is autorefresh on a state 0 MV. Every refresh is a full recompute, so at the default interval that’s a full rebuild of the MV every 5 minutes. If the MV doesn’t need to be that current, keep autorefresh on but lengthen the interval (minimum 60 seconds) to match how often the results are queried: ALTER MATERIALIZED VIEW mv_name AUTO REFRESH YES INTERVAL <seconds>. To change the default for every MV on the cluster or workgroup, use ALTER SYSTEM SET default_mv_auto_refresh_interval = <seconds>. For details, refer to ALTER MATERIALIZED VIEW and ALTER SYSTEM. Custom intervals aren’t supported on streaming MVs.

Autorefresh does not cascade. It refreshes only the configured MV, not its dependents. AUTO REFRESH YES is rejected at creation time on nested MVs and on external tables other than Iceberg, returning an error rather than being silently ignored. On a nested MV the message is “Auto-refresh is not supported for materialized views defined on other materialized views.” It is supported on local single-level MVs over Iceberg or datashare tables, but not on an MV that combines both.

Broken materialized views

An MV becomes permanently broken when the definition of its base table changes in certain ways. The state column shows values 101-105 for broken MVs, with a label indicating the cause (refer to SVV_MV_INFO for the full state code reference).

Renaming an object that an MV depends on is not blocked, and it leaves the MV unable to refresh. A rename of a column, table, or schema completes without an error and moves the MVs that reference it to a broken state (103-105). Once broken, an MV cannot be refreshed, and renaming the object back does not restore it. The fix is to DROP and CREATE the MV.

Dropping a column that an MV uses, or changing its data type, is rejected. Amazon Redshift returns an error and the MV stays healthy.

Identify affected MVs before you alter a base table or column. SVV_MV_DEPENDENCY only shows MV-on-MV relationships, not base-table dependents. Before you make such a change, use the v_view_table_column_dependency view in the amazon-redshift-utils GitHub repository to identify the dependent MVs and the specific columns they rely on.

Automatic query rewriting

Amazon Redshift can rewrite a query on base tables to read from an MV, even when the query doesn’t name it (refer to Automatic query rewriting to use materialized views). The query_count column includes these reads. The is_autorewrite_eligible column shows whether the MV’s definition allows rewriting.

Eligible doesn’t mean used. Amazon Redshift skips a stale MV until its next refresh, even though the flag stays true. The flag also reads true for nested and broken MVs, which aren’t used. To confirm a specific query was rewritten, run EXPLAIN on it. In a rewritten plan, the base tables the MV covers no longer appear. In their place is a scan of an internal table that holds the MV’s results.

Should this be a materialized view?

Not every cached result benefits from being an MV. The following characteristics indicate when an MV is a strong fit and when an alternative is likely better:

Favors a materialized view Points to an alternative
Analytical queries from dashboards and analysts Application lookups serving thousands of concurrent requests
Multiple users or queries read the result The result is read once or rarely
The query scans a large table but returns few rows The query returns most of the rows it scans
Query latency is unacceptable without pre-computation A regular view already meets the latency requirement
Readers can tolerate data that is a refresh interval old Readers need sub-minute freshness

When your workload matches the right-hand column, a different approach usually fits better: a regular view when the query is already fast enough, in-stream transformation when readers need sub-minute freshness, and an in-memory cache such as Amazon ElastiCache for application lookups at high concurrency.

Putting it together

Deploy the diagnostic view and stored procedure. Run them periodically. Weekly is sufficient for most clusters. Focus on:

  1. State 0 MVs with high refresh cost: Check incremental_blocker_hint. If it names an SQL construct, refactor it (for example, OUTER JOIN to INNER JOIN). If it reads unidentified, confirm the full recompute is justified.
  2. MVs with query_count = 0: Consider dropping them.
  3. Deep chains with stale ancestors: Refresh with REFRESH ... CASCADE to bring the chain current, and flatten chains that are deeper than they need to be.
  4. Broken MVs (state 101-105): Drop and recreate.
  5. Autorefresh MVs on high-churn tables: Compare refresh_count against query_count. When refreshes far outnumber reads, lengthen the autorefresh interval to match how often the results are actually queried.

The goal is MVs whose refresh cost is repaid by the queries they accelerate.

Automate the review. Rather than running the diagnostic by hand, schedule it. v_mv_health is consistently current because it reads the system views each time you query it, so only the procedure needs a schedule. For a basic setup, run CALL sp_populate_mv_definitions(); on a cadence with Amazon EventBridge and the Amazon Redshift Data API. For alerting or dashboards, refer to Building an event-driven application with AWS Lambda and the Amazon Redshift Data API, which uses Amazon EventBridge and AWS Lambda to run a stored procedure on a schedule.

Cleaning up

The diagnostic objects are lightweight (a view, a stored procedure, and a small definitions table), but if you no longer need them, remove them with:

DROP VIEW IF EXISTS v_mv_health;
DROP PROCEDURE sp_populate_mv_definitions();
DROP TABLE IF EXISTS mv_definitions;

Summary

In this post, we showed how to check the health of the MVs in an Amazon Redshift database from a single result set. The v_mv_health view and the sp_populate_mv_definitions() procedure surface the signals that matter. Together they cover refresh eligibility, cost, staleness, dependencies, broken states, and autorefresh behavior. Call the procedure, join its output to the view, and you can see which MVs need attention first.

With v_mv_health and sp_populate_mv_definitions() deployed in each database that contains MVs, you can make MV health a routine part of your operational reviews.

Next steps. The v_mv_health and v_view_table_column_dependency views are available in the amazon-redshift-utils GitHub repository, and the sp_populate_mv_definitions() procedure is available in the amazon-redshift-udfs GitHub repository. To go deeper on the topics this post touched, see Harnessing the power of nested materialized views and exploring cascading refresh and Incremental refresh for Amazon Redshift materialized views on data lake tables.

 


About the author

Nick Barclay

Nick Barclay

Nick is a Senior Specialist Solutions Architect for Data and AI at AWS, working with public sector customers. He has spent his career building data and analytics solutions across Australia, New Zealand, and the United States, and now helps AWS customers design and optimize their data platforms. When he’s not working, you’ll find him swimming off one of the beaches on Australia’s Gold Coast.

Handling large volume of transactions with AWS Lambda Managed Instances

Post Syndicated from Mythili Annamalai Sekar original https://aws.amazon.com/blogs/compute/handling-large-volume-of-transactions-with-aws-lambda-managed-instances/

Pre-scheduled events like concert ticket on-sales and product launches can drive a load spike 100 times higher than the baseline. The spike persists for a few hours while the inventory clears, then traffic returns to normal. If you miss the first minute of the on-sales, customers experience errors while competitors keep selling, and the event’s reputation takes a hit. Over-provision for the burst, and you pay for peak capacity 24 hours a day, 365 days a year. The actual peak lasts 10 minutes and happens 10 times a year.

In this post, you will see how AWS Lambda Managed Instances, a capability of AWS Lambda, handles predictable traffic bursts. It keeps compute capacity warm before the on-sales event opens and scales it back down afterward. You will see a reference ticketing implementation that sustains 40,000 transactions per second (TPS) for 120 seconds.

Understanding Lambda Managed Instances

Lambda Managed Instances runs your Lambda functions on Amazon Elastic Compute Cloud (EC2) instances in your account, including Graviton4 and memory-optimized instance types. AWS handles the underlying infrastructure lifecycle (provisioning, scaling, patching, and routing), while you benefit from EC2 pricing including Compute Savings Plans and Reserved Instances.

Lambda Managed Instances provides several benefits. With pre-provisioned execution environments (EE), capacity is already warm before traffic arrives, so there are no cold starts when a burst hits. Scaling bounds are configurable with MinExecutionEnvironments and MaxExecutionEnvironments for each function, and each new EE comes online in 1–2 seconds. With multi-concurrent invocations, one EE handles multiple requests at once, which improves utilization for I/O-heavy handlers. You also get EC2 pricing, so Compute Savings Plans and Reserved Instances apply instead of per-request Lambda pricing.

Lambda Managed Instances is well suited for predictable, spike-shaped workloads that need sustained compute capacity during the peak and can scale back down when traffic returns to the baseline.

Predictable burst workloads work best with Lambda Managed Instances

This post focuses on one of the key capabilities of Lambda Managed Instances: absorbing scheduled traffic bursts without provisioning for peak year-round. Some of the use cases where Lambda Managed Instances helps are ticket and event on-sales (concerts, sports, festivals), live streaming and voting events, and scheduled batch and report generation nightly or hourly jobs that consume steady compute during a defined window.

Use case: High-throughput ticket booking with Lambda Managed Instances

To demonstrate Lambda Managed Instances handling predictable traffic bursts, we built a concert ticketing backend. The solution implements the industry-standard queue-plus-hold-and-confirm booking flow, a pattern that ticketing platforms, airlines, and hotels use to prevent overselling during large traffic spikes. It separates user traffic management from the critical database transactions required to lock inventory.

The following diagram presents the detailed architecture of the solution we built.

Architecture of the ticketing solution: API Gateway routes to three Lambda Managed Instances handlers backed by ElastiCache and DynamoDB

Figure 1: Architecture of the concert ticketing solution across two AWS CloudFormation stacks

The solution deploys two AWS CloudFormation stacks in a single AWS Region. In the first stack, Amazon API Gateway routes public HTTPS traffic through three ticketing handlers hosted in Lambda Managed Instances on a shared capacity provider. Amazon ElastiCache Serverless for Valkey holds the atomic queue counter. Amazon DynamoDB holds durable per-seat state across a write-sharded inventory table (500 shards) and a bookings table. WarmThroughput is pre-provisioned to absorb the burst without waiting for adaptive capacity. A separate simulation stack drives synthetic load through a three-tier Lambda fan-out, reproducing the on-sales burst shape at any target TPS.

Public traffic and load-generation traffic take completely separate code paths, so the load generator’s own scaling behavior does not affect measurements on the ticketing stack. Each successful booking traverses three Lambda handlers. Handler queue-position assigns an atomic virtual queue number using an INCR on a single ElastiCache key. Handler inventory-lock places a time-bounded hold on a specific seat through a DynamoDB conditional UpdateItem, moving it from AVAILABLE to HELD with a time to live (TTL) of 180 seconds. Handler booking-confirm executes a single TransactWriteItems call that atomically transitions the seat from HELD to CONFIRMED and writes a durable booking record. Either both writes succeed or neither does.

All three handlers are stateless, because each request they receive is treated as an independent, standalone transaction. The mutable state lives in ElastiCache and DynamoDB, stored with safety features like conditional writes and atomic operations to avoid collisions, race conditions, and overwrites. This is why the multi-concurrent EE model of Lambda Managed Instances can serve thousands of simultaneous invocations per instance safely.

What makes this solution better with Lambda Managed Instances

With Lambda Managed Instances, you keep the same Lambda programming model, one function per handler, and the same deployment story, while Lambda Managed Instances handles the fleet. Raise MinExecutionEnvironments a few minutes before an on-sales opens, let it burst inside the Min to Max range while traffic is live, and scale back down when the window closes.

Deploy the application

For the full deployment instructions, refer to the README file in the GitHub repository after you have cloned it.

Prerequisites

Run the application

Once deployed, all test traffic goes through the simulation API. The sample includes an optional single-page web user interface (UI) (see the demo-ui/ folder in the repository) that wraps the same endpoints and renders the per-burst results interactively.

Read the “Cost considerations” section before running a simulation to understand the cost you will incur in the process.

1. Smoke test – low TPS to verify the pipeline end-to-end:

By default, deploy.sh provisions a cost-safe baseline of three warm EEs (Min=1, Max=3 per handler). The following smoke test works on this baseline.

curl -X POST https://<sim-api>/v1/simulation/start \
-H 'Content-Type: application/json' \
-d '{"run_id": "smoke-1", "target_tps": 50, "ramp_duration_seconds": 10, "sustain_duration_seconds": 60}'

2. Peak-load test

To reproduce the headline result, scale the Lambda handlers to the per-handler Min and Max values before firing the peak-load test. Refer to the Peak-load testing section of the GitHub repository before firing the burst. This also requires an EC2 vCPU and an API Gateway account-throttle quotas increase above the default.

3. View results

curl “https://<sim-api>/v1/dashboard?run_id=onsale-1”

The response includes total requests, total bookings written to DynamoDB, delivered peak TPS, and end-to-end latency percentiles.

Interactive demo UI

The sample repository ships an optional single-page web UI (in the demo-ui/ folder) that wraps the same simulation and dashboard endpoints as the preceding curl examples. Point it at your simulation API endpoint, then fire bursts and view results interactively without leaving the browser.

Configure and start the UI:

# In demo-ui/config.js, set SIMULATION_API_URL to your simulation endpoint.
cd demo-ui && python3 serve.py
# Serving on http://localhost:8000/

Fire a burst. Choose a target load (a preset button or a custom TPS value), set the sustain duration and ramp, then choose Start burst. The UI sends a POST request to /simulation/start and stores the returned run_id and event_id, so you can query DynamoDB directly if you want to drill deeper.

View results. After the burst completes, choose View results. The UI queries API Gateway metrics, DynamoDB, CloudWatch Logs Insights, and CloudWatch Lambda metrics in parallel to reconstruct the run: total requests received, durable bookings written, and a per-handler table showing EEs used, Lambda throttles, and p95/p99 latency.

Demo UI results panel showing 4,804,200 API Gateway requests and 1,750,886 durable bookings with per-handler latency

Figure 2: Demo UI results panel from a 40,000 TPS, 120-second run

The panel shows 4,804,200 API Gateway requests and 1,750,886 durable bookings, plus a per-handler table with the baseline EE pool (50/100/150 for queue-position, inventory-lock, and booking-confirm). During the burst, 36 additional EEs are pre-emptively warmed on booking-confirm, with zero Lambda throttles and p99 latencies of 4.7, 15.8, and 39.8 ms.

The identifiers panel at the top exposes the burst’s run_id (the partition key in the simulation metrics table) and event_id (used in both the inventory and bookings tables), so you can locate them directly in DynamoDB or CloudWatch Logs Insights for the specific burst you ran.

What we measured

Across a load range from 5,000 to 40,000 requests per second, the pre-warmed EE pool served every burst without cold-start impact. Key results from the headline 40,000 TPS × 120-second burst:

Metric Value
Delivered peak throughput 40,035 req/sec
Total HTTP requests 4,804,200
Durable bookings written to Amazon DynamoDB 1,750,886
End-to-end p50 / p99 latency 44 ms / 121 ms
AWS Lambda errors 0
AWS Lambda concurrency throttles 0
Amazon DynamoDB write throttles 0

The tested range spanned 5,000 to 40,000 requests per second, an 8× spread in delivered load. Across it, p99 latency stayed within a 3 ms band. Doubling sustained throughput from 20,000 to 40,000 requests per second held at exactly 121 ms, which confirms that the pre-warmed pool had substantial headroom throughout the tested range.

Observability

Lambda Managed Instances automatically publishes metrics to CloudWatch and structured logs to CloudWatch Logs, giving you visibility into function performance and capacity utilization without additional code. During a burst, the metrics that matter most are:

  • Duration percentiles per handler to confirm p99 latency stays flat as load rises.
  • ConcurrencyThrottles to detect if any handler hits its MaxExecutionEnvironments ceiling.
  • Errors for handler-side failures.
  • EE count (log-derived) to observe how quickly Lambda Managed Instances adds warm capacity above Min.

The sample repository includes a demo dashboard endpoint that queries CloudWatch metrics, CloudWatch Logs Insights, and DynamoDB in parallel to reconstruct a per-burst report, API Gateway request count, durable bookings landed, distinct EEs used, and per-handler latency percentiles. Reconciling the simulation-side counts against the AWS ground truth catches load-generator artifacts and confirms the system-under-test numbers.

Cost considerations

Lambda Managed Instances uses EC2 instance pricing plus a 15% management fee. Compute Savings Plans and Reserved Instances apply to the underlying instance cost.

The Lambda Managed Instances configuration runs a small always-on baseline and uses a scaling script to pre-warm to 50/100/150 EEs in the minutes leading up to each event, then scales back down to baseline afterward.

To simulate a peak load under controlled conditions, the fleet is pre-warmed for a 7-minute window (5-minute ramp and 2-minute burst) then scaled back down. The cost breakdown can be as follows.

Component Volume Rate Cost
API Gateway requests 4.8 M $3.50 per 1 M $16.80
DynamoDB writes 8.8 M WCU $1.25 per 1 M $11.00
Lambda Managed Instances compute (peak window) 75 × c7i.4xlarge × 7 min $0.821 / instance-hr $7.18
Lambda Managed Instances request charges 4.8 M $0.20 per 1 M $0.96
CloudWatch Logs (~2 GB) ingestion $0.50 per GB $1.00
Amazon ElastiCache Serverless for Valkey 4.8 M INCR operations ECPU-based $0.30
Total per burst ~$37.00

The estimated cost for simulating each burst is around $37.

Best practices

Based on experience building this solution, here are key recommendations:

  • Pre-warm before scheduled events. Raise MinExecutionEnvironments using an Amazon EventBridge Scheduler rule or a manual scaling script when you expect major spikes, giving Lambda Managed Instances enough time to scale the underlying instances. Restore the low baseline after the event.
  • Set MaxExecutionEnvironments aggressively. Lambda Managed Instances adds new EEs to your capacity provider. A high MaxExecutionEnvironments value costs nothing while idle but absorbs unexpected demand during the burst.
  • Write-shard hot DynamoDB tables. For workloads writing thousands of items per second to one logical entity (a single event, a single product SKU), spread writes across a synthetic sharded partition key. This sample uses 500 shards per event.
  • Use WarmThroughput on DynamoDB tables. Pre-provision physical partitions before the burst rather than waiting for adaptive capacity to expand during the first minute of the on-sales.
  • Keep hot atomic counters out of DynamoDB. Single-row atomic increments in DynamoDB cap at approximately 1,000 operations per second.
  • Make write endpoints idempotent. The sample uses a caller-provided booking_id on the confirmation endpoint so that retries produce the same booking record instead of duplicates.

Clean up

When you are done exploring the solution, remove the provisioned resources to avoid ongoing charges. For the full cleanup commands and exact steps, refer to the GitHub repository README file.

Conclusion

Lambda Managed Instances brings the same Lambda programming model to workloads that need sustained, pre-provisioned compute during predictable bursts: ticket on-sales, product launches, registration windows, and other scheduled surge patterns. The concert ticketing example shows how a queue-plus-hold-and-confirm pattern built on Lambda Managed Instances can sustain 40,000 requests per second while costing less than an always-on EC2 fleet sized for the same peak.

Explore the Lambda Managed Instances documentation and try building your own predictable-burst serverless application. You can find the complete code for this example in the GitHub repository.


About the authors

Mythili Annamalai Sekar

Mythili Annamalai Sekar

Mythili Sekar is a Solutions Architect at Amazon Web Services (AWS) working with Independent Software Vendors (ISVs), where she guides customers in designing secure, enterprise-scale architectures built on AWS best practices. Her work centers on serverless and generative AI, helping software companies turn new cloud capabilities into real, production-ready outcomes for their businesses.

Guy Haddad

Guy Haddad

Guy is a Senior Technical Account Manager at AWS where he assists customers with leveraging their AWS entitlements and adhering to architectural and operational best practices. He specializes in AI/ML and serverless technologies and has a deep background in software development, telecommunications and complex program deliveries.

Introducing the new AWS WAF security analytics dashboard

Post Syndicated from Praveen Hosur Narayana Gupta original https://aws.amazon.com/blogs/security/introducing-the-new-aws-waf-security-analytics-dashboard/

When blocked requests spike or a customer reports being locked out, a security team has to work out what is happening, where it is happening, and if AWS WAF is making the right call. Currently, that usually means opening each protection pack (web ACL) in turn, across domains, AWS Regions, and Amazon CloudFront distributions, or querying AWS WAF logs. In 2024, we introduced the AWS WAF traffic overview dashboard, which gives you a near real-time view of each protection pack (web ACL). Today, we’re introducing the AWS WAF security analytics dashboard: a single global page that you can use to investigate traffic across all of your protection packs (web ACLs), Regional resources, Amazon CloudFront distributions and then act on what you find. The dashboard can help you investigate and contain threats faster and reduce false positives that block real customers. You don’t need to build or maintain your own log analytics pipeline to do it.

In this post, we introduce the new dashboard and walk you through two use cases: investigating a traffic spike, and tuning your rules to reduce false positives.

Introduction to the security analytics dashboard

To open the dashboard, go to the AWS WAF console and choose Security analytics in the navigation pane. The dashboard shows request data across all protection packs (web ACLs) in your account. You can narrow it to specific protection packs (web ACLs) and set the time range. As shown in the following figure, at the top of the page, you can see total requests split by action (Allow, Block, CAPTCHA, and Challenge). The Requests chart shows how the traffic changes over time. Below that, traffic breakdown widgets show the top values for each dimension. The dashboard is built on full-fidelity request data, not samples.

Figure 1: The security analytics dashboard across all protection packs

Dimensions

You can filter, group, and break down traffic by dimensions that include, but aren’t limited to, the following:

  • Who’s sending traffic – Client IP, country, Autonomous System Number (ASN), user-agent, and JA4 TLS fingerprints
  • What they’re requesting – Host, path (URI), protected resource (Amazon CloudFront distribution, AWS Application Load Balancers, and others), Region, or protection pack (web ACL)
  • How AWS WAF responded – Action, rule triggered, labels, or response code

We’ll continue to add dimensions to the security analytics dashboard over time. The rest of this post refers to the preceding dimensions by the names used here. Two dimensions deserve a closer look:

  • Host – Many customers think about their applications by domain rather than by protection pack (web ACL). Grouping or filtering by Host will show you which of your domains are under attack or if a spike is hitting www, the API, or a staging site, even when one protection pack (web ACL) protects several domains or one domain spans several protection packs (web ACLs).
  • Labels – Labels are metadata that AWS WAF rules add to the requests they inspect. AWS Managed Rules for AWS WAF add them (for example, Bot Control labels for bot category, verification status, and signals such as non_browser_user_agent), and you can create rules to add custom labels. Because labels are recorded even when a rule only counts, filtering by a label shows you what a rule detected across all of your protection packs (web ACLs), without changing how your traffic is handled.

Controls

The dashboard is built around a few controls:

  • Filter requests – Combine conditions on any dimension, using Equals or Does not equal. Filters apply to the whole dashboard: the action split, the Requests chart, every widget, and the request list update to match. Your dashboard view is saved in your browser.
  • Group by – Split the Requests chart by any dimension, such as Host, Rule triggered, Country, or JA4, to see how traffic changes over time.
  • Show only, Exclude, and Block – Select any value in any widget to focus on it, remove it from view, or block it. For example, you can block a high-risk or unexpected IP address, or block traffic from a country, directly from the dashboard.
  • Requests and detections – Review the individual requests behind the charts. For each request, see HTTP request fields such as host, path, headers, user agent, and JA4 fingerprint. You can also see which detections and labels matched and whether each action was enforced or only counted . Previously, this level of detail meant working from a limited sample of requests or setting up and querying your own AWS WAF logs. Now it’s available directly on the dashboard.

Together, these controls help you answer questions such as:

  • What share of traffic across all your protection packs (web ACLs) and domains is being blocked, challenged, or allowed, and how is it changing?
  • Which rules are doing the most work, and are any of them blocking legitimate users?
  • Which IP addresses, ASNs, JA4 fingerprints, or countries are behind a spike, and what are they targeting?
  • How much of your traffic comes from bots, and are the bots verified?

Use case 1: Investigate a traffic spike

Suppose you receive an alert that blocked requests have increased. To investigate:

  1. Go to the Security analytics page in the AWS WAF console and select All protection packs.
  2. Assess the scope: The action split shows the overall block rate. Group the Requests chart by Host to see which domains are affected and check the Protection pack widget to see whether one application or several are involved. A campaign hitting your storefront, API, and media domains at the same time calls for a different response than a problem on a single endpoint.
  3. Identify the source: Group the Requests chart by Rule triggered to see which rules are mitigating the traffic and when they started firing. Then check the source and request dimensions. If one IP address sends a disproportionate share of requests, or many IP addresses share one JA4 fingerprint, the traffic likely originates from a single automated actor.

    Figure 2: Requests grouped by Rule triggered, showing which rules fired and when

  4. Confirm with request-level evidence: When you find an unexpected value, select it and choose Show only to filter the entire dashboard to it.
  5. Then confirm what you’re seeing in Requests and detections. Expanding a request shows all its details, including the HTTP request fields. For example, a curl user agent with no Accept-Language header, coming from a cloud hosting ASN, is likely automated traffic.

    Figure 3: Show only, exclude, or block it

After you confirm the source, choose a response that fits the traffic:

  1. One clear source: Choose Block this IP address directly from the dashboard. Use a geographic block only if you don’t expect legitimate customers from that country.
  2. Traffic spread across multiple IP addresses: Add a rate-based rule with a custom aggregation key such as JA4, or use AWS WAF Bot Control.
  3. Known bad actors and application-layer floods: Add the Amazon IP reputation list and the Anti-DDoS managed rule group.
  4. Faster detection time: Create Amazon CloudWatch alarms on AWS WAF metrics. For details, see Monitoring with Amazon CloudWatch.

When the change is in place, return to the dashboard to confirm that requests from that source now show the Block action.

Use case 2: Tune your rules and reduce false positives

Rules can fail in two ways: by blocking legitimate users, or by only counting traffic that they should stop. Filters and labels help you find both.

If you suspect that a rule is blocking a broad population, such as many distinct clients sending only a few requests each, enter Rule triggered = <rule name> in Filter requests. The dashboard shows only the requests that the rule acted on. Review the Country, Host, and Path (URI) breakdowns to understand who the blocked clients are and what they requested. If the blocked clients look like your typical customers, with common browsers, expected countries, and checkout or login paths, consider adding a scope-down statement to the rule or changing the action to Challenge.

Figure 4: The dashboard filtered to a single rule with Filter requests

Evaluate rules before you enforce them: When you evaluate a managed rule group in Count mode, filter by its labels to see exactly which requests each rule matched. For example, filter by a Bot Control label such as awswaf:managed:aws:bot-control:bot:unverified, then check Host, Path (URI), and User-Agent to see what those bots are after. Use Exclude on labels for known-good traffic, such as bot:verified search engine crawlers, to keep them out of your review.

On the Detections tab of Requests and detections, rows marked Counted, not enforced show what a rule would block. If the matches aren’t partners or monitoring tools, consider changing the rule action to Block.

Figure 5: Record details for a single request, including host, JA4 fingerprint, and protection pack

After you change a rule, compare the same time window on the dashboard to confirm that the rule now affects only the traffic you intended.

Conclusion

The traffic overview dashboard shows you what changed in each protection pack (web ACL). The security analytics dashboard helps you understand why it changed, and lets you act on it from the same page, across all of your protection packs (web ACLs), domains, Regions, and Amazon CloudFront distributions. Filters, grouping, and labels explain the traffic, and request-level evidence helps you act with more confidence. Together, they can help you investigate faster, reduce how often real users are blocked by mistake, and confirm that each rule changes affect the traffic you intend.

The security analytics dashboard is available at no extra cost in all AWS Regions where AWS WAF is offered, and shows up to 21 days of request data. To get started, open the AWS WAF console and choose Security analytics.

If you have feedback about this post, submit comments in the Comments section below.

Related reading

To learn more about visibility and bot management in AWS WAF, see the following resources:

If you have questions about this post, contact AWS Support or start a new thread on AWS re:Post.

Praveen Hosur Narayana Gupta

Praveen Hosur Narayana Gupta

Praveen is a Solutions Architect at AWS specializing in Edge, Networking, and Security services. He helps customers build secure, high-performance applications and protect them from Bots and DDoS attacks. Outside work, he’s an avid bird watcher and wildlife photographer.

Mohit Mysore

Mohit Mysore

Mohit is a Senior Technical Account Manager at AWS specializing in network and edge security. He helps customers protect their applications and strengthen their security posture at scale. Outside work, he enjoys traveling, watching soccer and Formula 1, and spending time with his family.

Kaustubh Phatak

Kaustubh Phatak

Kaustubh is a product leader specializing in AI/ML systems and enterprise security solutions. He has led cross-functional teams in deploying AI-powered products at scale, working closely with security architects and CISOs to address the intersection of AI innovation and cybersecurity risk. His work focuses on translating complex technical capabilities into business value, particularly in emerging technology domains where traditional frameworks don’t apply.

CVE-2026-123890: Critical Authentication Bypass in VMware vCenter

Post Syndicated from Rapid7 original https://www.rapid7.com/blog/post/etr-cve-2026-123890-critical-authentication-bypass-vmware-vcenter

Overview

On November 19, 2026, Broadcom published VMware Security Advisory VMSA-2026-0009, addressing five vulnerabilities in VMware products. The most severe of these, CVE-2026-123890, is an authentication bypass using an alternate path or channel (CWE-288) in the VMware Directory Service component of vCenter. Broadcom assigned the vulnerability a critical CVSSv3 score of 9.8.

A malicious actor with network access to an affected vCenter deployment may exploit CVE-2026-123890 without credentials to bypass authentication and gain unauthorized access to the system as the Administrator user. Mandiant privately reported the issue to Broadcom. Also, Mandiant/Google Threat Intelligence confirmed in-the-wild exploitation of CVE-2026-123890 by Chinese state-linked threat actor ARMORY CASTLE as early as September 2026.

CVE-2026-123890 is distinct from the vCenter authentication bypass vulnerability CVE-2026-59309 that Broadcom addressed earlier in 2026. The vendor FAQ states that all vCenter systems below the versions fixed by VMSA-2026-0009 require the new update, including systems that already received the earlier fix. Broadcom characterizes the vCenter update as an emergency change requiring prompt action.

Mitigation guidance

Vendor-provided updates are available for CVE-2026-123890. Broadcom lists no workaround, and the patches cannot be applied using vCenter Quick Patch. Organizations should prioritize upgrading vCenter on an emergency basis, outside of normal patch cycles, using the standard update mechanism for their deployment.

VMware Product

Affected Version

Fixed Version / Patch

VMware Cloud Foundation / vSphere Foundation

9.1.x.x

9.1.1.0200

VMware Cloud Foundation / vSphere Foundation

9.0.x.x

9.0.2.0200

VMware vCenter Server

8.0

8.0 U3m

VMware vCenter Server

7.0

Contact Broadcom Support (extended-support customers)

VMware Cloud Foundation

5.x

5.2.5

VMware Telco Cloud Platform

3.0, 4.x, 5.0.x, 5.1.x, 5.2

8.0 U3m — KB463116

VMware Telco Cloud Infrastructure

3.0

8.0 U3m — KB463116

The fixes are cumulative. The vCenter 8.0 U3m and 9.0.2.0200 updates introduce a “Back-in-Time” restriction that blocks upgrades to VMware Cloud Foundation 9.1.1; Broadcom advises customers in an active upgrade to review the vendor FAQ and interoperability matrix. Updating vCenter temporarily makes its management interfaces unavailable, but Broadcom states that running virtual machine and container workloads continue to operate.

For the latest affected-version and mitigation guidance, please refer to Broadcom’s security advisory and supplemental FAQ.

Rapid7 customers

Exposure Command, Vulnerability Management, and Nexpose

Exposure Command, InsightVM, and Nexpose customers can assess exposure to CVE-2026-123890 on VMware vCenter Server with an unauthenticated vulnerability check expected to be available in the Nov 20, 2026 content release.

Updates

  • November 19, 2026: Initial publication.

Introducing Amazon EMR Serverless Flex: Run non-urgent Spark jobs at a discount

Post Syndicated from Neil Mukerje original https://aws.amazon.com/blogs/big-data/introducing-amazon-emr-serverless-flex-run-non-urgent-spark-jobs-at-a-discount/

With Amazon EMR Serverless, you can run Apache Spark applications without provisioning or managing clusters. You submit jobs, EMR Serverless allocates the compute, scales it during execution, and releases it when the job finishes. There are no clusters to tune and no idle capacity to pay for. Since launch, customers have used EMR Serverless to run production extract, transform, and load (ETL) pipelines, machine learning (ML) feature engineering, and large-scale data transformations, replacing Spark clusters that required dedicated engineering effort to operate.

Customers also have workloads that aren’t time-sensitive. On EMR on EC2 or self-managed Spark clusters on EC2, teams use Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances to run these workloads on spare compute capacity at a lower cost. For top EMR customers, EC2 Spot Instances are a key cost-optimization lever, with many running a portion of their workloads on spare capacity. For customers looking to modernize from clusters to serverless to lower their operational overhead, the absence of an equivalent option on EMR Serverless meant they could only move part of their platform. The time-sensitive workloads came over, but the cost-optimized workloads running on spare capacity stayed on EC2 Spot Instances. Spare capacity support on EMR Serverless has been one of the most requested features, with customers telling us it’s a prerequisite for full migration.

Today we are launching Amazon EMR Serverless Flex, a new execution option that runs Spark batch jobs on spare compute capacity at a fixed 60 percent discount compared to standard pricing. With Flex, EMR Serverless now offers two execution options: Standard for workloads that need predictable start times and dedicated resources, and Flex for workloads where cost matters more than execution-time predictability. Customers who previously split their Spark platforms between EMR Serverless and EMR on EC2 or self-managed clusters on Spot Instances can now consolidate onto a single serverless platform. Customers who didn’t consider EMR Serverless at all, because spare capacity support was missing, can now evaluate EMR Serverless as a full replacement for their cluster-based deployments.

Why Flex matters for your Spark platform

When we talk to customers running Spark at scale, we see a common pattern: two tiers of workloads with different cost and reliability requirements.

Time-sensitive production workloads – ETL pipelines that feed dashboards, ML models, and downstream applications on a schedule. These need fast startup, dedicated compute, and predictable completion times. On EMR Serverless, these run on the Standard execution option.

Non-urgent batch workloads – Pre-production validation, test and development runs, backfills, nightly batch transformations, one-time data loads, and recurring reports with generous processing windows. These jobs are cost-sensitive but time-flexible. They need to finish, but not on a strict deadline.

On EMR on EC2 or self-managed Spark clusters on EC2, customers often deploy non-urgent workloads on Spot Instances. But without a spare-capacity option on EMR Serverless, moving to serverless meant giving up the cost lever on non-urgent batch workloads.

What it takes to run Spark on spare capacity today

On EMR on EC2, running Spark on Spot Instances is a shared responsibility between the service and the customer.

What the EMR service provides: EMR on EC2 provides the infrastructure features for running workloads on spare capacity:

  • Instance fleets with Spot allocation strategies (capacity-optimized) that spread capacity requests across instance types and Availability Zones.
  • Graceful decommissioning that gives Spark time to migrate shuffle data and finish running tasks when a Spot Instance receives a 2-minute termination notice.
  • Spark resilience enhancements (decommission-aware task scheduling, exclusion of decommissioning nodes, and shuffle block tracking) that help Spark recover from node loss without getting stuck on fetch failures.

What customers implement: Based on best practices that have evolved over the years, here’s what teams have built and maintained on top of the preceding EMR features:

  • Instance fleet diversification – Select the right mix of instance types across instance variants (m5, m5a, m5d, r5, r5a, r5d) that meet vCPU and memory requirements. Use the Spot Instance Advisor to target types with lower interruption rates. Configure allocation strategies and fallback behavior.
  • Multi-AZ placement – Launch clusters targeting multiple Availability Zones to tap into the capacity in the entire AWS Region.
  • Driver protection with YARN node labels – Configure YARN node labels to pin the Spark driver to On-Demand core nodes, keeping it off Spot task nodes. If the driver lands on a Spot Instance and that instance is reclaimed, the entire job fails. Setting this up requires YARN label configuration, node-label expression settings in Spark, and ongoing validation.
  • Configuring resilience and decommissioning settings – Tune decommissioning timeouts, understand how the Spark resilience enhancements work, and configure them correctly for each workload.
  • Ongoing tuning and debugging – Monitor interruption rates by instance type, adjust fleet composition as Spot pricing shifts, debug capacity-related job failures, and maintain the configuration as EMR release versions change.

What Flex handles for you

EMR Serverless Flex applies all of these spare-capacity best practices on your behalf. You add one parameter to your job submission with no fleet configuration or YARN tuning. EMR Serverless takes on the responsibility of running your workloads on spare capacity:

  • AZ selection – EMR Serverless evaluates spare capacity pools across Availability Zones in the AWS Region and places workers where capacity is most available.
  • Instance diversification – EMR Serverless spreads executors across multiple instance types and generations so that reclamation of one instance type doesn’t hit all workers at once.
  • Driver protection – The Spark driver runs on a dedicated Standard worker that isn’t subject to spare capacity reclamation. The driver holds job state (the directed acyclic graph, stage progress, and task assignments) so protecting it prevents job failure, without YARN node labels to configure.
  • Decommission-aware scheduling and worker exclusion – When a Flex worker receives a 2-minute reclamation notice, EMR Serverless stops scheduling new tasks on that worker and excludes it from future assignments. The Spark resilience enhancements that EMR developed for handling node loss (decommission-aware task scheduling, proactive worker exclusion, and shuffle block tracking) are applied automatically.
  • Automatic task retry – Spark’s built-in fault tolerance retries affected tasks on healthy Flex workers. The job continues without manual intervention.
  • Shuffle offload with Serverless Storage (optional) – With EMR Serverless Storage, shuffle data is written to remote storage instead of local disk. When a reclaimed executor’s tasks are retried, the new executor reads the stored upstream shuffle output instead of recomputing it, reducing the fetch-failure problem for shuffle-heavy jobs.

All existing EMR Serverless capabilities (AWS Lake Formation fine-grained access control, AWS Graviton support, AWS Glue Data Catalog integration, and data source connectors) work the same with Flex jobs.

What happens when you run a Flex job

Here is the life cycle of a Flex job from submission to completion:

  1. Job submission – You submit a Spark job with Flex enabled. EMR Serverless accepts the job and begins acquiring spare compute capacity.
  2. Capacity acquisition – EMR Serverless selects an Availability Zone within the chosen Region where spare capacity is most available and requests workers across a diverse set of instance types and generations. The Spark driver starts on a dedicated Standard worker.
  3. Job execution – Executors run on Flex workers. The job proceeds as it would on Standard using the same Spark runtime and APIs.
  4. If capacity is reclaimed – When a Flex worker receives a reclamation notice, EMR Serverless excludes it and stops scheduling new tasks on it. Spark retries the affected tasks on healthy Flex workers. If you have Serverless Storage enabled, retried tasks read stored shuffle data instead of recomputing upstream stages. Billing for the reclaimed worker stops at reclamation.
  5. Completion – The job completes and results are written to the output location.
  6. Failure handling – If no executor joins the driver within 20 minutes of job start, the job fails. To recover from failures, resubmit the job using the best practices you apply to restart a failed job.

How Flex workers are billed

Flex workers are billed at a fixed 60 percent discount on vCPU-hours and memory GB-hours compared to Standard workers. The Spark driver runs on a Standard worker and is billed at the Standard rate. The discount is fixed and unlike EC2 Spot doesn’t fluctuate with supply and demand.

Each worker is billed for the time it was part of the job run, per second with a 1-minute minimum. When a Flex worker is reclaimed, billing for that worker stops. For example, if a 10-minute job runs 20 Flex workers and 3 of them are reclaimed after 5 minutes, you are billed for 17 workers at 10 minutes and 3 workers at 5 minutes.

Flex spend appears as separate line items in AWS Cost Explorer and on your bill, so you can distinguish Flex spend from Standard spend. Flex also works with job-level cost allocation tags, so you can continue attributing costs by team or project. For pricing details, see Amazon EMR pricing.

Flex in practice: Resilience and cost

We ran a 103-query decision-support benchmark (derived from TPC-DS 1 TB and 3 TB scale factor) on Spark to compare Flex and Standard at two data scales. Both runs used the same queries, worker configuration, and release label (emr-7.14.0).

Scale factor Execution Queries completed Wall-clock time Estimated cost Total executors used Executors reclaimed Task failures
1 TB Standard 103 of 103 11.5 min $5.68 12 0 0
Flex 103 of 103 12.3 min $2.07 (64% savings) 15 1 0
3 TB Standard 103 of 103 12.4 min $6.29 12 0 0
Flex 103 of 103 16.1 min $2.32 (63% savings) 21 1 0

Resilience: In each Flex run, we observed one executor reclaimed mid-job. EMR Serverless excluded the reclaimed worker, and Spark retried the affected tasks on healthy executors. All 103 queries completed in both runs, with zero job resubmissions.

Cost: For this benchmark, we saw Flex reduce estimated cost by about 63 percent at both data scales, from $5.68 to $2.07 at 1 TB, and from $6.29 to $2.32 at 3 TB.

Note: The actual savings and frequency of reclamations will vary by workload, Region, and available spare capacity at the time of execution. We recommend running your own workloads on both Flex and Standard to establish a baseline. This workload is derived from the TPC-DS Benchmark and isn’t comparable to published TPC-DS Benchmark results, as this implementation doesn’t comply with all requirements of the TPC-DS Benchmark.

Runtime tradeoff – The 1 TB Flex run took 7 percent longer than Standard. The 3 TB run took 30 percent longer. The gap widened at larger scale because the reclaimed executor’s tasks needed more time to recompute. Flex jobs can also take longer to start if spare capacity isn’t immediately available. Flex works well for any job where the completion deadline is flexible, including short jobs. A job that runs in 10 minutes on Standard and 12 minutes on Flex is a good candidate if it doesn’t have a strict service level agreement (SLA).

Getting the best results from Flex

Three practices we recommend to get the most from Flex:

Avoid top-of-the-hour peaks

We observe that spare capacity availability drops when many jobs start simultaneously at the top of each hour. Shifting your job start times by even a few minutes (for example, submitting at 9:20 instead of 9:00) reduces contention for spare capacity and improves the chance of a fast start.

Run outside business hours when possible

Spare capacity is most available when overall compute demand is lower. We see the best availability when customers schedule non-urgent workloads to run after 6:00 PM in the Region, or on weekends, giving Flex jobs access to deeper capacity pools with fewer interruptions.

Use Serverless Storage

EMR Serverless Storage writes shuffle data to remote storage, so retried tasks read stored intermediate results instead of recomputing upstream stages. This reduces the runtime cost of interruptions on shuffle-heavy jobs and adds another layer of resilience.

Getting started

Flex is a job-run setting. You create your EMR Serverless application as usual and add one parameter when submitting a job:

aws emr-serverless start-job-run \
    --application-id <application-id> \
    --execution-role-arn <job-role-arn> \
    --flex-execution '{"enabled": true}' \
    --job-driver '{
        "sparkSubmit": {
            "entryPoint": "s3://<bucket>/scripts/my-spark-job.py"
        }
    }'

In EMR Studio, select the Enable Flex execution checkbox when submitting a job, as shown in the following screenshot:

EMR Studio job submission form with the Enable Flex execution checkbox selected

Figure 1: Enable Flex execution checkbox in the EMR Studio job submission form

You can run Flex and Standard jobs on the same application. For step-by-step instructions, see the Amazon EMR Serverless User Guide.

Conclusion

In this post, we shared how Amazon EMR Serverless Flex runs non-urgent Spark batch jobs on spare compute capacity with built-in resilience features, closing a gap that previously blocked customers from modernizing their entire Spark platforms onto EMR Serverless.

To get started, pick a non-critical job, submit it with --flex-execution '{"enabled": true}' and compare the cost and runtime against your Standard baseline. For more detail, see the Amazon EMR Serverless User Guide.


About the authors

Neil Mukerje

Neil Mukerje

Neil is a Principal Product Manager at Amazon Web Services. He is driven to build experiences that let customers achieve their goals efficiently. In his spare time, he enjoys reading, hiking, and tinkering with technology.

Manjeet Chayel

Manjeet Chayel

Manjeet serves as Big Data Manager, Worldwide Specialist Solutions Architects at AWS, where he leads a global team of specialist architects driving customer-facing engagements across Amazon EMR, AWS Glue, and the broader Big Data Analytics portfolio.

Announcing 30-minute timeout for on-demand AWS Lambda Event Functions

Post Syndicated from Tarun Rai Madan original https://aws.amazon.com/blogs/compute/announcing-30-minute-timeout-for-on-demand-aws-lambda-event-functions/

AWS Lambda now supports a 30-minute function timeout for Event Functions that run on demand, doubling the previous 15-minute limit for synchronous, asynchronous, and event source mapping (ESM) invocations. On Lambda Managed Instances, synchronous invocations also support up to 30 minutes, while asynchronous and ESM invocations keep the 90-minute timeout we announced in September 2026. You can now run AI inference, media transcoding, and financial calculations on Lambda for spiky or steady-state traffic without re-architecting your applications. The new timeout also applies when you use Lambda durable functions or invoke your functions from AWS Step Functions Standard workflows.

Evolution of function timeout on Lambda

Lambda launched in 2014 with a 60-second function timeout, which we extended to 5 minutes in 2015 and to 15 minutes in 2018.

Lambda now offers three form factors: Event Functions, Web Functions, and MicroVMs. Event Functions process events from more than 200 AWS event sources through synchronous, asynchronous, and ESM invocations. With Web Functions, you can run HTTP servers on Lambda with an HTTP-native, multi-concurrent programming model and opt-in multi-Region endpoints. MicroVMs give each user or session an isolated environment for running user-generated or AI-generated code.

By default, all three form factors run on demand, with Lambda provisioning compute as requests arrive. For steady-state traffic, Event Functions can also run on the Lambda Managed Instances capacity provider. With Lambda Managed Instances, you can process multiple concurrent requests per instance and choose specialized compute configurations, without managing infrastructure. You can also use EC2 pricing options such as AWS Savings Plans and Reserved Instances. Figure 1 shows the form factors.

The three Lambda form factors: Event Functions, Web Functions, and MicroVMs

Figure 1: Lambda form factors

Each form factor has its own duration limit (Table 1). Until today, Event Functions had a 15-minute timeout, except for asynchronous and ESM invocations on Lambda Managed Instances, which we extended to 90 minutes in September 2026.

Lambda durable functions run on demand and on Lambda Managed Instances, and provide application-level checkpointing and workflow-as-code. Your code saves execution state with step() and wait() operations and resumes from the last checkpoint after a failure. A durable execution (the complete lifecycle of a durable function) can run for up to 1 year, but each invocation within it is bound by the function timeout.

Customers told us that 15 minutes was still too short for many Event Functions workloads that run on demand, and for synchronous invocations on Lambda Managed Instances. Agentic AI requests that chain model calls and tool invocations can take longer than 15 minutes to return a result. Media transcoding and file conversion jobs often run past 15 minutes when input files are large. Financial calculations such as Monte Carlo simulations, bond pricing, and portfolio risk analysis are memory-intensive and need more continuous execution time. Report generation across multiple data sources can also exceed 15 minutes during peak volumes. ETL steps triggered by Amazon Simple Queue Service (Amazon SQS) or Amazon Simple Storage Service (Amazon S3) events can run just as long. Many of these workloads have spiky traffic that doesn’t justify steady-state capacity on Lambda Managed Instances.

With today’s launch, Event Functions support up to 30 minutes for synchronous, asynchronous, and ESM invocations, and up to 90 minutes for asynchronous and ESM invocations on Lambda Managed Instances. Table 1 shows the maximum durations across Lambda form factors.

Duration Event Functions Web Functions MicroVMs
On demand (default) Lambda Managed Instances
Without durable execution

Synchronous: 30 minutes

Asynchronous: 30 minutes

ESM: 30 minutes

Synchronous: 30 minutes

Asynchronous: 90 minutes

ESM: 90 minutes

15 minutes 8 hours (running and suspended)
With durable execution

Synchronous: 30 minutes

Asynchronous: 1 year

ESM: 30 minutes

Synchronous: 30 minutes

Asynchronous: 1 year

ESM: 90 minutes

– –

Table 1: Supported durations on Lambda form factors

Configuring the 30-minute function timeout

You can configure any Event Function with a timeout of up to 30 minutes (1,800 seconds). For asynchronous and ESM invocations on Lambda Managed Instances, you can configure up to 90 minutes (5,400 seconds). Lambda rejects higher values with a validation error. Your function runs exactly as before, with the same runtime, handler, IAM execution role, and virtual private cloud (VPC) configuration. The Init phase keeps its existing limits of 10 seconds for functions that run on demand and 15 minutes on Lambda Managed Instances. Provisioned concurrency and Lambda SnapStart, which aren’t available on Lambda Managed Instances, keep their 15-minute initialization limit.

To set the timeout, update your function configuration using the AWS Command Line Interface (AWS CLI):

aws lambda update-function-configuration \
--function-name my-demo-function \
--timeout 1800 \
--region us-east-1

To set the timeout in an AWS Serverless Application Model (AWS SAM) template, use the Timeout property:

MyFunction:
  Type: AWS::Serverless::Function
  Properties:
    FunctionName: my-demo-function
    Runtime: python3.14
    Handler: app.handler
    Timeout: 1800
    MemorySize: 2048

You can also update the timeout in the Lambda console under Configuration > General configuration (Figure 2), where synchronous test invocations now run for up to 30 minutes. If you use an AI coding assistant such as Kiro, Claude Code, or Codex, you can install the Agent Toolkit for AWS and configure the timeout with natural language prompts.

Lambda console General configuration page with the function timeout set to 30 minutes

Figure 2: Configuring Lambda function timeout

The new timeout applies to invocations that start after you update the function. For event source mappings, allow a few minutes for the change to propagate. Amazon CloudWatch metrics, AWS CloudTrail, and AWS X-Ray continue to capture the full invocation lifecycle. For details, see monitoring Lambda functions and monitoring durable functions.

For Event Functions that run on demand, the 30-minute timeout is available in all commercial AWS Regions. For synchronous invocations on Lambda Managed Instances, it is available in all AWS Regions where Lambda Managed Instances is available. Standard AWS Lambda pricing applies.

30-minute timeout and durable functions

The 30-minute function timeout and durable functions are complementary. The function timeout (--timeout) controls how long each individual invocation can run, whereas the durable execution timeout (ExecutionTimeout in --durable-config) controls the total elapsed time from execution start to completion. On replay after a failure, the function re-executes from the beginning and skips completed steps.

Functions that run on demand: Each durable function invocation can run for up to 30 minutes, whether you invoke it synchronously, asynchronously, or through an event source mapping. When you invoke the function asynchronously, the durable execution can run for up to 1 year. When you invoke it synchronously through the Invoke API or a function URL, the durable execution is limited to 30 minutes, and Lambda rejects higher values. ESM invocations have the same 30-minute limit.

Lambda Managed Instances: Synchronous invocations can run for up to 30 minutes, and asynchronous and ESM invocations can run for up to 90 minutes. When you invoke the function asynchronously, the durable execution can run for up to 1 year. The durable execution is limited to 30 minutes for synchronous invocations and 90 minutes for ESM invocations.

For idempotent jobs, such as an ETL step triggered by Amazon SQS, the extended timeout alone might be enough. A failed invocation returns the message to the queue for a fresh attempt. For jobs where re-execution is expensive, combine the extended timeout with durable functions. A 25-minute inference run that fails at minute 22 restarts from zero without checkpoints, but resumes from its last checkpoint with durable functions.

Invocation behavior: Synchronous, asynchronous, and event source mappings

Synchronous invocations (up to 30 minutes): Synchronous invocations now support up to 30 minutes, whether the function runs on demand or on Lambda Managed Instances. If another AWS service invokes your function synchronously, that service applies its own timeout, so verify that it supports the longer duration before you raise the function timeout. Set your client’s read and socket timeouts longer than the function timeout. For example, increase the read timeout of your AWS SDK client when you call the Invoke API. On Lambda Managed Instances, Lambda applies the 30-minute limit to synchronous invocations even if the function timeout is higher, and the GetFunctionConfiguration API reports the configured value.

Asynchronous invocations (up to 30 minutes, or up to 90 minutes on Lambda Managed Instances): If the function fails or times out, Lambda applies your configured retry policy (up to two retries by default). It then routes failed events to your dead-letter queue or on-failure destination.

Event source mappings (up to 30 minutes, or up to 90 minutes on Lambda Managed Instances): For SQS, set the queue’s visibility timeout to at least six times the function timeout, so at least 3 hours for a 30-minute function. This gives Lambda enough time to retry if a function is throttled while it processes a previous batch. Lambda validates this when you create the event source mapping, but it doesn’t prevent later changes to queue or function settings that create a mismatch.

For Amazon Kinesis Data Streams and Amazon DynamoDB Streams, Lambda processes one batch per shard at a time by default, so longer invocations slow how quickly each shard is read. Monitor the IteratorAge metric and increase the parallelization factor if records start to back up. Longer invocations also mean that a failure late in a batch wastes more work. Enable partial batch failure reporting so that Lambda retries only the failed records. You can use it with SQS, Amazon Kinesis, DynamoDB Streams, Amazon Managed Streaming for Apache Kafka (Amazon MSK), and self-managed Apache Kafka event source mappings.

Invocations from Amazon MQ (ActiveMQ and RabbitMQ) and Amazon DocumentDB (with MongoDB compatibility) event source mappings remain limited to 14 minutes. The same limit applies to Kafka event source mappings that were created before November 19, 2026, and don’t use provisioned mode. To use the 30-minute timeout with Kafka, create a new event source mapping or turn on provisioned mode for your existing one. For these mappings, keep the function timeout at 14 minutes (840 seconds) or less.

Considerations and best practices

Longer-running functions need extra care with concurrency and components that can expire mid-invocation, such as network connections and credentials.

Concurrency: Concurrency equals requests per second multiplied by average duration in seconds. At the same request rate, a function that runs for 30 minutes instead of 15 needs twice the concurrency. For example, 1 request per second at 30 minutes each needs 1,800 concurrent executions. Check your account concurrency quota and reserved concurrency settings before you raise the timeout. On Lambda Managed Instances, check the maximum scaling limits of your capacity provider.

Networking: Make sure idle connection timeouts on downstream services such as Amazon Relational Database Service (Amazon RDS), Amazon ElastiCache, and external APIs accommodate the full function duration. If your function routes traffic through a NAT gateway, which drops idle connections after 350 seconds, send keep-alive packets. For long synchronous invocations, also turn on TCP keep-alive on the calling client. DNS caching depends on your runtime and HTTP client. For example, the AWS SDK for Java relies on the JVM DNS cache, which on some configurations never refreshes. Set a short DNS TTL, such as 5 seconds for the JVM, and configure custom HTTP clients to re-resolve hostnames.

Credentials: If your function acquires temporary credentials or tokens, verify they remain valid for the full execution duration or refresh them in the background.

Idempotency: Lambda does not guarantee exactly-once processing, and longer-running functions widen the window for retries and duplicate deliveries. Use Powertools for AWS Lambda to make operations such as payments or database writes idempotent. In durable functions, steps have at-least-once execution semantics by default. The SDK skips completed steps during replay, but steps that fail before checkpointing might re-execute. You can use execution names as idempotency keys for durable functions.

Conclusion

The 30-minute function timeout brings longer continuous execution to Event Functions that run on demand and to synchronous invocations on Lambda Managed Instances. It complements the 90-minute timeout for asynchronous and ESM invocations on Lambda Managed Instances that we launched in September 2026. Workloads such as AI inference, media transcoding, and report generation that exceed 15 minutes can now run on Lambda without architectural workarounds, for spiky or steady-state traffic. If your workloads need a longer timeout, let us know on the AWS Lambda Roadmap GitHub page.

To get started, update your function’s timeout configuration and deploy. For details, see Configure Lambda function timeout in the AWS Lambda Developer Guide. For sample code that shows long-running functions with durable checkpointing, see durable functions examples. To learn more, visit the AWS Lambda product page.

 


About the authors

Tarun Rai Madan

Tarun Rai Madan

Tarun is a Principal Product Manager at Amazon Web Services (AWS), where he leads product areas across AWS Serverless Compute, including developer and agentic experiences, event-driven data ingestion, orchestration, and compute for data-intensive workloads. His work centers on removing infrastructure complexity so developers can concentrate on their differentiated business logic. Before joining AWS, he was an engineering leader in the semiconductor industry, where he developed high-performance processors for data center, wireless, and automotive applications.

Dan Slusariuc

Dan Slusariuc

Dan is a Software Development Manager at AWS, where he leads the Lambda Frontend team. His team builds the backend services that receive and route Lambda function invocations. He also leads compliance and availability for Lambda. In his prior role, Dan was a Software Development Engineer in Amazon Advertising, where he helped build the engine that generates banners for products listed on Amazon.

Snehil Ameta

Snehil Ameta

Snehil is a Software Development Engineer at AWS, where he works on AWS Lambda. He focuses on Lambda scalability and availability, and leads capabilities such as Lambda response streaming and Lambda Managed Instances. In his prior role, Snehil worked in Amazon seller services, where he helped build its recommendation service.

How AWS Security Incident Response triages your security findings

Post Syndicated from Anatoliy Kochetov original https://aws.amazon.com/blogs/security/how-aws-security-incident-response-triages-your-security-findings/

AWS Security Incident Response continuously triages the threat findings from Amazon GuardDuty and AWS Security Hub CSPM, but most of that work never shows up as a case. If cases are all you look at, it’s hard to tell whether the service is working at all, and that’s the question I hear most often when I help customers onboard. The Proactive response dashboard gives you a high-level view of this activity, and this post takes you behind the numbers to show what happens to an individual finding.

In this post, I walk you through what happens to a finding after it’s ingested by the service, what gets triaged, where each finding is sent, and how a triage decision becomes a case. I then show you how to confirm triage is working day-to-day, so your team can focus on the findings that require attention. The process flow is shown in Figure 1.

Figure 1: AWS Security Incident Response triage flow, from ingestion through to one of three outcomes

Figure 1: AWS Security Incident Response triage flow, from ingestion through to one of three outcomes

Ingestion

AWS Security Incident Response ingests threat findings from GuardDuty through Amazon EventBridge rules that are automatically deployed to your accounts during the onboarding process.

If you send third-party detection findings to Security Hub CSPM from tools such as CrowdStrike, Trend Micro, or Lacework, their threat detection findings are also ingested. Each of those findings must carry complete payload information such as resource ID, account ID, timestamp, and threat details. GuardDuty findings are ingested directly, so these findings aren’t re-ingested through Security Hub CSPM and there’s no duplication.

Triage

The service triages findings that indicate an active threat, such as compromised credentials, cryptomining, or data exfiltration. These represent activity that’s underway or has already occurred.

Posture and compliance findings aren’t triaged. A misconfiguration or a failed benchmark check reflects the state of your environment rather than an active threat, so those findings remain for your team to address through your existing remediation process.

Outcomes

For each threat finding, the service runs an automated triage process. It evaluates the finding payload, related AWS activity from AWS CloudTrail and Amazon VPC Flow Logs, AWS threat intelligence, and the context you provide about your environment. The service then maps the finding against a profile of known-good activity. What fits the profile is treated as expected, and what doesn’t is flagged for a closer look.

What happens next depends on that outcome. A finding is sent to one of three places:

  • Archived, no action needed – Automated triage or an AWS Security Incident Response engineer confirms the activity is benign or expected. The finding is archived in GuardDuty, with no case and no customer contact.
  • A proactive case is opened – Triage can’t confirm the activity is benign. This is either Unconfirmed Activity (the service needs you to confirm whether it’s expected) or Inappropriate/Unintended Activity (confirmed activity that needs your action). A proactive case is opened, your incident response team is engaged, and the finding stays Current until it’s resolved.
  • Retained for your review, no case – Posture-related findings, such as overly permissive policies, configuration, or missing encryption, aren’t threats. By design, they stay Current without archiving and no case is created.

When a proactive case is opened, you respond to it in the AWS Management Console for AWS Security Incident Response under Cases, where you can communicate directly with the team, share context, and resolve or reopen the case. Timely responses also help the service learn your environment, which helps reduce false positives over time. For the full workflow, see Responding to an AWS generated case.

The preceding outcomes use GuardDuty terms, because it’s the primary source. Third-party findings ingested through Security Hub CSPM follow the same triage logic, but their disposition shows as a workflow-status change rather than archiving. The finding moves from NEW to NOTIFIED when triage begins, which removes it from the default Security Hub CSPM view. If triage clears it as benign, it moves to SUPPRESSED. Otherwise it stays visible for your review.

Because only benign findings are archived, it’s normal to see many findings in Current, and that isn’t a sign the service missed them. Across customers, less than 1 percent of ingested findings result in a customer-facing escalation. Most are handled automatically, without a case.

Credential exfiltration finding example

Suppose GuardDuty generates an UnauthorizedAccess:IAMUser/InstanceCredentialExfiltration.OutsideAWS finding: temporary credentials created on an Amazon Elastic Compute Cloud (Amazon EC2) instance were used from an IP address outside AWS. Triage cross-references the payload against what it knows about your environment.

  • Cleared – The external IP belongs to a corporate virtual private network (VPN) range that your workloads regularly call through, and the service has seen this pattern before. The finding is archived in GuardDuty.
  • Escalated – The IP is in a hosting-provider range with no prior association to your accounts, and the AWS Identity and Access Management (IAM) role has never made calls from outside AWS. The service creates a proactive case, notifies your incident response team, and an engineer investigates whether the credentials were compromised.

Same finding type, two outcomes. The difference is context, and context is what triage builds by observing your environment over time.

How triage sharpens over time

The service builds a profile of what’s normal in your environment. It learns the IP ranges and domains you trust, and the IAM principals and behaviors your workloads rely on. When your team routinely connects from a known VPN range, or a specific IAM role runs nightly batch jobs, the service recognizes those patterns and measures new findings against them. Early on, engineers investigate more findings by hand as they learn your environment. Within a few weeks, automated triage handles most of those decisions on its own.

Triage is dynamic. A finding treated as expected today can be re-evaluated tomorrow if the context changes, so the service can still catch a real threat that reuses a pattern it formerly treated as safe. You can shape the profile by submitting known-good activity, such as trusted IP ranges or authorized scanning tools, through an AWS Support case. For details, see Submit known-good activity.

Archiving versus suppression

It’s possible to conflate the two mechanisms, so it’s worth separating them:

  • Archiving is the primary mechanism. When automated triage or an engineer determines a finding is benign, the service archives it in GuardDuty. The finding then stays there for 90 days.
  • Suppression rules are less common and work differently. A suppression rule acts on future findings. When the service identifies a recurring pattern of expected activity, it might create a GuardDuty suppression rule that automatically archives matching findings so they don’t generate new alerts. Suppressed findings aren’t sent to your configured export, such as an Amazon Simple Storage Service (Amazon S3) bucket or SIEM. When the service creates a rule, your incident response team is notified, and you can view and manage it in the GuardDuty console under Suppression rules.

A healthy deployment typically shows many archived findings and few or no suppression rules. That’s expected, and it means the service is handling findings dynamically rather than relying on static filters.

How to see what triage is doing

Triage runs on its own, and the service gives you a few ways to confirm it’s working.

Proactive response dashboard

The Dashboard page in the AWS Security Incident Response console has a Proactive response section. It counts how many findings were ingested, triaged, investigated, and escalated over the timeframe you select, the last 7, 30, or 90 days.

The Overview leads with two figures. Noise reduction is the share of ingested findings that automated triage resolved without opening a case, so you can see for yourself how much volume never reached your team. Time saved estimates the analyst hours that automation returned to you, counted as one hour for each ingested finding.

Below that, Finding evaluations breaks those stages down:

  • Ingest – Threat findings collected from your detection sources
  • Triage – Findings evaluated automatically to decide whether the activity is expected
  • Investigation – Findings automated triage couldn’t confirm, passed to an AWS Security Incident Response engineer for a closer look
  • Escalation – Investigations raised as a case when a genuine risk was detected or the activity couldn’t be confirmed as expected

A flow diagram on the dashboard shows how those findings move from their sources through triage and investigation, with branches for the ones closed as false positives and the ones still in progress. For more, see Viewing proactive response metrics.

Monthly report

The monthly report is the periodic companion to the dashboard. AWS Security Incident Response emails it to your incident response team a week or two after each month closes. It opens with the accounts monitored, findings processed, and investigations, then lists each investigation and its status, the triage flows that show the checks run for each finding type, and the context the service has learned about your environment, such as the trusted networks it now recognizes as normal.

Archived findings in GuardDuty

For a quick confirmation outside either view, open Findings in the GuardDuty console and filter to see Archived. That list also includes findings archived by you or by suppression rules, so look for the threat findings AWS Security Incident Response archived after triage. Each one is a finding that the service cleared as benign so your team didn’t have to.

Conclusion

AWS Security Incident Response continuously evaluates your threat findings so your team can spend its time on confirmed threats rather than alert volume. Findings that are evaluated as benign are archived automatically, the ones that need a response become proactive cases, and posture issues stay in Current for your team to handle.

Cases show only a small part of that work. When I deliver AWS Security Incident Response to customers now, I walk them through the triage flow in Figure 1, the Proactive response dashboard, and their monthly report, so they can see how triage works end to end.

To get started, see the onboarding guide. Setup takes about 10 minutes. For more information, see the AWS Security Incident Response product page or the User Guide.

If you have questions about this post, contact AWS Support or start a new thread on AWS re:Post.


Anatoliy Kochetov

Anatoliy Kochetov

Anatoliy is a Technical Account Manager and AWS Security Incident Response SME. He works with AWS Enterprise customers, providing ongoing support and technical guidance to help them build secure, resilient solutions using AWS best practices.

AWS Lambda Durable Functions observability with OpenTelemetry

Post Syndicated from Ricardo Marques original https://aws.amazon.com/blogs/compute/aws-lambda-durable-functions-observability-with-opentelemetry/

Gaining observability into AWS Lambda Durable Functions, a capability of AWS Lambda, can be challenging. A single workflow can span multiple invocations, queues, and downstream services, with no persistent host or centralized log file to inspect. With OpenTelemetry, you can close that gap by generating and correlating traces, metrics, and logs through an open specification. You can instrument once and export anywhere, which avoids vendor lock-in and establishes a baseline for operating serverless systems at scale.

With AWS Lambda Durable Functions, you can construct resilient workflows with multiple steps, which save their state and automatically recover from errors by replaying. A single durable execution can persist across several Lambda calls. It can continue after pauses, recover from errors, and restart from saved checkpoints. Although this model delivers fault tolerance, it introduces an observability challenge. Each invocation generates its own trace, fragmenting the end-to-end view of a single logical workflow.

This post introduces the OpenTelemetry plugin for the AWS Durable Execution SDK (aws-durable-execution-sdk-python-otel). With this plugin, you can generate deterministic trace IDs that unify all invocations of a single durable execution into one coherent distributed trace. In this post, you learn how to set up the plugin, see the traces it produces for the happy-path workflow, and apply foundational best practices.

The complete application code is available in our GitHub repository, so this post uses code snippets to illustrate the examples.

The problem: fragmented traces across invocations

Consider a document review workflow with a human approval gate:

import json

from aws_durable_execution_sdk_python import (
    DurableContext,
    durable_execution,
    durable_step,
)
from aws_durable_execution_sdk_python.types import WaitForCallbackContext


@durable_step
def extract_text(step_context, doc_id):
    step_context.logger.info(f"Extracting text from document: {doc_id}")
    return {"doc_id": doc_id, "text": "extracted content"}


@durable_step
def summarize(step_context, extracted):
    step_context.logger.info(f"Summarizing document: {extracted['doc_id']}")
    return {"summary": "Document summary here"}


@durable_step
def publish(step_context, summary, approval):
    step_context.logger.info(f"Publishing with approval from: {approval.get('reviewer')}")
    return {"status": "published"}


@durable_execution
def handler(event: dict, context: DurableContext) -> dict:
    extracted = context.step(extract_text(event["doc_id"]), name="extract-text")
    summary = context.step(summarize(extracted), name="summarize")

    def submit_for_review(callback_id: str, ctx: WaitForCallbackContext) -> None:
        notify_reviewer(callback_id, summary)

    approval = context.wait_for_callback(submitter=submit_for_review, name="human-review")
    if isinstance(approval, str):
        approval = json.loads(approval)

    if not approval.get("approved"):
        return {"status": "rejected", "reason": approval.get("feedback")}

    context.step(publish(summary, approval), name="publish")
    return {"status": "published"}

This workflow runs across at least two Lambda invocations. The first invocation executes extract-text and summarize, then suspends at human-review with zero compute cost while waiting. Hours later, when the reviewer responds, Lambda invokes the function again. It replays the completed steps and executes publish.

The following diagram illustrates the difference between the default behavior and the plugin’s unified tracing. On the left side, two independent traces are generated: Trace A captures the first invocation (totaling 2.69 seconds), while Trace B captures the second invocation after the callback resumes (replaying and publishing in 0.37 seconds). These traces have different trace IDs and no correlation between them. On the right side, with the OTel plugin, all operations appear under a single deterministic trace ID. The human-review wait (6 hours and 8 minutes at zero compute cost) is represented as a continuous span connecting both invocations, giving operators a complete timeline from start to finish. You can compare the total wall-clock duration of 152 seconds against only 0.019 seconds of actual compute time billed.

Figure 1. race comparison of fragmented traces without the plugin (left) versus a single unified trace with the OTel plugin.

Figure 1. Trace comparison of fragmented traces without the plugin (left) versus a single unified trace with the OTel plugin (right), showing how the Durable Execution SDK merges multiple Lambda invocations into one end-to-end trace.

The plugin achieves this through:

Capability What it does
Deterministic trace IDs Derives a stable trace ID from the AWS X-Ray header, so all invocations share one trace
Span-per-operation Each step, wait, or wait_for_callback gets its own span with real timing
Continuation spans Operations completing in a later invocation link back to the original span
Log correlation Stamps traceId and spanId onto every log record automatically
Configurable sampling Standard OpenTelemetry sampling to control trace volume
Figure 2. Amazon CloudWatch Application Signals Traces view. The durable execution appears as a single trace with OK status. The Duration column (152s) reflects the total wall-clock time including the human-review wait, while Response Time (0.019s) shows only the compute time.

Figure 2. Amazon CloudWatch Application Signals Traces view. The durable execution appears as a single trace with OK status. The Duration column (152s) reflects the total wall-clock time including the human-review wait, while Response Time (0.019s) shows only the compute time.

Choose execution view to see all operations grouped under the Workflow span. Use this view when the full execution, including waits and retries, finishes within your tracing vendor’s span ingestion window. For example, if your vendor rejects spans that started more than 8 hours ago, use execution view only for workflows that reliably finish well under 8 hours.Choose invocation view if an execution might exceed your vendor’s ingestion window, or if you want to see progress as each function execution terminates. Operations appear under the invocation that ran them. A wait’s first segment appears under the invocation that started it, even while the wait is still pending.In both views, the Workflow span covers the full execution, so the vendor’s ingestion window still applies to it. If an execution runs longer than the window, your vendor might reject the Workflow span even in invocation view.

The plugin sends Workflow as one span when the execution succeeds or fail. It doesn’t send that span in pieces. The vendor can, however, accept earlier spans from the same trace.

Why do we recommend someone to use invocation view for long running ones?Suppose a workflow waits 10 hours and the vendor rejects spans that started more than 8 hours ago. In invocation view, the invocation that starts the wait can export the wait’s initial span before the wait finishes. The invocation that resumes the workflow can later export the wait’s completion span. Those short spans can be accepted even if the vendor rejects the final Workflow span.

Getting started

In the following walkthrough, you configure your Lambda durable function to trace requests end-to-end.

Prerequisites

  • Python 3.11 or later.
  • An existing Lambda durable function (or a new one you want to instrument).
  • An AWS account with permissions to enable Amazon CloudWatch Application Signals tracing and add Lambda layers.

Step 1: Install the plugin

Add the plugin to your function’s dependencies:

pip install aws-durable-execution-sdk-python-otel

Or in your requirements.txt:

aws-durable-execution-sdk-python==3.0.2 
aws-durable-execution-sdk-python-otel==2.1.0

Step 2: Add the ADOT Lambda layer

With the AWS Distro for OpenTelemetry (ADOT) Lambda layer, you can export traces from your function.

MyDurableFunction:
  Type: AWS::Serverless::Function
  Properties:
    Runtime: python3.13
    Handler: index.handler
    Architectures:
      - x86_64
    DurableConfig:
      ExecutionTimeout: 31622400
    Layers:
      - !Sub arn:aws:lambda:us-east-1:615299751070:layer:AWSOpenTelemetryDistroPython:33
    Environment:
      Variables:
        AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument

Note: The examples in this post use AWS Serverless Application Model (AWS SAM) for brevity. You can achieve the same configuration using the AWS Command Line Interface (AWS CLI), AWS Cloud Development Kit (AWS CDK), Terraform, or AWS CloudFormation directly. Refer to the ADOT Lambda Layer ARNs page for the latest layer version and supported architectures. Pin the layer version in production deployments to avoid unexpected behavior from automatic version changes.

Step 3: Turn on X-Ray active tracing

The plugin uses the _X_AMZN_TRACE_ID environment variable to derive deterministic trace IDs. Turn on active tracing on your function:

MyDurableFunction:
  Type: AWS::Serverless::Function
  Properties:
    Tracing: Active

Also verify your function’s execution role includes the AWSXRayDaemonWriteAccess managed policy (or equivalent xray:PutTraceSegments and xray:PutTelemetryRecords permissions).

Step 4: Add the OTel plugin to your handler

Import InvocationOtelPlugin and pass it to the plugins parameter of @durable_execution:

from aws_durable_execution_sdk_python import DurableContext, durable_execution
from aws_durable_execution_sdk_python.config import Duration
from aws_durable_execution_sdk_python_otel import create_otel_plugin_factory
@durable_execution(plugins=[create_otel_plugin_factory()])
def handler(event: dict, context: DurableContext) -> str:
    data = context.step(lambda ctx: event["id"], name="fetch-data")
    context.wait(Duration.from_seconds(5), name="cooldown")
    return context.step(lambda ctx: f"processed {data}", name="process")

The plugin handles Tracer Provider setup, deterministic ID generation, and span lifecycle internally.

Testing and results

After you configure the plugin, deploy the function, invoke it, and confirm the unified trace appears in Amazon CloudWatch.

Testing the setup

Build and deploy your function:

sam build
sam deploy --guided

Then publish a function version (durable functions require a qualified ARN, not $LATEST):

aws lambda publish-version --function-name my-durable-function --region us-east-1

Invoke the function asynchronously (required when using wait_for_callback, since the function suspends mid-execution):

aws lambda invoke \
    --function-name my-durable-function \
    --qualifier 1 \
    --invocation-type Event \
    --region us-east-1 \
    --cli-binary-format raw-in-base64-out \
    --payload '{"doc_id": "test-doc-001"}' \
    response.json

Monitor the logs to get the callback ID:

aws logs tail /aws/lambda/my-durable-function --follow --region us-east-1

Wait 2-3 minutes (so the human-review span has visible duration in the trace), then send the callback:

aws lambda send-durable-execution-callback-success \
    --cli-binary-format raw-in-base64-out \
    --callback-id "<CALLBACK_ID_FROM_LOGS>" \
    --region us-east-1 \
    --result '{"approved": true, "reviewer": "alice"}'

Note: The send-durable-execution-callback-success command does not require --function-name. The callback ID already contains the reference to the execution.

What you see in Amazon CloudWatch Application Signals

After you invoke this function, open the Amazon CloudWatch console and choose Application Signals (APM), then Traces. You see a single unified trace containing:

  1. One root span representing the entire durable execution.
  2. Child spans for each named operation: extract-text, summarize, human-review, and publish.
  3. Accurate timing: each span reflects the actual wall-clock time of the operation, not replay time.
  4. Continuation links: if human-review completes in a different invocation (after the reviewer approves hours later), the span is linked back to the original trace.
Figure 3. Trace details view. The service map shows the flow: Client to Lambda Context (first invocation), then Lambda Function (second invocation after callback resume). Both invocations appear under a single trace ID with total duration of 4.55 sec.

Figure 3. Trace details view. The service map shows the flow: Client to Lambda Context (first invocation), then Lambda Function (second invocation after callback resume). Both invocations appear under a single trace ID with total duration of 4.55 sec.

Correlated logs

Because the plugin sets enrich_logger=True by default, your application logs are automatically stamped with traceId and spanId.

Figure 4. Expanded log entry in CloudWatch. Each log record includes traceId, spanId, and otelTraceSampled: true, which lets you correlate logs across all invocations of the same execution.

Figure 4. Expanded log entry in CloudWatch. Each log record includes traceId, spanId, and otelTraceSampled set to true, which lets you correlate logs across all invocations of the same execution.

You can combine the plugin’s log correlation with structured JSON logging to run powerful queries in Amazon CloudWatch Logs Insights:

fields @timestamp, @message, traceId, spanId
| filter traceId = "abcd1234567890abcdef1234567890ab"
| sort @timestamp asc

This returns logs from all invocations in chronological order, correlated to their respective spans.

Best practices

When you run the OTel plugin in production, keep the following performance, operational, and cost considerations in mind.

Performance considerations

  • Cold start impact: The ADOT layer adds overhead to cold starts and memory consumption. Measure the impact for your workload using Lambda Power Tuning. For latency-sensitive functions, consider using provisioned concurrency to keep warm environments available.
  • Memory overhead: The plugin and ADOT layer consume approximately 40–60 MB of additional memory. Verify your function has sufficient memory allocated.
  • Replay efficiency: During replay, the plugin correctly identifies replayed operations and does not emit duplicate spans. Only new operations generate new spans.

Operational guidelines

  • Name your operations: Always pass the name parameter to context.step(), context.wait_for_callback(), and other durable operations. These names appear as span names in your traces, making them human-readable.
  • Use structured logging: Combine the plugin’s log correlation with structured JSON logging to unlock powerful Amazon CloudWatch Logs Insights queries.
  • Monitor trace gaps: If you see traces with missing spans, check that X-Ray active tracing is enabled and the ADOT layer is correctly configured.

Cost management

  • Amazon CloudWatch Application Signals pricing: You are charged per trace recorded and per trace scanned. At high volumes, configure sampling to control costs. See Amazon CloudWatch Application Signals pricing.
  • Log volume: With enrich_logger=True, every log record includes trace context fields, slightly increasing log volume. This overhead is typically negligible but worth noting for high-throughput functions.

Pricing

The plugin is open source (Apache 2.0). Costs come from AWS X-Ray traces recorded and the ADOT Lambda layer overhead. The layer increases both the memory consumption and the cold start duration of your function. Lambda pricing is based on GB-seconds, so the additional memory allocated (rounded up to the nearest MB) multiplied by the additional execution time per invocation determines the cost impact. Each durable execution produces one trace regardless of invocation count, so X-Ray cost scales with executions, not invocations. Measure the overhead for your workload using Lambda Power Tuning and see AWS X-Ray pricing for current rates.

Beyond this post

This post focused on a single, representative workflow (sequential steps plus a human-approval callback) to keep the walkthrough concrete. It is not an exhaustive tour of the SDK. The Durable Execution SDK offers many other primitives that also produce spans and are worth exploring: durable timers (context.wait), condition polling (context.wait_for_condition), durable Lambda-to-Lambda calls (context.invoke), parallel and fan-out execution (context.parallel, context.map), nested child contexts, and automatic retries (with_retry and StepConfig), where each attempt and each parallel branch surfaces as its own span. We also did not cover failure paths, such as step errors and failure callbacks, which is where unified tracing is often most valuable.

Likewise, the OTel plugin exposes configuration we did not exercise here: toggling log enrichment (InvocationOtelPlugin(OtelPluginConfig(enrich_logger=False))), supplying a custom Tracer Provider via OtelPluginConfig(tracer_provider=...), and controlling trace volume through standard OpenTelemetry sampling (OTEL_TRACES_SAMPLER). This walkthrough sampled 100% of traces. Consult the SDK documentation for the full surface area, or access the GitHub repository.

Sending Lambda durable functions traces to Observability partners. The OpenTelemetry plugin exports telemetry data to Amazon CloudWatch and can work with any observability solution that supports OpenTelemetry. Dash0 is an OpenTelemetry-native observability platform that supports Lambda durable functions without custom configuration. Dash0 worked with AWS to validate the OpenTelemetry plugin for the AWS Durable Execution SDK for JavaScript prior to launch, Dash0 coordinates logs and spans. In Dash0, spans for the durable execution, each invocation, and each durable operation appear alongside the function’s inputs and outputs and the payloads of its AWS SDK calls. To learn more, visit About AWS Lambda Monitoring.

Clean up

To avoid ongoing charges, delete the resources created in this walkthrough:

sam delete --stack-name <your-stack-name>

This removes the Lambda function, AWS Identity and Access Management (IAM) roles, and any associated resources created by the SAM template.

If you manually added the ADOT Lambda layer or X-Ray tracing policies outside of your SAM template, remove those separately using the AWS CLI or console.

Trace data retained in X-Ray does not incur storage charges and expires automatically after 30 days.

Conclusion

In this post, we showed how to unify fragmented traces across multiple Lambda invocations using the OpenTelemetry plugin for the Durable Execution SDK. To explore the full SDK surface area, including parallel execution, durable timers, and failure-path tracing, see the AWS Durable Execution SDK Developer Guide. For more observability patterns, visit the AWS Observability Best Practices guide.

How Moonfrog scales analytics and lowers query cost with Amazon Redshift RG instances

Post Syndicated from Anirudh Chawla original https://aws.amazon.com/blogs/big-data/how-moonfrog-scales-analytics-and-lowers-query-cost-with-amazon-redshift-rg-instances/

Moonfrog Labs uses Amazon Redshift to analyze mobile game telemetry. Each session, purchase, and player drop-off generates events that product and analytics teams use to understand game performance. Faster loading and querying help those teams work with fresher data.

Moonfrog runs a single production analytics cluster with roughly 148 TB in Amazon Redshift Managed Storage (RMS) and about 4,000 warehouse tables. It also queries roughly 1,000 external data lake tables. The warehouse supports telemetry ingestion, revenue reporting, marketing attribution, and self-service dashboards for product, data science, analytics, and support teams.

In this post, we show how Moonfrog migrated its production analytics warehouse from Amazon Redshift RA3 to RG instances using elastic resize. We explain how the team selected capacity for growing analytics demand and more frequent loading, verified warehouse and data lake access, and assessed performance and costs after migration.

The challenge

Moonfrog’s RA3 cluster ran reliably, with no stability issues. Its constraint was compute headroom as analytics demand grew. Ingestion volume was steady before migration. Growth came from analytical queries and data lake scans. The team already planned to expand from five RA3 nodes to six or seven nodes and wanted capacity to load warehouse data more frequently.

The five-node ra3.4xlarge cluster averaged about 54% CPU utilization. In approximately 98% of hours, CPU peaked at or above 90%. Other indicators showed demand increasing:

  • SELECT queries per day increased 31% from January to July.
  • Mean workload management (WLM) queue wait for highest-priority queries rose from 505 milliseconds in May to 940.7 milliseconds in July.
  • Concurrency scaling usage nearly doubled over the same period, from 4,453 to 8,516 seconds per day.

A COPY operation averaged 16.8 seconds on RA3. That duration constrained how often the stats pipeline could load. The team evaluated RG as a way to support growing analytics demand and make more frequent loads practical.

Moonfrog also queried roughly 1,000 external tables in Amazon Simple Storage Service (Amazon S3). Data scanned through Redshift Spectrum increased 33%, from 84.9 TB in March to 112.5 TB in June. The separate Spectrum scanning charge added to the cost of expanding RA3 compute.

Moonfrog used WLM query monitoring rules to stop scans larger than one terabyte. These rules controlled scan spend under the RA3 configuration.

The one-year no-upfront reserved node commitment for ra3.4xlarge expired in July 2026. Renewal gave Moonfrog an opportunity to re-evaluate its compute commitment and determine whether switching to RG would provide additional headroom for its growing workload needs.

Solution overview

Amazon Redshift RG instances use AWS Graviton processors and an integrated data lake query engine. AWS publishes benchmarks of up to 2.2 times faster warehouse workloads and up to 2.4 times faster Apache Iceberg workloads, at 30% lower price per vCPU than RA3. These benchmarks are separate from Moonfrog’s production results. RG runs Apache Iceberg and Apache Parquet scans on cluster nodes, removing the separate per-terabyte Spectrum charge for these scans. Elastic resize supports changing the node family and capacity without a customer-managed export and reload.

Moonfrog retained its ingestion and analytics architecture at cutover. No service was added or removed, and applications continued to use the same warehouse.

The following diagram illustrates the architecture.

Architecture: ingestion writes files to Amazon S3, COPY loads Redshift warehouse tables, and the RG data lake engine queries external S3 tables in place

Figure 1: The ingestion pipeline writes columnar files to Amazon S3. COPY loads warehouse tables, and the integrated data lake engine on RG queries external tables in place. Conversational analytics is a planned use of the additional headroom

The architecture uses the following AWS services:

Implementation steps

Moonfrog established a baseline, used elastic resize to change the node family, and verified warehouse and data lake access. It then evaluated the early results before selecting its final configuration.

Step 1: Establish a baseline

Before the change, the team captured Amazon CloudWatch metrics for CPU utilization, WLM queue wait by priority, query duration by type, and queries per day. COPY loads and SELECT queries were measured separately. Billing records provided reserved compute, Spectrum scanning, and concurrency scaling costs so the comparison covered each billing line affected by the move.

AWS provides migration planning and workload validation guidance in the AWS RA3 to RG best practices blog.

Step 2: Validate the target configuration and use elastic resize

AWS recommends testing the target RG configuration in a non-production environment with representative workloads before migrating production. That testing is intended to measure benefits, identify issues, and validate the target size before the production migration.

On 26 July, Moonfrog used elastic resize to move from five ra3.4xlarge nodes to eight rg.4xlarge nodes, temporarily providing extra capacity to reduce migration risk. It did not export or reload the 148 TB warehouse. Schemas, ETL jobs, application SQL, drivers, connection strings, and BI tools continued to work.

Amazon CloudWatch recorded one five-minute interval in which the cluster reported unhealthy. Database connections recovered immediately afterward. This sampling interval describes the observed impact. It does not establish an exact five-minute outage duration.

AWS guidance describes elastic resize as a short operation, typically about 10 to 15 minutes. After an RA3-to-RG node-type change, data hydration brings frequently accessed data from managed storage into the new nodes’ local cache. This cache warming should be considered when interpreting early performance measurements.

Step 3: Verify warehouse and data lake access

Moonfrog verified that warehouse loads and external queries continued after the resize. External schemas and tables remained in use, with AWS Glue Data Catalog providing their metadata.

Moonfrog confirmed that per-terabyte S3 scanning charges were eliminated for its RG queries in the subsequent billing records. Those queries used provisioned cluster capacity, which remained billable.

Step 4: Review workload management rules

Moonfrog’s Auto WLM configuration included separate queues for administrators, analysts, ETL, loading, and visualization, along with short query acceleration. These queues carried across the migration. The team reviewed query monitoring rules because data lake scans now consumed RG cluster capacity.

The review accounted for data lake scans sharing compute with warehouse workloads. Rules that previously controlled scan spend could still protect these shared resources.

Results and key benefits

The following four CloudWatch charts cover 1 May to 25 October 2026. Each marks the 26 July resize and shades the 26 July–3 August period when the cluster was reduced from eight to seven to six RG nodes. Query duration and WLM wait charts show seven-day rolling means. Query counts include COPY, SELECT, INSERT, utility, and other operations.

Stage 1: Evaluate RG after cutover

The first evaluation focused on workload continuity and query runtime. Warehouse loads and data lake queries continued, and the applications used the same schemas, SQL, drivers, and connection strings. Queries on the initial RG configuration were completing within expected runtimes. Moonfrog then reduced the node count in stages to assess whether each reduction affected query runtime.

The cleanest comparison of COPY duration is 27–31 July, when load volume remained unchanged. Mean duration fell from 16.8 seconds on RA3 to 3.6 seconds on RG, about 79% shorter. This window falls within the 26 July–3 August right-sizing period, before the final six-node configuration was established.

Select six nodes

Moonfrog reduced the cluster from eight to seven and then six RG nodes within eight days. The team retained the additional node relative to its five-node RA3 baseline to support growing analytical demand and more frequent loading. It had already planned to expand RA3 to six or seven nodes.

The move from five RA3 nodes to six RG nodes increased node count by 20%. Published node specifications list 12 vCPUs per ra3.4xlarge and 16 per rg.4xlarge, so provisioned vCPUs increased from 60 to 96, or 60%. The results reflect the change in both node family and capacity.

Stage 2: Sustain performance as workloads grow

After validating the available headroom on the six-node configuration, Moonfrog increased loading frequency on 24 August. The cluster now executes more than 70,000 COPY operations a day, about 4.3 times the RA3 baseline. These loads use smaller, more frequent batches, so later COPY duration also reflects the changed loading pattern. Figure 2 shows the early duration improvement and subsequent performance as the workload changed.

In late August, after the 24 August increase in loading frequency, Moonfrog purchased a one-year no-upfront reservation for the six-node RG configuration. The purchase followed its evaluation of RG with production traffic and committed capacity for anticipated workload growth.

Customer CloudWatch chart of seven-day rolling mean COPY and SELECT duration from May to 25 October 2026. Resize on 26 July and right-sizing through 3 August.

Figure 2: Mean COPY and SELECT duration. The early COPY comparison uses unchanged volume. Later means also reflect smaller batches after 24 August

More daily queries with compute headroom

Daily queries across all types increased from around 49,122 on RA3 to 83,603 in August, 125,350 in September, and 125,374 during 1–25 October. From September onward, this was more than 2.5 times the RA3 baseline. COPY accounts for much of that increase, so total query growth should be read alongside the query mix in Figure 3.

Customer CloudWatch stacked-area chart of successful COPY, SELECT, utility, INSERT, and other queries per day, showing the increase in COPY frequency on 24 August.

Figure 3: Successful daily queries by type. COPY frequency increased on 24 August, after the migration and right-sizing period

Concurrency scaling remained enabled to absorb peak bursts alongside the six reserved nodes. Average daily usage was 14–29% below the RA3 baseline in August, September, and October, even as total query volume grew. Usage increased from September to October as demand rose.

Average CPU utilization fell from about 54% on RA3 to 35–39% from September onward, despite the higher query count. Figure 4 distinguishes daily average CPU from the typical peak, defined as the daily mean of each hour’s maximum CPU reading.

Customer CloudWatch chart of daily average CPU and the daily mean of hourly maximum CPU readings. Average utilization falls after the 26 July resize.

Figure 4: CPU utilization before and after the resize. The typical peak is an average of hourly maximum readings, not the single daily maximum

Queue waits as demand grows

Queue waits fell after migration and rose from their August lows as traffic increased in October. During 1–25 October, highest-priority waits averaged 561 milliseconds versus a July RA3 baseline of 940.7 milliseconds, and high-priority waits averaged 193 milliseconds versus 294 milliseconds. Normal-priority waits were about 11 milliseconds, above their July baseline of about 3.6 milliseconds. Figure 5 uses a separate vertical scale for each priority.

Customer CloudWatch chart with highest, high, and normal priority panels. Highest and high priority mean waits remain below their RA3 July baselines in October. Normal priority waits rise.

Figure 5: Seven-day rolling mean WLM queue wait by priority. Dashed lines mark July RA3 baselines. Each panel has its own scale

Integrated data lake queries and billing

External data lake queries continued after migration. Moonfrog confirmed that the separate Spectrum scanning charge remained exactly $0 from 27 July through the reported months:

  • The Amazon Redshift system views showed that external queries continued. A recent three-day sample recorded roughly 200,000 external queries against Amazon S3, including Parquet scans. External schemas and tables remained in use.
  • Billing records showed no separate per-terabyte scanning charge. External scans use provisioned RG capacity, which remains billable. Amazon S3 storage and other charges are outside this comparison.

Cost per query

Moonfrog chose additional capacity for growth, and monthly spend across reserved compute, Spectrum scanning, and concurrency scaling increased approximately 4–6%. On a consistent 30-day basis, cost per 1,000 total queries fell approximately 59% in September and 58% in October. Costs per 1,000 SELECT queries fell 38–50% across August through October.

Metric RA3 × 5
Before
RG × 6
August
RG × 6
September
RG × 6
Oct 1–25
Queries per day All types 49,122 83,603 125,350 125,374
Cost per 1,000 total queries $5.60 $3.45 −38% $2.28 −59% $2.33 −58%
Reported cost per 1,000 SELECT $45.60 $22.73 −50% $25.19 −45% $28.34 −38%

Cost per 1,000 total queries is calculated by dividing the combined 30-day spend for reserved compute, Spectrum scanning, and concurrency scaling by 30 times the average daily query count, then multiplying by 1,000. These are aggregate cost measures, not the execution cost of an individual query.

How the team plans to use the headroom

The larger configuration created room to test additional analytical use cases. Earlier measurements showed mean short-query duration falling from about 2.8 seconds to 1.8 seconds. That observation applies to the measured short-query workload and does not establish a response-time guarantee for every analytical question.

Moonfrog plans to evaluate conversational analytics using large language model (LLM) agents. Such agents may execute several queries to answer one question, so query latency is one part of the end-to-end response time. The additional compute headroom gives the team room to test these workloads alongside existing analytics.

Lessons learned

  • Use a reserved node renewal as an opportunity to review capacity and architecture for your growing workload needs. Moonfrog evaluated RG before making another one-year commitment.
  • Size the cluster for expected workload growth and assess results against the capacity provisioned. Moonfrog added a sixth node for headroom. Interpret total query counts alongside the query mix, because smaller, more frequent COPY batches increase the number of operations.
  • Include concurrency scaling in capacity planning. In this deployment it continued to absorb peak bursts alongside the reserved nodes, so evaluate its contribution when assessing warehouse throughput.
  • Separate immediate migration results from subsequent workload changes. The unchanged-volume COPY comparison captures the first days after cutover. Later results include the loading cadence change and rising demand. Measure dashboard latency and query abandonment separately to quantify the effect on users.

Next steps

Moonfrog has already increased warehouse loading frequency. The team plans to measure dashboard load times and query abandonment to assess how greater freshness and shorter query durations affect the experience of product and analytics teams.

Moonfrog is considering Apache Iceberg for data lake tables currently stored as Parquet files. The table format would add capabilities such as schema evolution and time travel. The team will evaluate these changes alongside performance and operating requirements.

The team also plans to evaluate broader natural language access. Governance must define which teams can access each table and how exploratory queries share cluster resources. End-to-end agent response time will need separate measurement.

Conclusion

Moonfrog migrated its 148 TB production Amazon Redshift warehouse from five RA3 nodes to RG using elastic resize, then settled on six nodes after eight days of measured right-sizing. RA3 had run reliably. The change addressed growing analytics demand and the need for more frequent warehouse loads. COPY duration fell from 16.8 seconds to 3.6 seconds at unchanged volume immediately after cutover.

With six RG nodes, Moonfrog handled more than 125,000 daily queries across all types from September onward, over 2.5 times the RA3 baseline. Average CPU utilization was 35–39%. More frequent COPY batches contributed to the change in query mix. External queries continued with no separate Spectrum scanning charge, and concurrency scaling provided burst capacity alongside the reserved nodes.

To evaluate RG for your warehouse, review Amazon Redshift pricing, the elastic resize documentation, and the RA3 to RG migration best practices. Establish a baseline and test representative workloads in a non-production environment first. Validate warehouse and data lake access, then compare total spend and cost per unit of work before committing to reserved capacity.


About the authors

Anirudh Chawla

Anirudh Chawla

Anirudh is an Analytics Solution Architect at AWS. He helps organizations harness their data effectively through AWS analytics solutions. His interests include building highly available distributed systems.

Poornima Bansal

Poornima Bansal

Poornima is a Senior Account Executive at Amazon Web Services (AWS) based in Bengaluru, India. She leads cloud and generative AI engagements with digitally native businesses. She is passionate about helping organizations harness AI and emerging technologies to drive innovation.

Sathish Balaji

Sathish Balaji

Sathish is an Analytics Specialist at Amazon Web Services. He focuses on building strong data foundations that enable analytics and AI. Outside work, he enjoys travelling and experiencing new cultures.

Monazir Hussain

Md Monazir Hussain

Monazir is a Senior Software Engineer at Moonfrog Labs, specializing in data platforms, cloud infrastructure, and distributed systems. He works on designing and scaling backend and data systems that power large-scale mobile games. He is passionate about solving complex scalability challenges, improving system reliability, and using automation and AI to simplify engineering and operational workflows.

Raunshan Kumar

Raushan Kumar

Raushan is a DevOps Engineer 2 working on cloud infrastructure and data platforms at Moonfrog. He specializes in designing, automating, and optimizing AWS infrastructure, with expertise spanning Kubernetes, CI/CD pipelines, infrastructure as code, and Amazon Redshift. He focuses on improving reliability, scalability, performance, and cost efficiency across Moonfrog’s gaming and analytics workloads.

Monitor the efficiency and availability of accelerated computing instances with Amazon EC2 Detailed Monitoring and performance status checks

Post Syndicated from Caroline Frost original https://aws.amazon.com/blogs/compute/monitor-the-efficiency-and-availability-of-accelerated-computing-instances-with-amazon-ec2-detailed-monitoring-and-performance-status-checks/

Amazon Elastic Compute Cloud (Amazon EC2) accelerated computing instances run some of the most demanding workloads customers operate, from large model training and distributed fine-tuning to high-throughput, live inference. These customers seek high availability and performance to make sure their training and inference run efficiently. To meet their performance goals, customers require visibility into the efficiency and performance of each accelerated computing chip in their fleet.

For example, central infrastructure or machine learning teams often operate accelerated computing capacity for many application teams. These infrastructure teams need a clear view of utilization and health across every instance in their fleet even if they don’t develop the applications running on those instances. They are often responsible for how efficiently that capacity is used and for hardware health, while application teams own the workload. Customers asked us to provide foundational telemetry for their accelerated computing efficiency and status that doesn’t require setup within the instance and complements their in-guest tooling.

Amazon EC2 now reports one-minute graphics processing unit (GPU) metrics through Detailed Monitoring and surfaces recoverable conditions with performance status checks. One-minute GPU metrics in Amazon EC2 Detailed Monitoring report per-GPU utilization and thermal data to Amazon CloudWatch, so you can identify underused capacity and tune workloads for cost and throughput. Performance status checks report recoverable hardware conditions, such as a GPU memory error, to help you manage GPU performance.

Feature overviews

Amazon EC2 reports GPU metrics and performance status checks without an in-guest agent and without using instance compute. These capabilities are available on supported NVIDIA GPU-based accelerated computing instance types. For the metrics supported on each instance family, see the Amazon EC2 Detailed Monitoring documentation.

One-minute accelerated computing metrics extend EC2 Detailed Monitoring and are published to CloudWatch as OpenTelemetry Metrics or CloudWatch Metrics. The metric name you add to a category determines which one CloudWatch publishes: include GPUUtilization for a CloudWatch Metric, instance.gpu.utilization for an OpenTelemetry Metric, or both names to publish both. Amazon EC2 publishes each metric to CloudWatch at two levels: one series for the instance, identified by InstanceId, and one series for each GPU, identified by InstanceId and GpuId. GpuId is the GPU UUID, which you can match to a device by running nvidia-smi -L on the instance.

Performance status checks add a fifth status check to the existing Amazon EC2 product check suite of system, instance, attached EBS, and application status checks. The check reports recoverable hardware issues that affect accelerated computing performance while the instance might remain running and reachable. It surfaces through the EC2 console, through DescribeInstanceStatus, and in CloudWatch metrics.

Evaluating accelerated computing utilization and performance with EC2 Detailed Monitoring

Amazon EC2 Detailed Monitoring reports per-GPU measurements, including compute activity, memory use, temperature, and data movement.

GPU utilization metrics report different aspects of GPU use. GPUPowerUtilization is available by default and reports active power use. Power utilization scales with the depth of work the GPU performs. Compare it with more granular utilization metrics to see how efficiently your workload uses the GPUs. GPU utilization shows the percentage of time the GPU was running work. Because a busy GPU might use only a fraction of its streaming multiprocessors (SMs), compare it with SM utilization to see how much of the compute capacity the workload uses. Memory utilization shows how much of the GPU’s memory capacity is in use, helping you assess memory headroom. Context accumulated time is a running total of the time a process has held the GPU open, whether or not it ran work. Use the CloudWatch metric math expression RATE(m1)/1000000, where m1 is GPUContextAccumulatedTime, to compare context activity with GPU utilization. A result close to 1 with low GPU utilization indicates that a GPU context is held open but idle.

GPU temperature is reported as a percentage of the GPU’s thermal throttle limit, and memory temperature as a percentage of the memory’s critical temperature. Both metrics can report values above 100 percent when a GPU or its memory runs above the limit. A GPU reduces its clocks when it reaches its throttle limit, so a value at or above 100 percent can explain a drop in throughput. You can correlate these measurements with GPU and interconnect utilization metrics to understand workload performance.

Interconnect utilization metrics show how much of the available bandwidth a workload uses for data movement. Each value reports the busier direction, either send or receive. NVLink utilization measures GPU-to-GPU traffic, such as collective operations in distributed training. It reports all of the GPU’s links together. PCIe utilization measures traffic between the host and the GPU.

Getting started with EC2 Detailed Monitoring

Amazon EC2 Detailed Monitoring publishes instance metrics to Amazon CloudWatch at one-minute intervals without running an agent in the guest operating system. You can publish these as OpenTelemetry Metrics or CloudWatch Metrics.

To get started, create a metric category, an Amazon EC2 resource containing the metrics you want to collect. Next, associate the category with one or more instances. For an existing instance, provide the category ID when you enable Detailed Monitoring with the MonitorInstances API. For a new instance, provide the category ID in the Monitoring object when you call RunInstances or configure a launch template.

The following example creates a metric category for GPU utilization, SM utilization, and memory utilization.

aws ec2 create-metric-category --metrics instance.gpu.utilization instance.gpu.sm_utilization instance.gpu.memory_utilization

The response includes a metric category ID. To associate the category with an existing instance, use MonitorInstances.

aws ec2 monitor-instances --instance-ids <instance-id> --metric-category-ids <metric-category-id>

To associate the category when you launch an instance, include the category ID in the Monitoring object.

aws ec2 run-instances --image-id <ami-id> --instance-type <instance-type> --count 1 --monitoring Enabled=true,MetricCategoryIds=<metric-category-id>

To apply a category to every instance launched from a launch template, include the category ID in the Monitoring object in the launch template data.

aws ec2 create-launch-template --launch-template-name accelerated-computing-monitoring --launch-template-data '{"Monitoring":{"Enabled":true,"MetricCategoryIds":["<metric-category-id>"]}}'

Categories apply to many instances at once, unlike per-instance metric settings. Define the metric set once and associate it wherever it applies. Every associated instance then reports the same set of metrics. Changing a category updates the metric set for every associated instance. The change is eventually consistent across running instances. Add the category to a launch template to apply the same metrics to new instances. This gives you one metric definition per workload type instead of one configuration per host. Metrics apply under standard Amazon CloudWatch pricing.

Managing accelerated computing availability with performance status checks

Accelerated computing hardware can experience non-terminal issues that affect performance and can be resolved with a GPU reset or an instance reboot. For example, when a GPU encounters an uncorrectable memory error, it schedules a repair that remaps the affected row to a spare. The repair takes effect the next time the GPU is reset or the instance is rebooted.

Amazon EC2 performance status checks surface these recoverable hardware conditions. For example, when Amazon EC2 detects a pending GPU memory row remap, the instance’s performance status check reports an impaired status and identifies reboot as the recommended remediation.

You can view performance status checks on the Amazon EC2 console or through DescribeInstanceStatus. Amazon EC2 also publishes the StatusCheckFailed_Performance metric to CloudWatch. A failed performance status check also increments the combined StatusCheckFailed metric. The following example uses the EC2 DescribeInstanceStatus API:

aws ec2 describe-instance-status --instance-ids i-0123456789abcdef0

The following response shows the system, instance, attached EBS, and configured application status checks passing while the performance status check is impaired:

{
  "InstanceStatuses": [
    {
      "AvailabilityZone": "us-east-1a",
      "InstanceId": "i-0123456789abcdef0",
      "InstanceState": {
        "Code": 16,
        "Name": "running"
      },
      "InstanceStatus": {
        "Details": [
          {
            "Name": "reachability",
            "Status": "passed"
          }
        ],
        "Status": "ok"
      },
      "SystemStatus": {
        "Details": [
          {
            "Name": "reachability",
            "Status": "passed"
          }
        ],
        "Status": "ok"
      },
      "AttachedEbsStatus": {
        "Details": [
          {
            "Name": "reachability",
            "Status": "passed"
          }
        ],
        "Status": "ok"
      },
      "ApplicationStatus": {
        "Status": "ok"
      },
      "PerformanceStatus": {
        "Status": "impaired",
        "ImpairedSince": "2026-10-18T14:32:00Z",
        "Remediation": "reboot"
      }
    }
  ]
}

Amazon EC2 sends a scheduled event for the instance if the condition requires further action after a reboot.

To act on these conditions across your accelerated computing instances, create a CloudWatch alarm on StatusCheckFailed_Performance.

  1. The alarm on StatusCheckFailed_Performance fires for an instance in a training cluster.
  2. You confirm the condition and the recommended action with DescribeInstanceStatus, which returns a status of impaired and a remediation of reboot.
  3. You checkpoint the job if applicable, then drain the node and reboot the instance at a planned point. On Amazon Elastic Kubernetes Service (Amazon EKS), cordon and drain the node. In Slurm, set the node to DRAIN.
  4. After the reboot completes, the performance status check returns to ok.

The following command creates that alarm for a single instance.

aws cloudwatch put-metric-alarm --alarm-name performance-status-impaired-<instance-id> --namespace AWS/EC2 --metric-name StatusCheckFailed_Performance --dimensions Name=InstanceId,Value=<instance-id> --statistic Maximum --unit Count --period 60 --evaluation-periods 2 --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold --alarm-actions <sns-topic-arn>

A CloudWatch Metrics Insights alarm evaluates a single query across all matching instances, so one alarm can cover a large fleet. To drive automation without an alarm, poll the DescribeInstanceStatus API.

Rebooting the instance is the recommended remediation. Alternatively, you can reset the affected GPU while keeping the other GPUs running. On the affected instance, run nvidia-smi -q -d ROW_REMAPPER and find the GPU that reports Pending: Yes. Stop all processes using that GPU, then use root permissions to run nvidia-smi -r -i <GPU_UUID>. The performance status check returns to ok shortly after the reset completes.

Bringing it all together

Accelerated computing metrics and performance status checks address complementary questions. Detailed Monitoring metrics help you find idle or underused capacity and tune workloads for cost and throughput. Performance status checks surface recoverable conditions and recommend remediation. Together, these EC2 signals report utilization and status across the capacity infrastructure teams manage, while in-guest tooling provides workload-specific analysis.

For example, a central infrastructure team monitors how efficiently workloads use the accelerated computing instances it manages for several application teams. It finds that one team’s GPUs report lower GPU utilization and power utilization than comparable workloads run by other teams. The infrastructure team shares the finding and relevant workload-tuning practices with the application team. After the application team updates that workload, the infrastructure team uses the same CloudWatch metrics to evaluate the change. The infrastructure team also monitors StatusCheckFailed_Performance across the fleet. When a check reports an impaired status and recommends a reboot, the orchestrator saves the active job, stops placing new work on the instance, and reboots it. After the check returns to ok, the orchestrator makes the instance available for new jobs. Together, these workflows help the team improve GPU utilization, protect job progress, and return affected capacity to service.

Amazon EC2 Detailed Monitoring reports accelerated computing metrics once per minute at the instance and GPU levels. Performance status checks report recoverable conditions at the instance level. Infrastructure teams can use these EC2 signals to monitor utilization and status across the capacity they manage. In-guest tools add process, container, Kubernetes pod, and sub-minute detail.

Amazon EC2 Detailed Monitoring reports metrics for your instances, including GPU utilization metrics reported per GPU. Performance status checks tell you when accelerated computing hardware needs attention. These capabilities help you evaluate utilization and manage the efficiency of your accelerated computing fleets. To learn more, see the Amazon EC2 Detailed Monitoring documentation and the EC2 status checks documentation.


About the author

Caroline Frost

Caroline Frost

Caroline is an Amazon EC2 Senior Product Manager – Technical, focusing on the EC2 instance lifecycle. She is passionate about building capabilities that improve the developer experience and create operational ease and simplicity.

Guardrails and cost control for your agentic workloads with Amazon Redshift

Post Syndicated from Jason Pedreza original https://aws.amazon.com/blogs/big-data/guardrails-and-cost-control-for-your-agentic-workloads-with-amazon-redshift/

Note (November 2026): This post covers both MCP servers because the Redshift MCP Server has Redshift-specific settings, such as read-only by default and a cap on result rows, that the AWS MCP Server doesn’t have yet. We’ll update this post when the AWS MCP Server reaches parity.

Guardrails and cost control matter more as AI agents handle an increasing share of the queries in your analytics environments. Asked “Why did revenue drop in the West region last weekend?”, an agent finds the tables, writes and runs the SQL on Amazon Redshift, and follows up. This is agentic analytics, on your data warehouse and lakehouse.

Two risks decide whether these agentic workloads reach production. The OWASP Top 10 for LLM Applications names both:

  • Excessive agency: The agent can do too much. An agent acts on what a model decides. With more tools or permissions than its task needs, a misread request or an instruction hidden in a support ticket can become a real action, such as exporting customer records.
  • Unbounded consumption: The agent can spend too much. A retry loop can rerun a heavy query all night. A query can return millions of rows that the agent then sends to the model as input tokens. Finally, one SQL query that runs Amazon Bedrock model inference through Redshift ML invokes the model once for every row it processes.

Redshift includes the controls for both at no additional charge: per-agent roles, row-level and column-level security, dynamic data masking, read-only tools, workload management, and usage limits. You pay for queries from agents as you do for queries from any other client: for the compute and storage they use (Amazon Redshift pricing). Your agent’s model provider bills its model tokens, for example at Amazon Bedrock pricing when the agent uses a model in Amazon Bedrock. You pay for the Amazon Bedrock, Amazon Bedrock AgentCore, and monitoring controls in this post at their own rates.

In this post, we show you how to add guardrails and cost controls to agentic analytics on Redshift, using one model: six guardrails, three enforcement layers, and one cost envelope.

The layered guardrail model

Each request passes through the agent layer, the access layer, and the data layer before it reaches your data warehouse or lakehouse, and the cost envelope limits spend across them.

The layered guardrail model for agentic analytics on Amazon Redshift, read top to bottom in four numbered rows. 1, agentic patterns: conversational analytics through AI assistants with MCP servers or apps with Knowledge Bases, Enrich with Redshift ML, Act with scoped tools behind AgentCore Gateway, and Operate with Kiro and Redshift skills. 2, the agent layer, mostly probabilistic, screens prompts and tickets, guides reasoning, and checks the answer. 3, the access layer, deterministic, keeps SQL tools read-only, blocks infrastructure changes, and scopes writes. 4, the data layer, prompt-independent, applies per-agent identity, row, column, and masking policies, output limits, and compute limits. One storage box holds Redshift managed storage, and Apache Iceberg in Amazon S3 and S3 Tables with Lake Formation permissions. Below, the cost envelope adds model tokens, query compute, and data at rest, with no new meter.

Figure 1: The layered guardrail model. The lower the layer, the less a prompt can change it

Each guardrail type controls one thing and maps to an OWASP risk. Each layer is who enforces it:

  • Agent layer: Your agent or application, its framework, and Amazon Bedrock.
  • Access layer: The tools between the agent and your data, such as MCP servers and AgentCore Gateway (a capability of Amazon Bedrock AgentCore), and the IAM policies on the identities those tools use.
  • Data layer: Amazon Redshift and, for lakehouse tables, AWS Lake Formation.

The following table lists the six guardrail types. Each row names the guardrail type, the main OWASP risk it reduces, and the layer and controls that enforce it.

Guardrail type Main risk (OWASP) Who enforces it
Input Prompt injection (LLM01) Agent layer: Bedrock Guardrails, input validation
Reasoning and planning Excessive agency (LLM06) Agent layer: skills, business context, tool allow lists, step limits
Tool and action Excessive agency (LLM06) Access layer: tools, IAM, Policy in Amazon Bedrock AgentCore, threshold approval
Data Sensitive information disclosure (LLM02) Data layer: privileges, row and column policies, masking
Output Sensitive information disclosure (LLM02), Improper output handling (LLM05) Data layer. Agent layer for answers
Runtime and cost Unbounded consumption (LLM10) Data layer limits, token caps, AWS Budgets

The agent layer is the first filter, but it cannot be the only one. Most Amazon Bedrock Guardrails checks are model-based classifiers, some with a strength that you tune, so they catch most problems but not all. The guardrails that protect your data in Redshift therefore sit in the access and data layers, which apply fixed rules to the caller’s identity. You need both. AWS Identity and Access Management (IAM) policies decide which Redshift workgroups and APIs it can call, and Redshift and Lake Formation permissions decide what its SQL can touch. Fixed rules give the same answer whichever agent wrote the request, so a prompt injection that gets past the agent layer still cannot reach data that the agent’s identity is not granted.

Agentic analytics patterns on Amazon Redshift

Conversational analytics, the most common pattern, answers business questions from your data in self-service and managed delivery models. The others enrich data, take actions, or help administrators run the warehouse itself.

Pattern How it connects to Redshift Starting agency
Conversational analytics, self-service AI assistants (agents that a person works with directly) such as Kiro, Amazon Quick, and the Amazon SageMaker Data Agent, through the AWS MCP Server or Amazon Redshift MCP Server over the Redshift Data API Read-only
Conversational analytics, managed Amazon Bedrock Knowledge Bases turns end users’ questions into SQL Read-only
Enrich Redshift ML with Amazon Bedrock calls a model from SQL None: the model returns a column value
Act One scoped tool per action Scoped writes. Human approval for consequential ones
Operate Administrators review and tune the warehouse with Kiro, Redshift skills, and the MCP Server review_cluster tool Read-only. Changes ship as reviewed code

Every pattern needs the same two scopes for least privilege: IAM permissions that limit which workgroups and APIs an identity can call, and database permissions that limit what its SQL can read or change. What differs is which identity reaches Redshift:

  • Self-service (AI assistants with the Redshift MCP Server or AWS MCP Server). The AI assistant acts on behalf of the user, with the AWS profile in its MCP configuration. If your roles have different scopes, set up a profile for each and point the server at the narrowest one for the task, such as a read-only analyst role for sales questions. The setup steps are in access-layer/redshift-mcp-server and access-layer/aws-mcp-server in the aws-samples repository.
    • Redshift MCP Server: A user whose profile assumes a DataPlatformAdmin role runs every agent query as IAMR:DataPlatformAdmin, with that role’s grants. The server is read-only by default, and the agent can’t switch profiles. To use a broader profile, change it yourself in the MCP configuration.
    • AWS MCP Server: The agent can switch on each call among the profiles listed in the MCP Proxy for AWS configuration. List only the profiles you want it to use, and put the read-only one first so it’s the default. On provisioned clusters, a script can also choose a database user if the role allows redshift:GetClusterCredentials, so leave that permission out.
    • Both: The MCP configuration only controls the MCP tools. An agent that can run shell commands can use any profile on the machine, so the IAM roles and database grants are the real boundary.
    • SageMaker Data Agent: The agent runs SQL itself, including cells it fixes, as the signed-in person. Without trusted identity propagation, Redshift sees one IAMR:user-<id>@<project> database user per project member. With it, Redshift sees the person’s AWS IAM Identity Center user, so row-level security, masking, and grants apply to each person.
  • Custom agents, for example on Amazon Bedrock AgentCore. The database user comes from the role that calls the Data API: the AgentCore Runtime execution role, or the AWS Lambda function’s role behind AgentCore Gateway, such as IAMR:agent-support. If the agent acts for a specific person, use trusted identity propagation. The agent’s tool exchanges the person’s sign-in token through AWS IAM Identity Center, optionally federated with your external identity provider, and calls the Data API as that person. Redshift then applies their grants (writes included), row-level security, and masking, plus Lake Formation permissions if you enable that integration.
  • Managed (Amazon Bedrock Knowledge Bases). The knowledge base runs the SQL it generates as one identity. On Serverless workgroups and provisioned clusters, that is its service role with IAM or the user in an AWS Secrets Manager secret. Provisioned clusters also accept a named database user (authentication methods). Knowledge Bases can’t carry the actual user with trusted identity propagation, so when one identity serves many end users, row-level security can’t tell them apart. With one shared knowledge base, a West region manager and an East region manager query as the same database user, so both get every row that user can read. Instead, create one knowledge base or agent for each audience, such as one per sales region, and scope its role, grants, and row-level security policy to that audience.
Three access paths and the identity Amazon Redshift sees on each. A self-service AI assistant with the Redshift MCP Server or AWS MCP Server acts with the user’s AWS profile, so its queries run as a database user such as IAMR:DataPlatformAdmin. A custom agent in AgentCore Runtime, or a Lambda tool behind AgentCore Gateway, uses its execution role, such as IAMR:agent-support. Bedrock Knowledge Bases uses one identity for all its end users: its service role, a database user on provisioned clusters, or a Secrets Manager user, so row-level security can’t tell those users apart. Every path needs an IAM policy and database permissions. A band says one identity per agent gives it its own policies, cost, and audit.

Figure 2: Who Redshift sees on each access path

Decide how much agency to grant

Grant agency, what an agent can do, on two planes. The data plane is SQL on Redshift managed storage or Apache Iceberg tables, governed by Redshift and AWS Lake Formation permissions. The control plane is API calls that create, resize, manage, or delete infrastructure, governed only by IAM. The AWS Agentic AI Security Scoping Matrix bounds agency with permissions and autonomy with oversight. Following it, start read-only (Scope 1), then add scoped writes with a human in the loop for consequential ones (Scope 2). Keep agents off the control plane, and widen autonomy (Scopes 3 and 4) only within those boundaries.

Agent layer: Guide the request

The agent layer, mostly outside Redshift, is the first filter. Its controls run in your agent, Amazon Bedrock, and Amazon Bedrock AgentCore, so you set them up and pay for them there. These include the step, retry, and token limits that cap the agent’s own model spend.

  • Input. Bedrock Guardrails filters prompt injection attempts and denied topics, and ApplyGuardrail works with your choice of model.
  • Skills. The redshift-guide skill gives agents Redshift-specific knowledge, so fewer queries fail. The Agent Toolkit for AWS (aws configure agent-toolkit, AWS Command Line Interface (AWS CLI) 2.35.0 or later) and the AWS Data Analytics Plugin deliver skills to Kiro and other AI assistants.
  • Business context. Business context and skill assets in the AWS Glue Data Catalog tell agents which tables and joins to use. They are in preview in the US East (N. Virginia), US East (Ohio), US West (Oregon), and Europe (Ireland) Regions.
  • Agent configuration and oversight. Limit an agent’s tools before it runs. In Kiro, for example, the MCP configuration lists the tools to approve automatically (autoApprove) and the tools to hide from the agent (disabledTools). Add project instructions, and set step and retry limits in agent frameworks.

Access layer: Limit what the agent can call

The access layer decides which tools an agent can call and what each call can do. Redshift enforces its guardrails the same way for every prompt.

Data plane: A narrow tool surface on every path

Figure 3 shows the tool guardrail each connection path needs, plus least-privilege IAM and database permissions. Each path has a worked example in the access-layer folder of the aws-samples repository, with setup, verification, and clean-up steps.

Four access paths, each with what the agent can send, its guardrail, and how it connects. Redshift MCP Server: any SQL. Read-only mode: nothing a query changes is kept. One Data API call per tool call. AWS MCP Server: Python in aws___run_script. An agent profile and IAM policies on aws:ViaAWSMCPService. One Data API call per statement. Custom MCP server: only tool parameters. Fixed SQL and validated parameters, with AgentCore Policy checking each call at AgentCore Gateway. Bedrock Knowledge Bases: questions turned into SQL. One database and up to 50 tables, with descriptions and example queries steering the SQL. One session per Retrieve call. A band adds CONNECTION LIMIT and statement_timeout on every path, and SESSION TIMEOUT for idle driver sessions.

Figure 3: Access paths and their guardrails

Agents with shell access can bypass an MCP server with the AWS CLI, so scope IAM on every path. To allow only the AWS MCP Server path, see AWS MCP Server only, later in this section.

The agent’s IAM policy, access-layer/shared/agent-data-plane-policy.json in the aws-samples repository, has three statements:

  • Discover lets the agent list workgroups. It uses "Resource": "*" because ListWorkgroups doesn’t support resource-level permissions, so it can’t limit where SQL runs.
  • RunSqlOnOneWorkgroup lets the agent sign in and run SQL on one workgroup only. This is the statement that limits where SQL runs. It includes redshift-data:BatchExecuteStatement, which the Redshift MCP Server needs before it can run writes or named transactions. Without that action, the server stays read-only by refusing writes itself. With it, writes still need ACCESS_MODE=read-write in the MCP configuration, which is off by default, and the agent’s database grants still limit what a write can change.
  • ReadOwnStatements lets the agent check status and read results only for statements it ran itself.

Custom MCP server: Tools that run fixed SQL. Expose only tools that each run one fixed, parameterized statement against a curated view, under the server’s own IAM role with SELECT on that view only. Have each tool set a fixed StatementName and SQL comment, so AWS CloudTrail and query history name the tool. In the sample server (access-layer/custom-mcp-server/sales_metrics_mcp.py), the agent supplies only a sales region, which the server checks against an allowed list, so prompt injection can at most make the tool query a different allowed region.

AWS MCP Server: Scope the credentials, not the script. Its aws___run_script sends SQL through the Data API as the caller’s mapped database user, so an administrator profile means administrator access. Four guardrails close that gap:

  • A dedicated agent profile in the MCP Proxy for AWS, for an agent role with read-only grants.
  • No way to switch database users. Allow redshift-serverless:GetCredentials or redshift:GetClusterCredentialsWithIAM, not redshift:GetClusterCredentials or Secrets Manager secrets, which let a caller choose another user.
  • Agent workgroups only. A service control policy in AWS Organizations denies Data API SQL through AWS managed MCP servers unless the workgroup is tagged agent-access=true. The same statement also works in an IAM policy or permissions boundary. SCPs skip the management account, so test on a sandbox OU first:
{
  "Sid": "AgentsQueryOnlyAgentWorkgroups",
  "Effect": "Deny",
  "Action": ["redshift-data:ExecuteStatement", "redshift-data:BatchExecuteStatement"],
  "Resource": "*",
  "Condition": {
    "Bool": { "aws:ViaAWSMCPService": "true" },
    "StringNotEquals": { "aws:ResourceTag/agent-access": "true" }
  }
}
  • AWS MCP Server only. To stop the agent’s shell from going around the server, deny redshift-data:* on the agent role unless the call comes through it. Also deny direct driver sign-in when aws:ViaAWSService is false: redshift-serverless:GetCredentials, redshift:GetClusterCredentialsWithIAM and redshift:GetClusterCredentials, plus GetIdentityCenterAuthToken when you use trusted identity propagation (access-layer/aws-mcp-server/deny-data-api-outside-aws-mcp.json). Attach it to agent roles or permission sets, not as an SCP. IAM can’t tell the local Redshift MCP Server from the AWS CLI, so on that path read-only database grants are the boundary:
{
  "Sid": "DataApiOnlyThroughAwsMcp",
  "Effect": "Deny",
  "Action": "redshift-data:*",
  "Resource": "*",
  "Condition": { "BoolIfExists": { "aws:ViaAWSMCPService": "false" } }
}

Bedrock Knowledge Bases: One database, listed tables, read-only grants. Inclusions (up to 50 tables) and exclusions narrow the generated SQL but aren’t a substitute for guardrails. A timeout of up to 200 seconds (60 in access-layer/bedrock-knowledge-base/kb-redshift-config.json) cancels the query. Scope the service role to one workgroup (and redshift-serverless:GetCredentials on Amazon Redshift Serverless) with SELECT on included tables only. Allow DescribeStatement, GetStatementResult, and CancelStatement on its own statements only. The sync fails without CancelStatement (access-layer/bedrock-knowledge-base/kb-service-role-policy.json).

Scope every write, and keep a human in the loop where it matters

Give an agent one scoped tool per action, never a generic write tool. A write can run on its own when its tool limits it to one validated, low-impact action, so approvals don’t slow routine work. Consequential, irreversible, or above-threshold writes need a human in the loop:

Mechanism How it works Can the agent bypass it?
Approve as data (recommended for data writes) The tool records a request. A people-only identity approves it No: Redshift enforces it
Approve in the agent The framework asks a person first Depends on the framework
Approve as code review Schema and infrastructure changes ship as reviewed code No: your pipeline enforces it

In the Act pattern, an agent writes to Redshift as part of a workflow, such as issuing a refund. Figure 4 shows how it combines both kinds of write:

  1. The agent’s only write tool, refund_order, runs as an identity that can only EXECUTE one procedure.
  2. The SECURITY DEFINER procedure checks the order and caps the amount at the remaining refundable amount and $500, locking support.refunds against concurrent over-refunds. A failed check writes nothing and returns an error the agent can report.
  3. Refunds up to $100 are approved. Larger ones wait for the refund_approvers role, through a second procedure that rejects self-approval.
  4. A payment process pays approved refunds only.
The Act pattern in four numbered steps. A support agent calls 1, the refund_order tool, which runs as an action identity that can call one procedure and read no tables. 2, the support.refund_order procedure in Redshift, checks the order and caps the amount at what is left and $500. Refunds up to $100 are approved, refunds from $100 to $500 wait for approval, and larger ones are rejected with an error and nothing written. 3, approve_refund, can be run only by refund_approvers, records the approver, and rejects self-approval. 4, the payment process, pays approved refunds only. A band explains that the agent’s identity has no table privileges, so it can’t go around the procedure.

Figure 4: A scoped action tool, with human approval above a threshold

The procedures and grants are in act-refund/refund_procedures.sql. Expose refund_order as its own tool, under an identity separate from the agent’s read path, so the read identity has no write privileges. Scheduled writes that no agent starts, such as a nightly Enrich job that labels tickets with Redshift ML, need a different control. Review the job’s SQL once before you deploy it, then run it under its own identity that no agent can use.

Control plane: Keep infrastructure changes out of agents’ reach

Control plane calls skip database grants, so IAM is the only gate, and an AI assistant’s credentials often reach its shell and the AWS CLI too.

Give every AI assistant a scoped identity. Run Kiro and other AI assistants, and their MCP servers, under an agent-role AWS profile with read-only control plane actions and a permissions boundary. Don’t allow redshift:Get*, which includes GetClusterCredentials. Share the profile through one AWS IAM Identity Center permission set (access-layer/shared/aws-config-example.ini). Role chaining caps sessions at 1 hour.

Deny infrastructure changes through managed MCP servers. With the aws:ViaAWSMCPService condition key, a service control policy can deny create, delete, modify, resize, pause, resume, reboot, restore, and tagging actions made through an agent but not directly (access-layer/shared/deny-infra-changes-through-agents.json). Tagging matters because the data plane deny trusts the agent-access tag.

Data layer: Decide what the agent can reach

The data layer decides what an agent can actually reach, on every query. As the Redshift MCP Server documentation puts it, “The database user’s privileges are the security boundary, not this server.”

Give every agent its own identity: One IAM role, database user, and database role. The IAM role agent-support signs in through the Data API as IAMR:agent-support. On Redshift Serverless, a RedshiftDbRoles tag can grant its database role at sign-in (the aws-samples repository shows the tag). Agents acting for a person can use trusted identity propagation. Grant the role after the user’s first sign-in:

-- One database role per agent: privileges go to the role, never to the user
CREATE ROLE agent_support;
GRANT USAGE ON SCHEMA sales TO ROLE agent_support;
GRANT SELECT ON sales.orders, sales.payments TO ROLE agent_support;
GRANT ROLE agent_support TO "IAMR:agent-support";

Then narrow what it sees (data-layer/fine-grained-access.sql):

  • Row-level security (RLS) filters rows only on tables you attach it to and makes a table default-deny, so attach policies for other roles first.
  • Column privileges remove columns, such as email.
  • Dynamic data masking transforms values, such as a card’s last four digits.

Limit what agents can discover. Turn on metadata security so catalog views and SHOW list only objects the role can access, so agents cannot plan around hidden tables (superusers and ACCESS CATALOG roles see all). Test on non-production first:

ALTER SYSTEM SET metadata_security = true; -- superuser only; applies to the whole cluster or workgroup
GRANT ACCESS CATALOG TO ROLE agent_monitoring; -- a monitoring role that must still see every object

Extend the rules to the lakehouse and other warehouses, so no path skips your filters. Redshift enforces Lake Formation permissions on Iceberg tables, including Amazon S3 Tables (a capability of Amazon S3), and federated permissions carry RLS and masking to an agent warehouse. Plain datashares do not carry RLS filters.

Treat free text as untrusted: Keep Redshift ML instructions in the static PROMPT prefix, and pass row text only as input.

Allow only approved models. The model’s IAM role decides at inference time, so allow bedrock:InvokeModel only on approved model ARNs. By default only superusers can create models. Grant CREATE MODEL to an ML platform role, with ASSUMEROLE ... FOR CREATE MODEL only on approved IAM roles.

Audit each agent’s reach with HAS_TABLE_PRIVILEGE (observe/detection/agent-table-privileges.sql). Until you revoke the default PUBLIC ASSUMEROLE grant, assume every user can use every IAM role attached to the warehouse. The AWS Identity and Access Management Access Analyzer (IAM Access Analyzer) check-access-not-granted check can fail deployments that grant redshift:GetClusterCredentials.

Test as the agent: Superusers and sys:secadmin bypass RLS.

Output guardrails: What the agent sees, keeps, and says

An agent has three outputs: results it reads, copies it keeps, and its answer. Redshift controls what enters the agent and its database-side copies. The application and agent layer govern the rest.

What the agent sees and says

The data layer’s policies double as output guardrails: results hold only permitted rows and readable columns, masked values stay masked, and Lake Formation filters do the same for Iceberg tables.

The sensitive information filtering in Bedrock Guardrails doesn’t inspect toolResult content, so mask in Redshift.

Cap result size too: provisioned clusters can abort on return_row_count, and the Redshift MCP Server’s MAX_RESULT_ROWS (1,000 by default) refuses larger results, protecting the context window, not compute.

Data-layer controls end when results leave Redshift, so check the answer too: Bedrock Guardrails can mask PII and flag ungrounded answers, and the generated SQL makes the answer traceable.

What the agent keeps: Database-side exports and copies

A copy starts with none of the source table’s policies, so treat every database-side write path as an output path:

Export path Guardrail
UNLOAD to Amazon S3 REVOKE ASSUMEROLE ON ALL FROM PUBLIC FOR ALL (which also blocks COPY and CREATE MODEL). Grant ASSUMEROLE ... FOR UNLOAD to pipeline roles only
CETAS and INSERT into external tables Agent roles own no external schemas
Writes to Apache Iceberg tables No Lake Formation write permissions for agent-facing roles
CTAS, SELECT INTO, and INSERT ... SELECT REVOKE CREATE ON SCHEMA public FROM PUBLIC. No CREATE or INSERT for agents
Datashares No datashare privileges for agent roles

The MCP Server’s read-only mode denies UNLOAD and CALL, but other paths bypass it, so privileges are the durable line (data-layer/exports-and-copies.sql). Audit exports in SYS_UNLOAD_HISTORY, SYS_QUERY_HISTORY, and CloudTrail S3 data events.

Figure 5 summarizes which layer controls each output.

Grid of output guardrails by layer for what the agent sees, keeps, and says. Agent layer: optional ApplyGuardrail screening of retrieved text, retained results governed by your application, and Bedrock Guardrails checks on the answer with the SQL kept. Access layer: the MCP Server MAX_RESULT_ROWS cap, and read-only mode that denies UNLOAD and CALL, with one approved write tool and approval above a threshold. Data layer: row-level security, column privileges, masking, and Lake Formation filters, plus ASSUMEROLE scoped for UNLOAD, s3:PutObject only to approved buckets, and no write or datashare privileges for agent roles. Data layer controls end when the result leaves Redshift.

Figure 5: What the agent sees, keeps, and says, and which layer controls each

The cost envelope: Predictable by design

Each agent question has three cost parts (Figure 1): model tokens at Bedrock rates, query compute on Redshift provisioned clusters or Amazon Redshift Serverless RPUs that you already pay for (older RA3 and DC2 clusters also bill lakehouse scans per terabyte), and data at rest. These compute guardrails are part of Redshift, and you pay only for the Redshift compute your queries use:

Guardrail Provisioned Serverless
Separate lane for agents WLM queues by user role Queues by user role, or a dedicated workgroup
Stop a runaway query Query monitoring rules that log, hop, or abort Workgroup query limits or queue rules that log or abort
Cap each statement statement_timeout per agent user Same as provisioned
Cap connections CONNECTION LIMIT and SESSION TIMEOUT per agent user Same as provisioned
Cap capacity Fixed node count. Concurrency scaling limits Max capacity in RPUs
Cap spend per period Concurrency scaling and Spectrum limits RPU-hour limits per day, week, or month

Set three ceilings on day one (Figure 6): a per-statement timeout for each agent user, max capacity on the agent workgroup, and a daily usage limit. Queue rules are checked every 10 seconds, so the timeout is the tighter cap.

Nested boxes showing the cost envelope, from outer to inner: per account (AWS Budgets alerts and AWS Cost Anomaly Detection), per Bedrock principal (cost by IAM principal in CUR 2.0 with activated agent tags, and a budget action that denies the agent’s Bedrock calls), per period (usage limits on RPU-hours), per workload (a dedicated workgroup or queue with max capacity), per query (query limits and statement_timeout), and per LLM call (a trusted tool fixes maxTokens, input size, and row count. Approved models only). The outer two act on billing data and the inner ones in real time. A band lists the three ceilings to set on day one.

Figure 6: The cost envelope. Each ceiling catches what the one inside it misses

Max capacity and the usage limit do different jobs (see Billing for on-demand compute capacity). Max capacity caps RPUs. The usage limit logs, alerts, or stops user queries once the period’s RPU-hours are used (cost-envelope/serverless-ceilings.sh). emit-metric only publishes a metric. Alarm on it, or choose Alert on the AWS Management Console.

Each Serverless workgroup or RG cluster accepts at most 2,000 connections. In our tests, Data API tools opened one connection per call, but an idle open driver transaction kept Serverless compute running until the 1-hour timeout. On Serverless, deny redshift-serverless:GetCredentials when aws:ViaAWSService is false. In our tests, this blocked driver sign-in, not direct Data API calls. On each non-superuser agent user (superusers ignore connection limits), set:

ALTER USER "IAMR:agent-support" SET statement_timeout TO 120000;
ALTER USER "IAMR:agent-support" CONNECTION LIMIT 10 SESSION TIMEOUT 300; -- size the limit to the agent's concurrency

Spend less per answer

  • Reuse work: The result cache, composition for new queries, and materialized views for repeated joins.
  • Right-size the engine. For lake-heavy agents, RG instances cost 30% less per vCPU than RA3, with no per-terabyte scanning charges.
  • Spend and store less. Every byte saved is a byte agents don’t read. For local tables, keep ENCODE AUTO so Redshift selects optimal column compression by data type. Iceberg tables use zstd compression by default, helping reduce storage and scan costs (cost-envelope/storage-compression.sql).

Control model spend per agent

Bedrock attributes inference cost to the calling IAM principal (Introducing granular cost attribution for Amazon Bedrock). Since Redshift ML calls Bedrock with the role in CREATE EXTERNAL MODEL, give each agent its own role. A trusted tool, the only identity allowed to run the model, fixes maxTokens and bounds input (cost-envelope/model-spend-summarizer.sql), and a budget action can deny bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream to the role as a period-level brake, not a real-time stop.

Per-agent users make SYS_QUERY_HISTORY a compute ledger. Tag both ledgers per AGENTCOST05-BP01. Because the Redshift MCP Server in read-only mode refuses SET query_group, attribute by IAM role.

Observe and prove

Security reviewers ask what the agent did, and FinOps reviewers ask what it cost. Per-agent identity answers both.

Figure 7 shows how the following records feed back into tighter guardrails.

A closed loop of five numbered steps. 1, Attribute: agent-* names for agent identities plus listed exceptions, self-reported application names for tracking only, and an estimated cost on a shared workgroup. 2, Audit: SYS_QUERY_HISTORY for what ran, CloudTrail for who and when, and AgentCore and Bedrock logs joined on role and time. 3, Retain: system tables kept in S3 Tables past 7 days, to profile agents across model changes. 4, Review: run as a monitoring identity with sys:monitor, audit IGNORE RLS and tag-granted roles, and alarm on shadow agents and loops. 5, Tighten: give a shadow agent its own identity or block it, stop loops with step and retry limits, and narrow grants and ceilings.

Figure 7: The observe and prove loop

What ran, and who called it. SYS_QUERY_HISTORY lists each agent user’s queries, and an agent- prefix on every agent identity lets one filter cover them. CloudTrail adds who and when. With trusted identity propagation, both identify the person.

Which client ran it. SYS_CONNECTION_LOG records each session’s self-reported application name, which we use for tracking, not as a control. CloudTrail records a shared role’s session name and SourceIdentity, which name a person only when your identity provider sets them or a trust policy condition key requires them (observe/attribution/queries-per-client.sql).

Find shadow agents, which are MCP calls from an identity without the agent- prefix. The application_name setting and the CloudTrail userAgent or userIdentity.invokedBy fields mark MCP traffic (observe/detection/shadow-agents.sql).

Keep a long-term record: System table integration with Amazon S3 Tables keeps system views, normally held 7 days, in read-only Iceberg tables as long as you set. Protect Bedrock invocation logs, which hold unmasked input, with Amazon CloudWatch Logs data protection.

Catch runaway loops with a scheduled check for repeated query shapes and errors (observe/detection/runaway-loops.sql).

Conclusion

In this post, we showed how the layered guardrail model runs agentic analytics on Amazon Redshift, with most controls built into Redshift. The access layer keeps a human in the loop for consequential writes, and the data layer decides what each query can reach.

To get started:

  1. Connect safely. Run the Amazon Redshift MCP Server read-only with a per-agent IAM role, and install redshift-guide from the Agent Toolkit for AWS.
  2. Bound the data. Grant only needed tables and columns, attach RLS and masking, revoke ASSUMEROLE and CREATE from PUBLIC, and test metadata security on non-production first.
  3. Bound the spend. Set a statement timeout, max capacity, and a daily usage limit, and test that the usage-limit alarm fires.

To go further, read Query Amazon Redshift using natural language with Kiro and Integrate Amazon Bedrock with Amazon Redshift ML. The complete SQL and CLI samples from this post are in the aws-samples repository. The repository README lists the prerequisites and the clean-up steps for what the samples create. Leave questions or feedback in the comments.


About the authors

Jason Pedreza

Jason Pedreza

Jason is a Senior Analytics Specialist Solutions Architect at AWS, where he helps customers architect petabyte-scale data platforms for AI-driven analytics. He focuses on modern lakehouse design, large-scale data migrations, and building the data foundations that power ML and generative AI applications.

Dan Jin

Dan Jin

Dan is a Sr. Analytics Specialist Solutions Architect at AWS based out of the San Francisco Bay Area. With over 20 years of experience in database and data analytics products, he has deep expertise in data system architecture and technical solution design. He focuses on helping customers modernize their data warehouses on Amazon Redshift and adopt AI-driven analytics.

Amazon S3 Tables with dbt: Iceberg-based data transformations using AWS Glue

Post Syndicated from Yukiko Iwazumi original https://aws.amazon.com/blogs/big-data/amazon-s3-tables-with-dbt-iceberg-based-data-transformations-using-aws-glue/

Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), deliver a cloud object store with built-in Apache Iceberg support and streamline storing tabular data at scale. When you integrate them with dbt and AWS Glue, you can run SQL-based data transformations with the reliability and performance of a modern lakehouse, bringing software engineering best practices to the data space.

This post demonstrates how to configure and implement dbt transformations using Amazon S3 Tables with the dbt-glue adapter. Starting with version 1.12.4, the dbt-glue adapter adds incremental merge support for S3 Tables, which the patterns in this post build on. We cover the specific configuration requirements for the dbt-glue adapter and practical implementation patterns built on S3 Tables incremental models: the merge strategy to keep the latest state per record and the append strategy to build an audit-friendly change history.

Example scenario

In this walkthrough, we simulate an ecommerce system that tracks customer profiles and their orders. We first run the pipeline against an initial dataset, then simulate an update to the upstream source data and run it again to demonstrate incremental updates.

Our example scenario follows a data pipeline architecture that uses S3 Tables throughout. Raw ecommerce data (customers.csv and orders.csv) lands in a regular S3 bucket as comma-separated values (CSV). dbt then transforms it through two layers of S3 Tables. A staging layer cleanses and standardizes each source, and a mart layer produces analytics-ready dimension and fact tables. You can query the resulting mart data from a variety of analytics and business intelligence (BI) tools such as Amazon Athena and Amazon Quick Sight. In this post, we query it from Athena.

The mart layer demonstrates two incremental strategies side by side: the fact_orders model uses merge to keep the latest state per order, while fact_order_history uses append to accumulate a full change history.

The data pipeline flow looks like the following chart:

Source Data (Regular S3 - CSV)
    └─ customers.csv, orders.csv
    ↓
Staging Models (S3 Tables)
    ├─ stg_customers
    └─ stg_orders
    ↓
Mart Models (S3 Tables)
    ├─ dim_customers
    ├─ fact_orders                (incremental with merge --- latest state per order)
    └─ fact_order_history         (incremental with append --- change history)
    ↓
Consumption Layer (Athena, Quick Sight, BI tools)

The following diagram shows the architecture behind this pipeline: dbt-glue submits each model as a Spark SQL statement to an AWS Glue interactive session, which reads from the raw S3 bucket and writes the staging and mart tables as S3 Tables within the same data lake.

dbt-glue submits each model to an AWS Glue interactive session that reads the raw S3 bucket and writes staging and mart tables as S3 Tables

Figure 1: Architecture of the dbt transformation pipeline on Amazon S3 Tables with AWS Glue

Prerequisites

Before you begin, make sure that you have the following prerequisites:

Create S3 Tables namespace and AWS Glue databases

In this section, you set up the AWS infrastructure that dbt-glue needs: an S3 table bucket and namespace to hold your dbt models, and an AWS Glue database for your source data.

First, set your environment variables. Replace YOUR_ACCOUNT_ID, YOUR-REGULAR-BUCKET, and YOUR_Glue_Role_ARN with your data:

export DBT_GLUE_REGION="us-east-1"
export AWS_ACCOUNT_ID="<YOUR_ACCOUNT_ID>"
export S3_TABLES_BUCKET_NAME="dbt-glue-s3-tables-bucket"
export DBT_ROLE_ARN="<YOUR_Glue_Role_ARN>"
export DBT_S3_LOCATION="s3://<YOUR-REGULAR-BUCKET>/glue/data/"
export DBT_S3_TABLES_BUCKET="${AWS_ACCOUNT_ID}:s3tablescatalog/${S3_TABLES_BUCKET_NAME}"

The DBT_S3_TABLES_BUCKET format is AWS_ACCOUNT_ID:s3tablescatalog/BUCKET_NAME, not an Amazon Resource Name (ARN).

Second, create an S3 table bucket and namespace:

  1. Create S3 table bucket.
    aws s3tables create-table-bucket \
      --region $DBT_GLUE_REGION \
      --name $S3_TABLES_BUCKET_NAME

  2. Create namespace within the table bucket.
    aws s3tables create-namespace \
      --region $DBT_GLUE_REGION \
      --table-bucket-arn "arn:aws:s3tables:${DBT_GLUE_REGION}:${AWS_ACCOUNT_ID}:bucket/${S3_TABLES_BUCKET_NAME}" \
      --namespace dbt_glue_s3_tables

    Note that the S3 Tables namespace is where the S3 Tables corresponding to your dbt models will be created. This is different from regular S3 buckets. Because the “Integration with AWS analytics services” option for S3 Tables is enabled, creating the namespace automatically creates a corresponding AWS Glue database. Verify that it exists in the next step:

  3. Verify that the database exists.
    aws glue get-database \
      --region $DBT_GLUE_REGION \
      --catalog-id "$DBT_S3_TABLES_BUCKET" \
      --name dbt_glue_s3_tables

    Third, create an AWS Glue database for source data tables:

  4. Create an AWS Glue database for source tables.
    aws glue create-database \
      --region $DBT_GLUE_REGION \
      --database-input Name=raw_ecommerce,Description='Raw e-commerce data'

Create sample dataset

In this section, you create sample ecommerce data to transform with dbt: two CSV files (customers and orders), upload them to a general purpose S3 bucket, and register them as AWS Glue tables that your dbt models can read from.

Note: All sample data in this post is synthetic. It was generated to mimic realistic patterns for demonstration purposes only and doesn’t represent any real individuals or transactions.

Complete following steps to create a sample dataset:

  1. Create a directory for sample data and save the following two CSV files:
    customers.csv:

    1,[email protected],Alejandro,Rosalez,2024-01-15,Premium
    2,[email protected],Ana Carolina,Silva,2024-02-20,Standard
    3,[email protected],Arnav,Desai,2024-03-10,Premium
    4,[email protected],Carlos,Salazar,2024-04-05,Standard
    5,[email protected],Diego,Ramirez,2024-05-12,Basic

    orders.csv:

    1001,1,2024-06-01,150.00,completed,2024-06-01T10:00:00
    1002,2,2024-06-02,75.50,completed,2024-06-02T10:00:00
    1003,1,2024-06-03,200.00,completed,2024-06-03T10:00:00
    1004,3,2024-06-04,125.75,pending,2024-06-04T10:00:00
    1005,4,2024-06-05,89.99,completed,2024-06-05T10:00:00

  2. Create the bucket for source data if it doesn’t exist, and upload the CSV files to the S3 bucket. Replace YOUR-SOURCE-DATA-BUCKET with your bucket name.
    export SOURCE_BUCKET="<YOUR-SOURCE-DATA-BUCKET>"
    
    # Create S3 bucket for source data
    aws s3 mb s3://${SOURCE_BUCKET} --region ${DBT_GLUE_REGION}
    
    # Upload CSV files
    aws s3 cp customers.csv s3://${SOURCE_BUCKET}/raw/customers/ --region ${DBT_GLUE_REGION}
    aws s3 cp orders.csv s3://${SOURCE_BUCKET}/raw/orders/ --region ${DBT_GLUE_REGION}

  3. Create an AWS Glue database and tables for source data.
    export SOURCE_DATABASE="raw_ecommerce"
    
    # Create Glue table for customers
    aws glue create-table \
      --database-name ${SOURCE_DATABASE} \
      --table-input '{
        "Name": "customers",
        "StorageDescriptor": {
          "Columns": [
            {"Name": "customer_id", "Type": "bigint"},
            {"Name": "email", "Type": "string"},
            {"Name": "first_name", "Type": "string"},
            {"Name": "last_name", "Type": "string"},
            {"Name": "registration_date", "Type": "string"},
            {"Name": "customer_segment", "Type": "string"}
          ],
          "Location": "s3://'${SOURCE_BUCKET}'/raw/customers/",
          "InputFormat": "org.apache.hadoop.mapred.TextInputFormat",
          "OutputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
          "SerdeInfo": {
            "SerializationLibrary": "org.apache.hadoop.hive.serde2.OpenCSVSerde",
            "Parameters": {
              "separatorChar": ","
            }
          }
        },
        "TableType": "EXTERNAL_TABLE"
      }' \
      --region ${DBT_GLUE_REGION}
    
    # Create Glue table for orders
    aws glue create-table \
      --database-name ${SOURCE_DATABASE} \
      --table-input '{
        "Name": "orders",
        "StorageDescriptor": {
          "Columns": [
            {"Name": "order_id", "Type": "bigint"},
            {"Name": "customer_id", "Type": "bigint"},
            {"Name": "order_date", "Type": "string"},
            {"Name": "order_amount", "Type": "double"},
            {"Name": "order_status", "Type": "string"},
            {"Name": "updated_at", "Type": "string"}
          ],
          "Location": "s3://'${SOURCE_BUCKET}'/raw/orders/",
          "InputFormat": "org.apache.hadoop.mapred.TextInputFormat",
          "OutputFormat": "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat",
          "SerdeInfo": {
            "SerializationLibrary": "org.apache.hadoop.hive.serde2.OpenCSVSerde",
            "Parameters": {
              "separatorChar": ","
            }
          }
        },
        "TableType": "EXTERNAL_TABLE"
      }' \
      --region ${DBT_GLUE_REGION}

Install dbt, the dbt CLI, and the dbt adapter

The dbt CLI is a command-line interface for running dbt projects. It’s free to use and available as an open source project. Before installing dbt-core, make sure your Python version is 3.10, 3.11, or 3.12.

  1. Install dbt and the dbt CLI with the following code:
    $ pip3 install --no-cache-dir dbt-core

    For more information, refer to How to install dbt, What is dbt?, Viewpoint, and What version of Python can I use?.

  2. Install the dbt-glue adapter with the following code:
    $ pip3 install --no-cache-dir dbt-glue

    Note that you need to install dbt-glue 1.12.4 or higher, which adds support for incremental append and merge models on S3 Tables.

Create a dbt project

In this section, you create a new dbt project using dbt init, which generates the standard directory structure (such as models, macros, and tests) that you will populate with the S3 Tables models in the next sections.

  1. Run the dbt init command to create and initialize a new empty dbt project.
    $ dbt init

  2. For a project name, enter dbt_glue_s3_tables.
  3. For a database, choose [1] glue.

Please ignore dbt_project.yml file [ERROR not found] message as we are going to create it in the next step.

Configure dbt profile

The dbt profile configuration is critical for S3 Tables integration. In this section, you create a dbt profile that connects to AWS Glue Interactive Sessions and configures Spark to read and write S3 Tables through the Iceberg catalog, then validate the connection with dbt debug.

  1. Create ~/.dbt/profiles.yml with the following structure:
    dbt_glue_s3_tables:
      target: dev
      outputs:
        dev:
          type: glue
          glue_version: "6.0"
          role_arn: "{{ env_var('DBT_ROLE_ARN') }}"
          region: "{{ env_var('DBT_GLUE_REGION') }}"
          workers: 2
          worker_type: G.1X
          schema: dbt_glue_s3_tables
    
          # S3 Tables Spark configuration (space-delimited format)
          conf: "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog --conf spark.sql.catalog.spark_catalog.type=glue --conf spark.sql.catalog.glue_catalog=org.apache.iceberg.spark.SparkCatalog --conf spark.sql.catalog.glue_catalog.warehouse={{ env_var('DBT_S3_LOCATION') }} --conf spark.sql.catalog.glue_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog --conf spark.sql.catalog.glue_catalog.io-impl=org.apache.iceberg.aws.s3.S3FileIO --conf spark.sql.catalog.glue_catalog.glue.id={{ env_var('DBT_S3_TABLES_BUCKET') }} --conf spark.sql.defaultCatalog=glue_catalog"
    
          # Enable S3 Tables support
          datalake_formats: "iceberg"

  2. Run the following command to validate the configurations:
    cd dbt_glue_s3_tables
    dbt debug --profiles-dir ~/.dbt/

    Expected output is Connection test: [OK connection ok].

Configure dbt project

The dbt_project.yml configuration determines which models use S3 Tables. Create or update dbt_glue_s3_tables/dbt_project.yml with the following structure:

name: 'dbt_glue_s3_tables'
version: '1.0.0'
config-version: 2
profile: 'dbt_glue_s3_tables'
model-paths: ["models"]
test-paths: ["tests"]
macro-paths: ["macros"]
models:
  dbt_glue_s3_tables:
    # S3 Tables for staging
    staging:
      +materialized: table
      +file_format: 's3tables'  # S3 Tables for staging
      +table_properties:
        'format': 'iceberg'
        'write.format.default': 'parquet'
        'write.parquet.compression-codec': 'snappy'

    # S3 Tables for marts
    marts:
      +materialized: table
      +file_format: 's3tables'  # S3 Tables for marts
      +table_properties:
        'format': 'iceberg'
        'write.format.default': 'parquet'
        'write.parquet.compression-codec': 'snappy'

Create dbt models

In this section, you build out the full model layer: a sources.yml that registers your raw AWS Glue tables, two staging models that clean and prepare the customer and order data, and three mart models that turn those staged data into a dimension table and two fact tables, one using incremental merge and one using incremental append.

First, create sources.yml defines your raw data tables:

  1. Delete the example folder under dbt_glue_s3_tables/models. This is an example folder created by default and unnecessary now.
  2. Define source tables in models/sources.yml with the following structure:
    version: 2
    
    sources:
      - name: raw_ecommerce
        description: "Raw e-commerce data in regular S3 buckets"
        schema: raw_ecommerce
    
        tables:
          - name: customers
            description: "Customer master data"
            columns:
              - name: customer_id
                tests:
                  - unique
                  - not_null
              - name: email
                tests:
                  - unique
                  - not_null
    
          - name: orders
            description: "Order transactions"
            columns:
              - name: order_id
                tests:
                  - unique
                  - not_null
              - name: customer_id
                tests:
                  - not_null
                  - relationships:
                      arguments:
                        to: source('raw_ecommerce', 'customers')
                        field: customer_id
              - name: updated_at
                tests:
                  - not_null

    This sources.yml defines your raw data tables, so you can reference them with the {{ source() }} function in your models. With this file, you can also define data tests like unique, not_null, and referential integrity checks in one central place.

    Second, create the staging and mart models. Staging models are where you unpack and transform raw data. Mart models are what downstream consumers actually query.

  3. Create models/staging/stg_customers.sql:
    -- Staging model as S3 Tables with incremental merge
    {{ config(
        materialized='incremental',
        incremental_strategy='merge',
        unique_key='customer_id',
        on_schema_change='sync_all_columns',
        file_format='s3tables'
    ) }}
    
    SELECT
        customer_id,
        LOWER(TRIM(email)) AS email,
        first_name,
        last_name,
        CAST(registration_date AS DATE) AS registration_date,
        customer_segment
    FROM spark_catalog.{{ source('raw_ecommerce', 'customers') }}
    WHERE customer_id IS NOT NULL

    stg_customers.sql reads from the raw customers source, applies light cleansing (email normalization, date casting, null filtering), and writes to S3 Tables using an incremental merge on customer_id to update existing records rather than duplicating them.

  4. Create models/staging/stg_orders.sql:
    -- Staging model as S3 Tables with incremental merge
    {{ config(
        materialized='incremental',
        incremental_strategy='merge',
        unique_key='order_id',
        on_schema_change='sync_all_columns',
        file_format='s3tables'
    ) }}
    
    SELECT
        order_id,
        customer_id,
        CAST(order_date AS DATE) AS order_date,
        CASE
            WHEN LOWER(TRIM(order_status)) = 'completed'  THEN 'completed'
            WHEN LOWER(TRIM(order_status)) = 'shipped'    THEN 'shipped'
            WHEN LOWER(TRIM(order_status)) = 'processing' THEN 'processing'
            WHEN LOWER(TRIM(order_status)) = 'cancelled'  THEN 'cancelled'
            ELSE 'unknown'
        END AS status,
        CAST(order_amount AS DECIMAL(10,2)) AS total_amount,
        EXTRACT(YEAR  FROM CAST(order_date AS DATE)) AS order_year,
        EXTRACT(MONTH FROM CAST(order_date AS DATE)) AS order_month,
        CAST(updated_at AS TIMESTAMP) AS updated_at
    FROM spark_catalog.{{ source('raw_ecommerce', 'orders') }}
    WHERE order_id IS NOT NULL

    stg_orders.sql reads from the raw orders source, normalizes status to a fixed set of values (defaulting to unknown for anything unexpected), casts order_date, total_amount, and updated_at to their proper types, and derives order_year and order_month for downstream partitioning.

  5. Create models/marts/dimensions/dim_customers.sql:
    -- Dimension table using S3 Tables
    {{ config(
        materialized='table',
        file_format='s3tables',
        partition_by=['registration_year'],
        table_properties={
            'format': 'iceberg',
            'write.format.default': 'parquet'
        }
    ) }}
    
    SELECT
        customer_id,
        email,
        first_name,
        last_name,
        registration_date,
        YEAR(registration_date) as registration_year,
        customer_segment,
        CASE
            WHEN customer_segment = 'Premium' THEN 1
            WHEN customer_segment = 'Regular' THEN 2
            ELSE 3
        END as segment_priority,
        CURRENT_TIMESTAMP() as dbt_updated_at
    FROM {{ ref('stg_customers') }}

  6. Create models/marts/facts/fact_orders.sql:
    -- Fact table using S3 Tables with incremental merge
    {{ config(
        materialized='incremental',
        file_format='s3tables',
        incremental_strategy='merge',
        unique_key='order_id',
        partition_by=['order_year', 'order_month'],
        table_properties={
            'format': 'iceberg',
            'write.format.default': 'parquet'
        }
    ) }}
    
    SELECT
        o.order_id,
        o.customer_id,
        o.order_date,
        YEAR(o.order_date)  AS order_year,
        MONTH(o.order_date) AS order_month,
        o.status,
        o.total_amount,
        c.customer_segment,
        c.segment_priority,
        CURRENT_TIMESTAMP() AS dbt_updated_at
    FROM {{ ref('stg_orders') }} o
    LEFT JOIN {{ ref('dim_customers') }} c
      ON o.customer_id = c.customer_id

  7. Create models/marts/facts/fact_order_history.sql:
    -- Fact table using S3 Tables with incremental append
    {{ config(
        materialized='incremental',
        file_format='s3tables',
        incremental_strategy='append',
        partition_by=['event_year', 'event_month'],
        table_properties={
            'format': 'iceberg',
            'write.format.default': 'parquet'
        }
    ) }}
    
    SELECT
        order_id,
        customer_id,
        status,
        order_date,
        total_amount,
        updated_at,
        CURRENT_TIMESTAMP() AS recorded_at,
        YEAR(updated_at)    AS event_year,
        MONTH(updated_at)   AS event_month
    FROM {{ ref('stg_orders') }}
    
    {% if is_incremental() %}
        WHERE updated_at > (SELECT MAX(updated_at) FROM {{ this }})
    {% endif %}

Run the dbt project

Now it’s time to run the dbt project. Complete following steps:

  1. To run the project dbt, you should be in the project folder:
    cd dbt_glue_s3_tables

  2. Run the dbt command:
    dbt run --profiles-dir ~/.dbt/ --full-refresh

This command starts an AWS Glue interactive session, compiles the dbt models, and materializes all five tables (stg_customers, stg_orders, dim_customers, fact_orders, and fact_order_history) in your S3 Tables namespace. The --full-refresh flag rebuilds the incremental models from scratch, so it’s the right choice for the first run.

Query using Amazon Athena

Before testing the incremental merge and append behavior in the next section, confirm what the initial --full-refresh run loaded into the mart tables. Query the target tables.

  1. In the Athena console, choose the data source and database that correspond to your S3 Tables catalog:
    • Catalog: s3tablescatalog/dbt-glue-s3-tables-bucket.
    • Database: dbt_glue_s3_tables.
  2. Run the following query to view the data loaded into the fact_orders table:
    SELECT * FROM dbt_glue_s3_tables.fact_orders ORDER BY order_id;

    The result returns the five order records:

    #    order_id    customer_id    order_date    order_year    order_month    status    total_amount    customer_segment    segment_priority    dbt_updated_at
    1    1001    1    2024-06-01    2024    6    completed    150.00    Premium    1    2026-10-16 09:41:46.590373 UTC
    2    1002    2    2024-06-02    2024    6    completed    75.50    Standard    3    2026-10-16 09:41:46.590373 UTC
    3    1003    1    2024-06-03    2024    6    completed    200.00    Premium    1    2026-10-16 09:41:46.590373 UTC
    4    1004    3    2024-06-04    2024    6    unknown    125.75    Premium    1    2026-10-16 09:41:46.590373 UTC
    5    1005    4    2024-06-05    2024    6    completed    89.99    Standard    3    2026-10-16 09:41:46.590373 UTC

  3. Run the following query to view the data loaded into the fact_order_history table:
    SELECT * FROM dbt_glue_s3_tables.fact_order_history ORDER BY updated_at;

    The result returns the five order history records.

    #    order_id    customer_id    status    order_date    total_amount    updated_at    recorded_at    event_year    event_month
    1    1001    1    completed    2024-06-01    150.00    2024-06-01 10:00:00.000000 UTC    2026-10-16 09:41:22.801835 UTC    2024    6
    2    1002    2    completed    2024-06-02    75.50    2024-06-02 10:00:00.000000 UTC    2026-10-16 09:41:22.801835 UTC    2024    6
    3    1003    1    completed    2024-06-03    200.00    2024-06-03 10:00:00.000000 UTC    2026-10-16 09:41:22.801835 UTC    2024    6
    4    1004    3    unknown    2024-06-04    125.75    2024-06-04 10:00:00.000000 UTC    2026-10-16 09:41:22.801835 UTC    2024    6
    5    1005    4    completed    2024-06-05    89.99    2024-06-05 10:00:00.000000 UTC    2026-10-16 09:41:22.801835 UTC    2024    6

Incremental merge and append on Iceberg table

After the initial --full-refresh run succeeds, verify that the merge and append incremental strategies behave as expected when the source data changes. This demonstrates the row-level update capability that Iceberg on S3 Tables provides.

Update the sample source data

To simulate a real-world scenario where upstream source data changes over time (for example, a customer updating their profile, or new customers signing up), modify customers.csv to include both an update to an existing record and a new record.

    1. Update two CSV files:

customers.csv:

1,[email protected],Alejandro,Rosalez,2024-01-15,Premium
2,[email protected],Ana Carolina,Silva,2024-02-20,Standard
3,[email protected],Arnav,Desai,2024-03-10,Premium
4,[email protected],Carlos,Salazar,2024-04-05,Standard
5,[email protected],Diego,Ramirez,2024-05-12,Basic
6,[email protected],Jane,Doe,2024-07-01,Premium

Changes:

      • customer_id=1: email updated to [email protected] (to test the merge UPDATE).
      • customer_id=6: new record added (to test the merge INSERT).

orders.csv:

1001,1,2024-06-01,150.00,completed,2024-06-01T10:00:00
1002,2,2024-06-02,75.50,completed,2024-06-02T10:00:00
1003,1,2024-06-03,200.00,completed,2024-06-03T10:00:00
1004,3,2024-06-04,125.75,completed,2024-06-11T09:30:00
1005,4,2024-06-05,89.99,completed,2024-06-05T10:00:00
1006,6,2024-07-02,45.00,completed,2024-07-02T10:00:00

Changes:

    • order_id=1004: status updated from pending to completed with a later updated_at (to test the merge UPDATE).
    • order_id=1006: new record added with a later updated_at, associated with the new customer customer_id=6 (to test both the merge INSERT in fact_orders and the append in fact_order_history).
  1. Upload the updated files to S3.
    # Upload CSV files
    aws s3 cp customers.csv s3://${SOURCE_BUCKET}/raw/customers/ --region ${DBT_GLUE_REGION}
    aws s3 cp orders.csv s3://${SOURCE_BUCKET}/raw/orders/ --region ${DBT_GLUE_REGION}

Re-run dbt project

  1. Run the following command.
    dbt run --profiles-dir ~/.dbt/

By running without --full-refresh, dbt runs its normal incremental logic instead of rebuilding the tables from scratch. In the staging layer, stg_customers and stg_orders apply the changes with a MERGE INTO statement that updates customer_id=1 and order_id=1004 in place and inserts customer_id=6 and order_id=1006. Downstream, fact_orders (merge) reflects the 1004 update and the 1006 insert as the latest state, while fact_order_history (append) records both 1004 and 1006 as new history rows.

Verify the results in Amazon Athena

After the re-run completes, use Athena to confirm that the merge and append strategies behaved as expected.

  1. Run the following query to view the updated data in the fact_orders table:
    SELECT * FROM dbt_glue_s3_tables.fact_orders ORDER BY order_id;

    fact_orders returns 6 rows (one per order), with 1004 now completed and 1006 added.

    #    order_id    customer_id    order_date    order_year    order_month    status    total_amount    customer_segment    segment_priority    dbt_updated_at
    1    1001    1    2024-06-01    2024    6    completed    150.00    Premium    1    2026-10-16 10:06:55.944983 UTC
    2    1002    2    2024-06-02    2024    6    completed    75.50    Standard    3    2026-10-16 10:06:55.944983 UTC
    3    1003    1    2024-06-03    2024    6    completed    200.00    Premium    1    2026-10-16 10:06:55.944983 UTC
    4    1004    3    2024-06-04    2024    6    completed    125.75    Premium    1    2026-10-16 10:06:55.944983 UTC
    5    1005    4    2024-06-05    2024    6    completed    89.99    Standard    3    2026-10-16 10:06:55.944983 UTC
    6    1006    6    2024-07-02    2024    7    completed    45.00    Premium    1    2026-10-16 10:06:55.944983 UTC

  2. Run the following query to view the updated data in the fact_order_history table:
    SELECT * FROM dbt_glue_s3_tables.fact_order_history ORDER BY updated_at;

    fact_order_history grows from 5 rows to 7 rows on the second run (the 2 rows whose updated_at advanced: 1004 and 1006). Order 1004 now has two history rows: the original unknown status and the new completed status.

    #    order_id    customer_id    status    order_date    total_amount    updated_at    recorded_at    event_year    event_month
    1    1001    1    completed    2024-06-01    150.00    2024-06-01 10:00:00.000000 UTC    2026-10-16 09:41:22.801835 UTC    2024    6
    2    1002    2    completed    2024-06-02    75.50    2024-06-02 10:00:00.000000 UTC    2026-10-16 09:41:22.801835 UTC    2024    6
    3    1003    1    completed    2024-06-03    200.00    2024-06-03 10:00:00.000000 UTC    2026-10-16 09:41:22.801835 UTC    2024    6
    4    1004    3    unknown    2024-06-04    125.75    2024-06-04 10:00:00.000000 UTC    2026-10-16 09:41:22.801835 UTC    2024    6
    5    1005    4    completed    2024-06-05    89.99    2024-06-05 10:00:00.000000 UTC    2026-10-16 09:41:22.801835 UTC    2024    6
    6    1004    3    completed    2024-06-04    125.75    2024-06-11 09:30:00.000000 UTC    2026-10-16 10:06:35.291993 UTC    2024    6
    7    1006    6    completed    2024-07-02    45.00    2024-07-02 10:00:00.000000 UTC    2026-10-16 10:06:35.291993 UTC    2024    7

Clean up

To avoid ongoing charges, clean up the resources you created during this walkthrough. Complete these steps in the specified order to facilitate proper resource deletion.

  1. Drop the tables dbt created in the S3 Tables namespace.
  2. Remove the namespace from the table bucket.
  3. Delete the table bucket.
  4. Delete the Data Catalog database.
  5. Delete the data this walkthrough wrote to your S3 bucket: the dbt warehouse data under s3://<YOUR-REGULAR-BUCKET>/glue/data/ and the sample CSV files under s3://<YOUR-REGULAR-BUCKET>/raw/. Leave the bucket itself in place if you use it for other workloads.
  6. (Optional) Remove the local dbt project directory (dbt_glue_s3_tables) and the sample customers.csv / orders.csv files.

Conclusion

In this post, we showed how to build an end-to-end dbt transformation pipeline on Amazon S3 Tables with AWS Glue 6.0, using a staging and mart architecture. At the core of this pipeline are incremental models on S3 Tables, which let us transform and maintain data incrementally with SQL. We demonstrated two incremental strategies working together: the merge strategy to keep the latest state per order in fact_orders, and the append strategy to build an audit-friendly change history in fact_order_history.

S3 Tables support in the dbt-glue adapter is currently experimental (as of version 1.12.4) and will continue to evolve in upcoming releases. As with any experimental feature, we recommend thoroughly testing it in your own environment before use.

To learn more, see Amazon S3 Tables and the dbt-glue adapter.


About the authors

Yukiko Iwazumi

Yukiko Iwazumi

Yukiko is a Cloud Support Engineer on the AWS Big Data Support team. She enjoys helping customers build and optimize their ETL workloads. In her spare time, she enjoys taking walks and playing games.

Chiho Sugimoto

Chiho Sugimoto

Chiho is a Go-to-market Data & AI solutions architect at AWS in Japan. She is passionate about helping customers build data lakes using ETL workloads. She loves planetary science and enjoys studying the asteroid Ryugu on weekends.

Neela Kulkarni

Neela Kulkarni

Neela is a Senior Solutions Architect at Amazon Web Services (AWS). She primarily serves independent software vendors (ISVs), helping them build and scale solutions in the Data, Analytics space by providing architectural guidance and best-practice recommendations for new and existing workloads.

Yota Hamaoka

Yota Hamaoka

Yota is a Senior Big Data Specialist Solutions Architect at Amazon Web Services. He helps customers accelerate their analytics journeys by leveraging his expertise in data lakes, big data processing, and data warehousing.

Srishti Mittal

Srishti Mittal

Srishti is a Product Manager at AWS, with a focus on making open data lakes performant and interoperable across analytics engines. She leads product strategy for open table formats such as Apache Iceberg, partnering with customers and field teams to turn real-world data lake challenges into product capabilities.

[$] Added indirection allows for larger BPF programs

Post Syndicated from daroc original https://lwn.net/Articles/1106537/

The BPF verifier has historically worked by tracing every possible path of
execution through a program, checking, at each instruction, that various safety
properties hold. This works, but it can result in an exponential increase in the
number of states that need checking in programs with complicated control-flow.
Alexei Starovoitov, one of the kernel’s core BPF maintainers, has shared

a patch set
that substantially reworks how the verifier tracks states, in
order to reduce the size of that exponential explosion.

PostgreSQL 19 released

Post Syndicated from jzb original https://lwn.net/Articles/1108561/

The PostgreSQL Global Development Group has announced
the release of PostgreSQL 19. Notable features in this release include the
introduction of the REPACK (CONCURRENTLY) command to optimize tables
without disruption to ongoing operations, better control over plan stability and
query performance with the new pg_plan_advice extension, and a number
of performance improvements.

In September, LWN covered the
removal of several features initially slated for this release. See the release notes for
a full list of new features and changes.

What Is Preemptive MDR? A Guide For Security Leaders

Post Syndicated from Emma Burdett original https://www.rapid7.com/blog/post/dr-what-is-preemptive-mdr-security-leaders-guide

When an MDR provider contains a compromised endpoint, your team needs to know whether the weakness that enabled the attack exists elsewhere and who is responsible for addressing it. Without that follow-through, the immediate incident may be resolved while another attacker still has a way in.

Preemptive managed detection and response (MDR) is Rapid7’s approach to connecting exposure context, threat intelligence, AI-assisted investigation, and human expertise. It supports proactive cybersecurity by helping teams prioritize weaknesses before exploitation, investigate threats with a clearer understanding of risk, and use their findings to strengthen defenses.

How does preemptive MDR differ from traditional MDR?

Managed detection and response provides security specialists who monitor, investigate, and respond to threats around the clock. Many services also include proactive threat hunting and recommendations to improve security.

Preemptive MDR connects that work more directly to what makes an organization vulnerable. When analysts investigate suspicious activity on a server, knowing that it is internet-facing, supports a critical application, and has a vulnerability under active exploitation helps them assess urgency and determine where to look next.

In Rapid7’s model, exposure findings inform investigation and remediation priorities, while investigations reveal weaknesses that need attention. Security leaders gain a clearer connection between the threats their provider handles and the conditions their organization needs to address.

Within a broader preemptive cybersecurity strategy, MDR connects exposure information with the people investigating and responding to threats. The aim is to reduce opportunities for compromise and intervene before an attack causes harm, while retaining the expertise to contain and recover from incidents.

How exposure context and proactive threat hunting inform MDR

By combining information about assets, vulnerabilities, business criticality, and attacker behavior, preemptive MDR helps teams decide which weaknesses require action first. Exposure context also informs targeted hunting and the investigation of suspicious activity.

Consider a hypothetical organization running an internet-facing application affected by a newly disclosed vulnerability. The security team needs to establish whether the vulnerable software is present, what the application supports, and whether attackers are exploiting the weakness.

Those findings can guide remediation priorities and targeted hunting for signs of compromise. If suspicious activity appears, analysts investigate it with an understanding of the affected system and the potential route an attacker could take through the environment.

Depending on the evidence, the response may involve containing a system, examining related accounts, or expanding the investigation to other assets. The organization’s remediation teams also need clear recommendations about the underlying exposure.

What the investigation reveals then feeds back into detection rules, hunting priorities, and response procedures. This gives leaders a way to track an identified risk through investigation, action, and outstanding work, with clearer ownership at each stage.

What role does AI play in preemptive MDR?

Before analysts can assess a potential threat, they often need to retrieve logs, examine user activity, and connect evidence across several systems. AI can accelerate this work by gathering and correlating information for review.

Agentic investigation capabilities can perform elements of that process in parallel, test possible explanations, and organize findings. Their usefulness depends on the available data: missing identity activity or asset information can leave gaps in the account of what happened.

Rapid7’s approach keeps human expertise central to validation and response decisions. Analysts assess the evidence alongside the organization’s priorities and the consequences of an action, including whether isolating a system could interrupt a critical service.

What to look for in a preemptive MDR provider

When evaluating a service described as preemptive, ask the provider to show how exposure information changes its work. A redacted investigation should make clear what analysts knew about the affected asset, how that influenced their reasoning, and which actions followed.If an investigation identified a weakness, was it assigned to an owner and addressed? Did the findings change detection coverage or future priorities? Where AI contributed, the provider should explain its role and the oversight applied.

Incident response remains part of the requirement. Rapid7 MDR includes unlimited incident response alongside exposure-informed defense and AI-assisted investigation, supporting containment, detailed analysis, and threat eradication.

Agree on responsibilities before that support is needed. Establish which actions the provider can take, when customer approval is required, and who owns remediation. Those arrangements determine how findings become action, particularly when taking a system offline carries business consequences.

Once the service is operating, review progress through specific cases: exposures addressed before exploitation, threats contained, and weaknesses closed after investigation. Ask to see the evidence behind those outcomes and any work still outstanding, so your team can judge where risk has changed and where further attention is needed.

Explore Rapid7 MDR to see how exposure-informed defense, AI-assisted investigation, and expert response work together.

The collective thoughts of the interwebz