Tag Archives: Amazon EMR

Accelerating Spark queries with Iceberg materialized views

2026-09-10 Yuzhou Sun

Post Syndicated from Yuzhou Sun original https://aws.amazon.com/blogs/big-data/accelerating-spark-queries-with-iceberg-materialized-views/

In this post, you learn how to reduce Apache Spark query execution time with Apache Iceberg materialized views without changing a single SQL query.

Organizations running analytical workloads on their data lakes often hit a common wall: queries that are slow and costly, yet difficult to rewrite by hand. Multi-table joins, heavy aggregations, and window functions over large fact tables all drive up execution times, but the SQL behind them often can’t be changed. It might come from business intelligence (BI) dashboards, packaged independent software vendor (ISV) applications, or legacy reports, where editing the source introduces regression risk that outweighs the performance gain.

Starting with Amazon EMR 7.12.0 and AWS Glue 5.1, you can accelerate these queries without rewriting them. Automatic query rewrite analyzes the logical plan of each incoming query and compares it against a metadata cache of available MVs. When the optimizer finds a materialized view (MV) that satisfies all or part of a query, it rewrites the plan to read from that MV instead of the base tables. Matches can be structural (aggregations and joins) or exact (more complex patterns like window functions). If no MV matches, the original query runs unchanged with no impact on correctness.

If you have previously tried to speed up slow analytical queries, you might have considered one of the following alternatives. Here is how automatic query rewrite compares:

Query modification approach	Stored results	Refreshes	Modification to existing queries
Standard views in AWS Glue	No (re-runs each time)	n/a	Required
Custom ETL pipeline	Yes	Manual	Required
Hand-rolled rewrite	Yes	Manual	Required
Materialized views with automatic rewrite enabled	Yes	Automatically through AWS Glue Data Catalog on a schedule when configured	Not required when supported

In this post, we:

Give a high-level overview of how automatic query rewrite works in Apache Spark.
Walk through a concrete example, showing how the same query can benefit from MVs at different levels of coverage.
Discuss the trade-offs so you can choose the right MV shape for your workload.

Prerequisites

To use automatic query rewrite with Iceberg materialized views, you need:

Amazon EMR release 7.12.0 or later, or AWS Glue 5.1 or later.
Source tables in Apache Iceberg or Parquet format, registered in the AWS Glue Data Catalog, in the same AWS Region and account as the materialized view. Parquet source tables are supported for automatic query rewrite starting with Amazon EMR 7.14.0 and AWS Glue 8.1.
An Amazon Simple Storage Service (Amazon S3) Tables (a capability of Amazon S3) bucket, or an S3 general purpose bucket, for the materialized view data.
Permissions for the definer role. You can use AWS Identity and Access Management (IAM) policies or AWS Lake Formation.
Automatic query rewrite turned on in your Spark session: --conf spark.sql.optimizer.answerQueriesWithMVs.enabled=true.
For Parquet source tables, set spark.sql.materializedView.v1SourceTables.enabled=true and spark.sql.materializedView.v1ETagVersioning.enabled=true.

For more Spark configurations, see Introducing Apache Iceberg materialized views in AWS Glue Data Catalog.

How it works

Here is how MVs and automatic query rewrite work together:

You define a SQL query with aggregations, joins, or filters across your supported source tables.
AWS Glue Data Catalog stores the precomputed results as an Apache Iceberg table in your Amazon S3 bucket. You can store it in a general purpose S3 bucket or in Amazon S3 Tables. Any Apache Iceberg-compatible query engine can read the materialized view, including Amazon Athena, Amazon EMR, AWS Glue, Amazon Redshift, and Iceberg-compatible third-party query engines. Automatic query rewrite is available on the AWS optimized Spark runtime in Amazon Athena, Amazon EMR, and AWS Glue. Other engines can query the materialized view directly, but they don’t rewrite queries to use it automatically.
Automatic refresh keeps the MV current on a schedule that you define, for example SCHEDULE REFRESH EVERY 1 DAY. You set it at creation time or later with ALTER MATERIALIZED VIEW ... ADD SCHEDULE REFRESH. At that scheduled time, the refresh process checks the current Apache Iceberg snapshot ID or Parquet file ETags and refreshes the MV when it detects source-table changes.
Automatic query rewrite redirects matching queries to the MV at query optimization time. Automatic query rewrite in Apache Spark uses two matching strategies:
- Structural rewrite (adapted from Amazon Redshift) handles an MV defined as a single SELECT-FROM-WHERE-GROUP-BY block over INNER joins. The optimizer can roll up an MV’s aggregates to a coarser grain and pull extra query predicates up onto the MV scan.
- Exact-match rewrite handles MVs defined as other shapes, such as window functions and outer joins, by matching a canonicalized form of the MV body against subtrees of the query plan.

When the optimizer evaluates a query, it consults a metadata cache of MVs from the configured catalogs and chooses the best match. It also checks MV staleness during optimization. It skips stale MVs, so rewrite won’t return stale results. If no MV matches, the original query runs unchanged.

Note that automatic query rewrite is opt-in: set spark.sql.optimizer.answerQueriesWithMVs.enabled=true when creating the Apache Spark session.

Example: One query with three potential MVs

An MV doesn’t need to cover an entire query to help it. Automatic query rewrite in Apache Spark operates on subtrees: when an MV matches a portion of your query plan, the rewriter substitutes that subtree and lets the rest of the query run on the rewrite output unchanged. The same query can therefore be served by many possible MV designs, each making a different trade-off between per-query speedup, storage cost, and reuse across other queries.

To make this concrete, consider a typical analytics query: “Top 100 preferred US customers by total store spending.” It joins fact and dimension tables, applies two selective filters on the customer dimension, aggregates per customer, ranks the result with a window function, and keeps only the top 100:

SELECT c_customer_id, total_revenue, num_transactions, avg_purchase, revenue_rank
FROM (
    SELECT cust.c_customer_id,
        SUM(sales.ss_quantity * sales.ss_sales_price) AS total_revenue,
        COUNT(*) AS num_transactions,
        AVG(sales.ss_quantity * sales.ss_sales_price) AS avg_purchase,
        RANK() OVER (ORDER BY SUM(sales.ss_quantity * sales.ss_sales_price) DESC) AS revenue_rank
    FROM base_catalog.base_db.store_sales sales
    INNER JOIN base_catalog.base_db.customer cust
        ON sales.ss_customer_sk = cust.c_customer_sk
    WHERE cust.c_birth_country = 'UNITED STATES'
        AND cust.c_preferred_cust_flag = 'Y'
    GROUP BY cust.c_customer_id
) ranked
WHERE revenue_rank <= 100
ORDER BY revenue_rank;

Query 1: The original query. Top 100 preferred US customers by total store spending, before any materialized view.

Three MV designs cover progressively more of this query, from a single-table pre-aggregate to the full query body itself:

Tier 1: Pre-aggregate store_sales only, no join, no filter. This tier is a single-table aggregate of store_sales at customer-surrogate-key grain. The query still must join the customer table, apply both filters, re-aggregate at c_customer_id grain, and run the window function.

CREATE MATERIALIZED VIEW mv_catalog.mv_db.customer_tier_1 AS
SELECT ss_customer_sk,
    SUM(ss_quantity * ss_sales_price) AS sum_revenue,
    COUNT(ss_quantity * ss_sales_price) AS count_revenue,
    COUNT(*) AS num
FROM base_catalog.base_db.store_sales
GROUP BY ss_customer_sk;

Tier 1 MV: Single-table pre-aggregate of store_sales by customer surrogate key (no join, no filter).

The following plans compare the original query plan to the rewritten plan:

Window, filter, Sort
+- Aggregate by c_customer_id
:  total_revenue = SUM(ss_quantity * ss_sales_price)
:  num_transactions = COUNT(*)
:  avg_purchase = AVG(ss_quantity * ss_sales_price)
+- Project
   +- Join Inner ON ss_customer_sk = c_customer_sk
      :- BatchScan store_sales <- reads the large store_sales table
      +- Filter c_birth_country='UNITED STATES' AND c_preferred_cust_flag='Y'
         +- BatchScan customer

Plan 1: Original plan. Scans the large store_sales table.

Window, filter, Sort
+- Aggregate by c_customer_id <- rolls up pre-aggregated sums
:  total_revenue = SUM(sum_revenue) <- sum of sum_revenue
:  num_transactions = SUM(num) <- sum of num
:  avg_purchase = SUM(sum_revenue) / SUM(count_revenue) <- sum of sum_revenue / sum of count_revenue
+- Project
   +- Join Inner ON ss_customer_sk = c_customer_sk
      :- BatchScan customer_tier_1 <- reads pre-aggregated MV
      +- Filter c_birth_country='UNITED STATES' AND c_preferred_cust_flag='Y'
         +- BatchScan customer

Plan 2: Rewritten plan (Tier 1). Reads the pre-aggregated customer_tier_1 MV.

Tier 2: Pre-join store_sales x customer, pre-apply one filter (c_preferred_cust_flag = ‘Y’). The middle tier pre-joins both tables and bakes in the preferred-customer filter. The query still must apply the country filter as a residual on the MV scan and run the RANK() window.

CREATE MATERIALIZED VIEW mv_catalog.mv_db.customer_tier_2 AS
SELECT cust.c_customer_id, cust.c_birth_country,
    SUM(sales.ss_quantity * sales.ss_sales_price) AS sum_revenue,
    COUNT(sales.ss_quantity * sales.ss_sales_price) AS count_revenue,
    COUNT(*) AS num
FROM base_catalog.base_db.store_sales sales
INNER JOIN base_catalog.base_db.customer cust
    ON sales.ss_customer_sk = cust.c_customer_sk
WHERE cust.c_preferred_cust_flag = 'Y'
GROUP BY cust.c_customer_id, cust.c_birth_country;

Tier 2 MV: Pre-joins store_sales and customer, with the preferred-customer filter applied.

Rewritten query plan:

Window, filter, Sort
+- Aggregate by c_customer_id <- rolls up pre-aggregated sums
:  total_revenue = SUM(sum_revenue) <- sum of sum_revenue
:  num_transactions = SUM(num) <- sum of num
:  avg_purchase = SUM(sum_revenue) / SUM(count_revenue) <- reads pre-aggregated MV
+- Filter c_birth_country='UNITED STATES' [residual filter on MV scan]
   +- BatchScan customer_tier_2 <- reads pre-aggregated MV

Plan 3: Rewritten plan (Tier 2). Country filter applied as a residual on the MV scan.

Tier 3: Match the entire query, including the window function and top N filter. This is the most specific tier. The MV body is the target query verbatim (minus the top-level ORDER BY, which is meaningless for a stored set). The MV stores the top-ranked rows the query asks for (rank ≤ 100).

CREATE MATERIALIZED VIEW mv_catalog.mv_db.customer_tier_3 AS
SELECT c_customer_id, total_revenue, num_transactions, avg_purchase, revenue_rank
FROM (
    SELECT cust.c_customer_id,
        SUM(sales.ss_quantity * sales.ss_sales_price) AS total_revenue,
        COUNT(*) AS num_transactions,
        AVG(sales.ss_quantity * sales.ss_sales_price) AS avg_purchase,
        RANK() OVER (ORDER BY SUM(sales.ss_quantity * sales.ss_sales_price) DESC) AS revenue_rank
    FROM base_catalog.base_db.store_sales sales
    INNER JOIN base_catalog.base_db.customer cust
        ON sales.ss_customer_sk = cust.c_customer_sk
    WHERE cust.c_birth_country = 'UNITED STATES'
        AND cust.c_preferred_cust_flag = 'Y'
    GROUP BY cust.c_customer_id
) ranked
WHERE revenue_rank <= 100;

Tier 3 MV: Stores the exact ranked output of the query (exact-match path).

This tier exercises the exact-match rewrite path: the rewriter canonicalizes the MV body and matches it against the query’s logical plan.

Rewritten plan:

Sort revenue_rank ASC
+- BatchScan customer_tier_3 <- reads around 100 stored rows

Plan 4: Rewritten plan (Tier 3). Reads around 100 stored rows.

The trade-off

The three tiers trade per-query speedup against reuse and storage. In our testing on TPC-DS 3 TB, we observed the following:

MV design	Pre-computed	Reuse	Per-query speedup	MV size
Baseline (no MV)	nothing	n/a	1x	n/a
Tier 1: store_sales agg by customer surrogate key	aggregate of all sales per customer	broadest: any per-customer aggregation	~5x faster	0.07% of store_sales for TPC-DS 3 TB
Tier 2: store_sales x customer agg, one filter pre-applied	join + aggregate, preferred customers only	medium: any country filter, preferred customers	~10x faster	0.04% of store_sales for TPC-DS 3 TB
Tier 3: entire query body verbatim (exact-match)	exact ranked output of this query	narrowest: only this exact query shape	20x+ faster	negligible (only 100 rows)

Performance measured on TPC-DS 3 TB. Speedup is the ratio of baseline execution time to MV-accelerated execution time. Results might vary based on data characteristics, cluster size, and query complexity.

In addition, MVs incur additional cost. Each one runs a query against your source tables once and stores the result. The more pre-computation it does (joining more tables, applying more filters), the more time it takes.

The following chart plots per-query speedup and creation time for the three tiers in our testing on TPC-DS 3 TB. Per-query speedup rises steadily, from about 5x at Tier 1 to over 20x at Tier 3. Creation time doesn’t follow the same pattern: it peaks at Tier 2. Tier 2 pre-joins and aggregates all preferred customers across every country, so it materializes the most data work. Tier 3 applies both filters, so it processes far fewer rows and costs less to create.

Chart comparing three materialized view designs. In our testing with TPC-DS 3 TB, we observed per-query speedup rises from about 5x (Tier 1) to over 20x (Tier 3), while creation time peaks at Tier 2, which materializes the most data work. Stacked bars show creation time split into catalog setup, data work, and commit.

Figure 1: Per-query speedup and creation time across the three materialized view tiers, measured on TPC-DS 3 TB

Start by identifying one expensive query that runs repeatedly with stable filters. It is likely a good candidate for an exact-match MV.

Validating automatic query rewrite

To confirm that your query benefited from automatic rewrite:

Query plan inspection: Check the query’s optimized logical plan or physical plan for a leaf scan node referencing the MV (for example, BatchScan mv_catalog.mv_db.your_mv_name). If the MV appears as a scan source, rewrite succeeded.
Log confirmation (Amazon EMR 7.14.0+): Look for INFO-level log entries such as AQMV outcome: rewritten=true, mvs=[mv_name], duration=12ms.
No-rewrite diagnostics (Amazon EMR 7.14.0+): If rewrite didn’t occur, check the MVRewriteMetricsEvent in the Apache Spark Event Log for the specific reason the optimizer skipped the MV.

If you have set spark.sql.optimizer.answerQueriesWithMVs.enabled=true but your query still runs against the base tables, check the following common causes:

Write commands block rewrite by default. INSERT and MERGE statements don’t trigger rewrite. Set spark.sql.optimizer.answerQueriesWithMVs.commandBlockingEnabled=false to turn on rewrite within write command subqueries.
The MV is stale. Rewrite skips the MV when one or more source tables have changed since its last refresh. Wait for the next scheduled refresh, or force an immediate refresh with REFRESH MATERIALIZED VIEW <mv_name>.
Heuristic candidate filtering. The optimizer uses heuristic checks to narrow the set of MV candidates before attempting a full match. In some cases, an MV that could benefit the query might be filtered out early by these heuristics.
Spark version mismatch (Amazon EMR 7.13.0+). Automatic query rewrite skips MVs whose stored IMV_sparkVersion does not match the cluster’s current Apache Spark version. To bypass this check, set spark.sql.materializedView.sparkVersionCompatibilityCheck.enabled=false.
MV metadata cache not loaded. The metadata cache loads lazily during optimization of the first rewritable query in a Spark session. If your critical query fires before the cache is warm, the MV will not be available. Run a small warm-up query (for example, SELECT 1 FROM <some_iceberg_table>) at session start to pay this cost off the critical path.
MV metadata cache memory limit reached. If the cache was disabled or stopped loading MVs because of reaching its memory limit, increase spark.driver.memory.
Too many tables in configured catalogs. If there are many tables or MVs in the configured catalogs, the cache might not finish loading before your query starts. Place MVs in a dedicated catalog, add it to spark.sql.materializedViews.additionalCatalogs, and set spark.sql.materializedViews.scanCurrentCatalog=false to skip scanning the current catalog.
Parquet base tables have additional limitations and configuration requirements. For automatic query rewrite with Parquet base tables, set spark.sql.materializedView.v1SourceTables.enabled=true and spark.sql.materializedView.v1ETagVersioning.enabled=true. Without ETag versioning, Spark can’t determine a usable source-table version and skips the MV. Partitioned Parquet base tables are also subject to additional validation limits.

Performance considerations

Turning on automatic query rewrite has overhead: it introduces trade-offs that might affect some queries negatively:

Optimization overhead. Enabling rewrite adds processing time during query optimization as the optimizer evaluates MV candidates against the query plan. This overhead applies to every query in the session, including those that ultimately don’t match any MV.
Reduced task parallelism. Reading from an MV instead of the original base table might produce fewer tasks or introduce data skew, depending on the MV’s data layout. This reduces parallelism compared to a direct scan of the larger, more evenly distributed source table.

Conclusion

In this post, we showed how automatic query rewrite can accelerate your existing Apache Spark workloads. It uses Apache Iceberg materialized views in the AWS Glue Data Catalog, without changing a single line of SQL. By storing precomputed results as managed Apache Iceberg tables, the AWS Glue Data Catalog lets the Apache Spark optimizer transparently substitute matching query plans. You get the performance benefit of pre-aggregation without the application-level rewiring. BI dashboards, ISV-generated reports, and legacy pipelines all benefit the moment a matching MV exists.

We walked through three MV designs for the same analytical query, each striking a different balance between per-query speedup, storage footprint, and reuse across your workload. As the trade-off table shows, our testing found that a narrow, exact-match MV delivered 20x+ acceleration for a single query shape. A broader pre-aggregate served an entire family of queries at a more modest ~5x gain. The right choice depends on how many queries share the same join-and-aggregate pattern and how frequently your source data changes.

To get started:

Launch an Amazon EMR 7.12.0+ cluster or an AWS Glue 5.1+ job.
Create an MV over your most expensive repeating query using CREATE MATERIALIZED VIEW in the AWS Glue Data Catalog.
Turn on automatic query rewrite by setting spark.sql.optimizer.answerQueriesWithMVs.enabled=true in your Spark session configuration.
Verify the rewrite by inspecting the optimized query plan for an MV scan node, or by checking INFO-level logs on Amazon EMR 7.14.0+.

Queries with multi-table joins, heavy aggregations, or window functions over large fact tables are strong initial candidates. Start with one high-cost, frequently executed query. Validate the speedup, then expand to broader MVs as you identify shared patterns across your workload.

Special thanks to everyone who contributed to the automatic query rewrite feature and this blog: Andre Hernich, Leon Lin, Yiyang Chen, Geeta Krishna Panda, Ashok Chintalapati, Muhammad Malik, Rishabh Bhatia, and Giovanni Fumarola.

References

For more detail, see the following resources:

About the authors

How Moovit achieved 33% cost optimization through architectural modernization

2026-09-03 Saar Porat

Post Syndicated from Saar Porat original https://aws.amazon.com/blogs/big-data/how-moovit-achieved-33-cost-optimization-through-architectural-modernization/

Moovit, part of Mobileye (Nasdaq: MBLY), is a leading Mobility-as-a-Service (MaaS) solutions provider and the creator of a leading urban mobility app. Moovit’s iOS, Android, and web apps offer users a smart mobility experience to get to their destination using any mode of public and shared transportation. Transit riders can benefit from mobile ticketing to plan, pay, and ride with transit services. Introduced in 2012, Moovit now serves over 1.7 billion users in more than 3,500 cities across 112 countries, in 45 languages.

Behind these user-facing experiences is a data platform that processes large volumes of mobility, application, and operational data to support product analytics, business intelligence (BI), monitoring, and data science. As the platform grew, Moovit needed to keep analytical workloads reliable and cost-efficient without slowing down teams that depend on fresh data every day.

Over several years, Moovit’s Amazon Redshift cluster grew continuously. It started with an expanding fleet of DC2 nodes, migrated to RA3 nodes, and scaled multiple times to keep pace with growing data demands, ultimately becoming the backbone of their entire data platform.

To address this growth, Moovit transformed their data architecture by building an optimal multi-engine lakehouse architecture and assigning each workload to the most suitable option. This modernization reduced their Amazon Redshift cluster by 50 percent, while establishing a flexible, multi-engine architecture ready for future use cases.

In this post, we share how Moovit gained visibility into workload patterns, cleaned up unnecessary load, selected candidates for offloading, and ran a successful proof of concept (POC) on Amazon EMR Serverless. Moovit ultimately divided the workload between multiple engines, building a modern and cost-optimized data platform that combines provisioned Amazon Redshift, Amazon Redshift Serverless, and Amazon EMR.

The challenge: Outgrowing a single-engine data platform

The Amazon Redshift engine handled a wide variety of workloads, including:

Heavy ETL processing: Raw data ingestion from Amazon Simple Storage Service (Amazon S3) followed by complex aggregation pipelines (daily user-aggregation running once per day with a 3-day lookback, and weekly 10-day-lookback jobs).
Near-real-time operational monitoring: Queries executing every 20 minutes against raw data for system-health dashboards.
Business-intelligence reporting: Tableau extracts and live dashboards.
Data-science workloads: Exploratory analysis and model-feature engineering.
Ad-hoc analysis: Non-recurring queries done by analysts and engineers.

With business growth, storage grew by orders of magnitude over the past decade as the platform expanded. All these varied workloads competed for the same engine and pushed it to its limits. Jobs experienced increasing queue times, service level agreements (SLAs) were at risk, and adding nodes provided minimal performance gains, creating a need to isolate workloads.

Gaining visibility: Measuring workload impact

Moovit’s first modernization milestone was to create a trusted measurement foundation before changing any workloads. Instead of treating warehouse activity as a single opaque stream, the team implemented automated query attribution that continuously classified each query by workload owner and execution context. The classification combined multiple signals: who executed the query (user or service account), recognizable query-signature patterns, and metadata emitted by orchestration frameworks and scheduled processes.

This produced a historical, query-level map of platform usage that answered three critical questions: who is generating load, what kind of workload is running, and how expensive each workload is in runtime and resource terms. With that baseline in place, the team made offload decisions from evidence rather than assumptions. This approach prioritized the largest and most stable optimization opportunities first and reduced the risk of moving business-critical workloads without visibility.

These classifications and workload metrics were reflected in a Tableau report that aggregated query activity by classification label and execution context. The view exposed operational dimensions such as classification, time granularity, service class, execution-time bucket, unload flags, and sample-query context, supporting both trend monitoring and root-cause drill-down.

The worksheet was parameterized to support multiple measurement modes over the same grouped workload population: total execution time, execution plus queue time, total CPU time, average execution time per query, and ratio-based efficiency views (execution/CPU and CPU/execution). This let the team compare “heavy by volume” workloads against “inefficient by behavior” workloads without creating separate artifacts.

For decision-making, CPU time was used as the primary impact metric because it best represented sustained compute pressure. Execution time, queue time, query-count normalization, and workload-management segmentation were treated as secondary evidence to distinguish:

compute-heavy but healthy workloads
queue-constrained workloads
high-frequency/low-cost workloads
noisy or weakly classified workloads that required attribution cleanup first

Using this framework, prioritization became systematic: first improve classification coverage, then rank workloads by CPU contribution, then validate with queue and workload management (WLM) signals, and finally choose the action path per workload (optimize SQL, reschedule, isolate, retire, or move to another engine).

The following figure shows an example of one of the dashboard widgets (CPU time by query).

Dashboard widget showing CPU time consumed by each query

Figure 1: CPU time by query, highlighting the most resource-intensive queries and their usage patterns

Cleanup: Reducing unnecessary data warehouse load

With a long-running data platform, in most cases the workloads will start accumulating, some of which become irrelevant at some point. For example, a report which was created and scheduled, yet it became irrelevant after a few years, but still running since no one disabled it. It’s important to indicate these workloads in general to reduce unnecessary load, yet even more critical before doing any significant architectural changes or migrations. Before migrating any workloads, Moovit first reduced unnecessary warehouse load.

The team:

Removed unused processes that were still consuming cluster resources.
Reduced unnecessary frequency where possible: some jobs ran more often than downstream consumers needed.
Reviewed workload-management guardrails to verify resource allocation matched actual priorities.

This cleanup phase was a prerequisite to migration. By removing waste first, the team verified that the workloads eventually selected for offloading were genuinely heavy rather than simply unoptimized or unnecessary.

The no-longer-relevant processes consumed around 7 percent of overall CPU time and were removed before the optimization work began.

Workload selection: Choosing what to offload

With a clear picture of workload patterns, Moovit faced a common decision point: continue scaling the existing Redshift cluster, or re-architect towards a multi-engine approach. The team evaluated two main paths:

Re-architect with Redshift multi-cluster and data sharing: Identify workloads that could benefit from resource isolation, then redistribute processing and queries between multiple Redshift clusters, combining both serverless and provisioned options. This would redistribute load across use-case-optimized clusters and potentially save costs through better resource use.
Re-architect with purpose-built engines: Identify workloads that could benefit from alternative processing frameworks and offload them to more suitable engines. This would reduce pressure on Amazon Redshift while building a more flexible, cost-efficient architecture.

Moovit decided to do both, because while some workloads benefited from being offloaded, others benefited from isolated Amazon Redshift compute.

The measurement data revealed a primary candidate for offloading: raw-data aggregation pipelines. This workload loaded raw data into Amazon Redshift from Amazon S3, then performed heavy sessionization and aggregation transformations. Raw tables were still used for ad-hoc and exploratory analysis, but recurring production consumers primarily depended on aggregated outputs, making these transformations strong candidates for offloading.

Proof of concept: Offloading to EMR Serverless with Spark SQL

With target workload identified, Moovit initiated a POC using Amazon EMR Serverless with Spark SQL. The choice of EMR Serverless was driven by several factors:

Spark SQL compatibility: The existing Redshift SQL logic could be ported with minimal changes to Spark SQL syntax.
Serverless simplicity: No cluster-management overhead during the evaluation phase.
Data-lake native: Processing could occur directly on data in Amazon S3.

The POC defined quantified success criteria measured over five or more consecutive runs:

Runtime reduction: Greater than or equal to 40 percent reduction for the transform portion of selected pipelines.
Amazon Redshift cost reduction: Greater than 30 percent reduction in Redshift RA3 compute with no performance degradation for remaining workloads.
Data-quality parity: Exact match between Spark and Amazon Redshift outputs on row counts, distinct users, and all published metrics over a frozen parity window.

Overcoming initial performance challenges

The first POC attempts exposed significant challenges. Early Spark jobs with 100 executors took approximately 4 hours, far exceeding the 30–40-minute baseline on Amazon Redshift. Beyond raw performance, the team encountered memory pressure, data-parity gaps between Spark and Amazon Redshift outputs, and subtle SQL behavior differences between the two engines.

The team systematically diagnosed and resolved these issues:

Execution-plan analysis: Reviewing the Spark execution plan revealed suboptimal query patterns that generated excessive data shuffles.
Query rewrites: Rewriting specific SQL constructs to align with Spark’s distributed processing model, including splitting large monolithic logic into staged transformations.
Reducing or rewriting expensive DISTINCT patterns: Identifying and eliminating unnecessary DISTINCT operations that created heavy shuffle pressure.

After applying these optimizations, execution time dropped from 4 hours to approximately 10 minutes, and the required executors dropped to fewer than 50, surpassing the original performance.

Validation: Ensuring data parity before cutover

Before transitioning any workload to production, Moovit implemented a rigorous validation process. The new Spark output was compared with the previous Amazon Redshift output using multiple dimensions:

Row counts: ensuring no data was lost or duplicated.
Distinct users: verifying entity-level completeness.
Metric parity: all published business metrics matched.
Daily trends: time-series patterns remained consistent.
Row-level checks: spot-checking individual records for correctness.

Only after all validation checks passed consistently over multiple consecutive runs did the team proceed with cutover for each workload.

Moving to production: Expanding workload offloading

With a successful POC demonstrating both performance gains and cost savings, Moovit progressively moved additional workloads from Amazon Redshift to EMR:

Heavy-aggregation jobs: The primary daily and weekly aggregation pipelines transitioned fully to EMR.
Data-transformation stages: Preprocessing steps that previously consumed Redshift compute moved to Spark, with only final aggregated results loaded back into Amazon Redshift for BI consumption.
Weekly batch workloads: Large batch jobs that previously created resource contention during weekend processing windows.

The transition used a measured approach: each workload was migrated individually, with data-quality validation confirming parity before decommissioning the equivalent jobs which were running on Redshift.

Additional optimizations: Redshift Serverless, workload isolation, and Amazon EMR on Amazon EC2

Beyond EMR offloading, Moovit implemented further architectural improvements to isolate workloads and optimize costs.

Amazon Redshift rightsizing: Iterative cluster optimization

With heavy workloads successfully offloaded and isolated, Moovit proceeded to right-size the Redshift cluster. Rather than a single resize, the team reduced the cluster incrementally, two nodes at a time, using elastic resize. At each step, they validated that:

Existing BI workloads maintained acceptable performance.
Queue wait times remained within SLA thresholds.
No workload degradation was observed under peak loads.

This iterative approach minimized risk and allowed the team to find the optimal cluster size with confidence.

Workload isolation with Redshift Serverless

Amazon Redshift persisted as the engine of choice for serving curated BI data. However, not all Amazon Redshift workloads needed provisioned capacity:

Ad-hoc analyst queries: Moved to Redshift Serverless, isolating unpredictable workloads from the provisioned cluster through data sharing.
Data-science workloads: Transitioned to Redshift Serverless for flexible exploration without impacting production.

This workload isolation through Redshift Serverless provided resource separation without requiring additional provisioned capacity. The architecture now used data sharing to provide a unified view across provisioned and serverless clusters.

Moovit also refined workload isolation by rebalancing WLM priorities on the provisioned cluster. Because the ETL queue mainly handled raw data loading from Amazon S3 (which was not the bottleneck after heavy aggregations moved to Spark), its priority was reduced. At the same time, with most human users moved to Redshift Serverless, Tableau serving workloads on provisioned Redshift were prioritized higher to keep dashboard performance predictable. The final result: a 50% reduction in provisioned Redshift capacity.

Transitioning to EMR on EC2

EMR Serverless proved efficient for the POC phase: it allowed fast iteration without cluster management overhead. However, for longer-term recurring production workloads, Moovit moved to EMR on EC2 to better fit their production cost and infrastructure model, using existing compute reservations.

The transition between EMR deployment options required zero application code changes, demonstrating the flexibility of the EMR deployment options.

AI-assisted SQL translation

Additionally, Moovit used AI-assisted development tools, Claude Code and Cursor, to accelerate parts of the SQL transition process. These tools helped engineers identify Redshift SQL and Spark SQL syntax differences, suggest rewrites, and debug migration issues, while validation and production approval remained under engineer review.

Results: A modern multi-engine architecture

The architectural modernization delivered measurable outcomes:

Cluster size reduction: Redshift cluster size reduced to 50 percent of the initial capacity.
Performance improvement: Key aggregation jobs ran faster and more consistently on EMR (50 percent execution time reduction for p90).
Workload isolation: No single workload type could impact others through resource contention.
33 percent overall data pipeline cost reduction: Combined savings from cluster reduction, transition to EMR, and efficient serverless usage.
Future flexibility: The multi-engine architecture provided pathways for additional use cases without architectural changes.

The following figures compare aggregation-job performance before and after the transition.

Chart comparing aggregation-job execution times before and after the transition, with longer, inconsistent runtimes before and shorter, stable runtimes after

Figure 2: Aggregation-job execution times before and after the transition

Chart comparing wall-clock time for job executions across percentiles, with p90 at 5.48 hours before the transition and 2.77 hours after

Figure 3: Wall-clock time for job executions by percentile, before and after the transition

The resulting architecture assigned each workload to the engine that fits it best:

Workload type	Engine	Rationale
Heavy ETL and aggregation	Amazon EMR (Spark SQL)	Distributed processing on Amazon S3. No data warehouse load required
Ongoing processing and BI reporting	Amazon Redshift provisioned	24/7 running processes
Ad-hoc queries	Amazon Redshift Serverless	Burst capacity with workload isolation
Data science	Amazon Redshift Serverless	Flexible exploration without impacting production

Lessons learned

The Moovit modernization journey produced several key insights applicable to similar architectural transitions:

Measure before you move: Establishing baseline metrics and automated classification was essential for identifying true offloading candidates. Without granular workload-level measurements, the team would not have identified which specific processes were exhausting the cluster.
Clean up before you migrate: Reducing unnecessary load first verified that migration efforts targeted genuinely heavy workloads rather than simply unoptimized or unused processes.
Small SQL changes, big impact: Moving from Redshift SQL to Spark SQL required relatively minor syntax adjustments. The core business logic remained intact, and most transformations translated directly with minimal refactoring.
Optimize for the engine: Porting SQL queries to Spark without optimization produced initially poor results for some workloads. Understanding Spark’s distributed execution model and optimizing for it was critical for achieving target performance.
Validate rigorously: Multi-dimensional data-parity checks (row counts, distinct users, metrics, daily trends, and row-level spot checks) gave the team confidence to cut over without data-quality regressions.
Moving between EMR options is straightforward: EMR Serverless proved very efficient for starting fast and evaluating Spark. When Moovit needed to move to EMR on EC2 to use existing reservations, the transition required no application code changes.
Iterative cluster rightsizing: Rather than a single resize, Moovit reduced the Redshift cluster incrementally (two nodes at a time) using elastic resize, validating performance at each step before proceeding further.

Conclusion

Looking ahead, as another potential optimization, Moovit will be evaluating the new Amazon Redshift RG instances for provisioned clusters, providing up to 2.2x better price performance and priced 30% lower than RA3, powered by AWS Graviton.

The broader takeaway is that AWS provides multiple purpose-built engines that can be used in a single data platform. In Moovit’s case, the biggest improvement came from assigning each workload to the engine that fit it best: Amazon Redshift for curated analytical serving, Redshift Serverless for isolated exploratory workloads, and Amazon EMR for large-scale transformations over data in Amazon S3. This architecture gives Moovit a foundation for future optimization and flexibility as data volumes grow and new analytical use cases emerge.

About the authors

Query Amazon S3 Tables from Amazon EMR Trino using the Iceberg REST endpoint

2026-09-02 Shubham Purwar

Post Syndicated from Shubham Purwar original https://aws.amazon.com/blogs/big-data/query-amazon-s3-tables-from-amazon-emr-trino-using-the-iceberg-rest-endpoint/

Organizations running analytics on Amazon Simple Storage Service (Amazon S3) data lakes often struggle with the operational overhead of managing Apache Iceberg tables, including compaction, snapshot expiration, and metadata tracking, while still needing fast, interactive SQL access across large volumes of data. Amazon S3 Tables, a capability of Amazon S3, addresses this by providing a purpose-built storage layer with native Apache Iceberg support and automated table maintenance. When you query S3 Tables from Amazon EMR using Trino and the Iceberg REST endpoint, you get a fully managed, open-standards-based analytics stack without the undifferentiated heavy lifting of table upkeep.

When paired with Amazon EMR running Trino, organizations gain access to a high-performance distributed SQL query engine capable of processing large-scale datasets. Trino’s ability to query data across multiple sources, combined with the automated optimization features of S3 Tables, creates a flexible analytics platform. The integration uses Apache Iceberg’s REST catalog specification, providing a standardized interface that supports compatibility across different compute engines while maintaining full control over query execution and data processing logic.

This architectural pattern is particularly valuable for organizations seeking to modernize their data platforms without vendor lock-in, as it relies on open standards and formats. The solution delivers high-throughput query performance with distributed SQL execution while significantly reducing the operational burden of managing table metadata, compaction, and snapshot lifecycle management. In this post, we show you how to create and query Amazon S3 Tables using Trino on Amazon EMR through the Apache Iceberg REST catalog endpoint.

Solution overview

This implementation demonstrates a complete integration between the Trino distribution on Amazon EMR and Amazon S3 Tables through the Apache Iceberg REST catalog endpoint. The architecture uses several key AWS services working in concert:

Amazon EMR serves as the managed compute layer, providing a scalable Hadoop framework that hosts the Trino query engine. Amazon EMR handles cluster provisioning, configuration management, and automatic scaling, allowing teams to focus on analytics rather than infrastructure management.

Apache Trino acts as the distributed SQL query engine, offering ANSI SQL compatibility and the ability to process queries across massive datasets with low latency for interactive workloads. Its connector architecture supports integration with various data sources, including the Iceberg REST catalog.

Amazon S3 Tables provides the storage and catalog layer, managing Apache Iceberg tables with built-in optimization. The service automatically handles compaction, snapshot expiration, and metadata management, reducing operational overhead while maintaining query performance. S3 Tables exposes a REST API endpoint that conforms to the Apache Iceberg REST catalog specification, which provides standardized integration with any Iceberg-compatible engine.

Apache Iceberg REST endpoint serves as the communication protocol between Trino and S3 Tables. This RESTful interface handles catalog operations including namespace management, table creation, metadata retrieval, and transaction coordination. The endpoint supports AWS Signature Version 4 authentication for secure access to table resources.

The data flow follows this pattern: Users submit SQL queries through the Trino CLI or JDBC interface. Trino’s Iceberg connector communicates with the S3 Tables REST endpoint to retrieve table metadata and plan query execution. The query engine then reads data directly from S3 using optimized file formats (Parquet, ORC) while using Iceberg’s metadata layer for partition pruning and predicate pushdown. Write operations follow a similar path, with Trino coordinating with S3 Tables to commit new data files and update table metadata atomically.

This architecture delivers several key benefits: separation of compute and storage for independent scaling, automated table maintenance reducing operational costs, open-source format compatibility preventing vendor lock-in, and fine-grained access control through AWS Identity and Access Management (IAM) and AWS Lake Formation integration.

Architecture diagram showing Trino on Amazon EMR querying Amazon S3 Tables through the Apache Iceberg REST catalog endpoint

Figure 1: Solution architecture for querying Amazon S3 Tables from Trino on Amazon EMR

Prerequisites

Before getting started, make sure that you have the following:

An active AWS account with billing enabled.
An AWS Identity and Access Management (IAM) user with specific permissions to create and manage resources, such as a virtual private cloud (VPC), subnet, security group, IAM roles, Amazon EMR, Interface VPC endpoints, S3 Tables bucket and S3 buckets.
Sufficient VPC capacity in your chosen AWS Region.

For this post, we create the solution resources in the US East (N. Virginia) Region (us-east-1) using AWS CloudFormation templates. In the following sections, we show you how to configure your resources and implement the solution.

Note: Querying Amazon S3 Tables through Trino on Amazon EMR requires Trino version 475 or later, available in Amazon EMR 7.11 and later.

Part A: Configure Amazon S3 Tables integration with Trino on Amazon EMR using AWS CloudFormation

In this post, you use the CloudFormation template emr-trino-s3tables.yaml.

This template deploys the following resources: a VPC with one private subnet, an S3 Tables interface VPC endpoint for private access, and an Amazon EMR cluster running Trino integrated with Amazon S3 Tables through the Apache Iceberg REST catalog endpoint.
It also creates an S3 Tables bucket, a general-purpose S3 bucket, IAM roles, and security groups.
At deploy time, it dynamically generates the Trino catalog configuration and bootstrap script.

To create the solution resources, complete the following steps:

Launch the stack emr-trino-s3tables.yaml using the CloudFormation template.

Provide the parameter values as listed in the following table.

Parameters	Description	Sample value
Stack Name	Name of CloudFormation stack	emr-s3tables-trino
VPC CIDR block	IP range (CIDR notation) for this VPC.	10.0.0.0/16
Private Subnet CIDR block	IP range (CIDR notation) for the private subnet in the second Availability Zone.	10.0.1.0/24
Resource name Prefix	Short prefix applied to every resource name	emr-s3tables
S3 Tables bucket name	Name of S3 table Bucket	trinoemrs3tablebuck
EMR release	Release version of Amazon EMR	EMR 7.12

The stack creation process can take approximately 15 minutes to complete. You can check the Outputs tab for the stack after the stack is created, as shown in the following screenshot.

Figure 3: CloudFormation stack outputs

Figure 3: CloudFormation stack outputs

Understanding the deployment

The CloudFormation template performs several key tasks:

Infrastructure provisioning: Sets up the Amazon EMR cluster with Trino, VPC, subnet, security group, and S3 table bucket.
Configuration: Creates necessary Trino configuration files.
Integration configuration: Sets up the Iceberg REST connector for S3 Tables.

Part B: Connecting Trino to Amazon S3 Tables with Iceberg REST endpoint

The CloudFormation template automatically configures the S3 Tables catalog in Trino on Amazon EMR. In the next section, we examine the configuration that drives this integration.

1. Catalog configuration details

A catalog in Trino on Amazon EMR is the configuration that grants access to a specific data source. Each Trino on Amazon EMR cluster can have multiple catalogs configured, allowing access to different data sources simultaneously.

As part of this setup, the CloudFormation template creates a catalog properties file at /etc/trino/conf/catalog/s3tables_irc.properties with the following configuration:

connector.name=iceberg
iceberg.catalog.type=rest
iceberg.rest-catalog.uri=https://s3tables.<REGION>.amazonaws.com/iceberg
iceberg.rest-catalog.warehouse=arn:aws:s3tables:AwsRegion:<ACCOUNT-ID>:bucket/<BUCKET-NAME>
iceberg.rest-catalog.sigv4-enabled=true
iceberg.rest-catalog.signing-name=s3tables
iceberg.rest-catalog.view-endpoints-enabled=false
fs.hadoop.enabled=false
fs.native-s3.enabled=true
s3.region=us-east-1
s3.iam-role=arn:aws:iam::<ACCOUNT-ID>:role/service-role/<ROLE-NAME>

2. S3 Tables Iceberg REST endpoint configuration properties

The following table lists the key properties in the catalog configuration on Trino:

Property name	Description
iceberg.rest-catalog.uri	REST server API endpoint URI (necessary).
iceberg.rest-catalog.warehouse	Warehouse ID or location for the catalog (necessary). For S3 Tables, this is the ARN for the S3 table bucket as shown in the preceding properties example.
iceberg.rest-catalog.sigv4-enabled	Must be set to ‘true’ (necessary)
iceberg.rest-catalog.signing-name	Must be set to ‘s3tables’ (necessary)
iceberg.rest-catalog.view-endpoints-enabled	Must be set to ‘false’ (necessary)
fs.hadoop.enabled	Must be set to ‘false’
fs.native-s3.enabled	Must be set to ‘true’
s3.iam-role	Amazon Resource Name (ARN) of the IAM role with permissions to S3 Tables. In this post, we use the same role, which is the service role for Amazon EMR.
s3.region	AWS Region, for example us-east-1

This configuration establishes a connection between Trino and the S3 Tables REST endpoint. You can have multiple catalogs registered, one per S3 table bucket, which is determined by the iceberg.rest-catalog.warehouse property.

3. Configure Amazon EMR service IAM role trust relationships

The Amazon EMR service role requires proper trust relationships to function correctly. Navigate to the IAM console and configure the trust policy for your Amazon EMR service role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "elasticmapreduce.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        },
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ACCOUNT-ID>:role/service-role/AmazonEMR-InstanceProfile"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

This trust policy establishes two critical relationships:

The Amazon EMR service can assume the role to manage cluster operations.
The EC2 instance profile can assume the role to access S3 Tables with elevated permissions.

4. Working with S3 Tables in Trino on Amazon EMR

Now that you have Trino on Amazon EMR set up and configured to work with S3 Tables, you can explore how to work with this integration.

4.1. Connecting to Trino on Amazon EMR

Navigate to Amazon EMR and select Connect to the primary node using AWS Systems Manager Session Manager for passwordless SSH.

Figure 4: Connecting to the primary node with Session Manager

When you’re connected, you can use the Trino CLI with your S3 Tables catalog:

sudo su - hadoop
trino-cli --catalog s3tables_irc

This connects you to the Trino on Amazon EMR using the S3 Tables integration you configured.

Trino CLI connected to the s3tables_irc catalog on Amazon EMR

Figure 5: Trino CLI connected to the S3 Tables catalog

4.2. Examples: Creating and querying tables

In this section you run through some example queries to demonstrate the functionality.

4.2.1 Creating a namespace

First, you create a namespace (schema) in S3 Tables. A namespace in S3 Tables is a logical container or organizational unit that helps group related tables and objects together.

CREATE SCHEMA blog_namespace;
USE blog_namespace;

4.2.2 Creating a table

Create a table with various data types. You don’t need to specify the table type as Iceberg explicitly because you’re connecting to the Iceberg catalog. You can use all standard Iceberg capabilities, such as partitioning and sorting. Furthermore, some of the important Iceberg table properties that support table maintenance operations are configured with default values. You also have the option to edit the configurations using S3 Tables maintenance APIs.

CREATE TABLE IF NOT EXISTS customers (
customer_sk INT,
customer_id VARCHAR,
salutation VARCHAR,
first_name VARCHAR,
last_name VARCHAR,
preferred_cust_flag VARCHAR,
birth_day INT,
birth_month INT,
birth_year INT,
birth_country VARCHAR,
login VARCHAR
) WITH (
format = 'PARQUET',
sorted_by = ARRAY['customer_id']
);

Table property explanation:

format = 'PARQUET': Specifies Parquet as the file format for optimal compression and query performance.
sorted_by = ARRAY['customer_id']: Defines sort order within data files, improving query performance for customer_id filters.

Verify the table creation:

SHOW TABLES;

You should see customers in the output, confirming the table exists in the S3 Tables catalog.

4.2.3 Inserting data

You can insert some sample data into your table. You can also use an existing table in any of the catalogs configured in Trino on Amazon EMR to read data and write into the S3 table with an INSERT INTO ... SELECT statement.

INSERT INTO customers VALUES
(1, 'AAAAA', 'Mrs', 'Martha', 'Rivera', 'Y', 8, 4, 1984, 'US', 'mrivera'),
(2, 'AAAAB', 'Mr', 'Mateo', 'Jackson', 'N', 22, 6, 2001, 'US', 'mjackson'),
(3, 'BAAAA', 'Ms', 'Mary', 'Major', 'Y', 16, 2, 1999, 'US', 'mmajor'),
(4, 'BBAAA', 'Mr', 'Paulo', 'Santos', 'N', 30, 3, 1973, 'US', 'psantos'),
(5, 'AACAA', 'Ms', 'Ana', 'Silva', 'N', 2, 6, 1982, 'CA', 'asilva'),
(6, 'ABAAA', 'Mr', 'Alejandro', 'Rosalez', 'N', 5, 12, 1988, 'US', 'arosalez'),
(7, 'BBAAA', 'Ms', 'Nikki', 'Wolf', 'N', 6, 1, 2006, 'MX', 'nwolf'),
(8, 'ACAAA', 'Mr', 'Arnav', 'Desai', 'N', 15, 7, 1976, 'US', 'adesai');

This INSERT operation demonstrates Trino’s ability to write data to S3 Tables. Behind the scenes, Trino:

Writes data files in Parquet format to S3.
Communicates with the S3 Tables REST endpoint to register the new files.
Atomically commits the transaction, updating table metadata.

4.2.4 Querying data

Execute a SELECT query to retrieve and verify the inserted data:

SELECT * FROM customers LIMIT 10;

The query should return all eight customer records with proper formatting. You can also execute more complex analytical queries:

-- Count customers by country
SELECT birth_country, COUNT(*) as customer_count
FROM customers
GROUP BY birth_country
ORDER BY customer_count DESC;

-- Find customers born after 1990
SELECT first_name, last_name, birth_year
FROM customers
WHERE birth_year > 1990
ORDER BY birth_year;

These queries demonstrate Trino’s SQL capabilities and the integration with S3 Tables for both read and write operations.

4.3 Explore advanced features

S3 Tables with Iceberg provides several features for data management:

4.3.1 Time travel queries

Step 1: Check available snapshots.

-- Query table as of a specific timestamp. Check available snapshots
SELECT * FROM "customers$snapshots";

Step 2: Query the table as of a specific snapshot.

SELECT * FROM customers FOR VERSION AS OF <snapshot_id_from_step1>;

4.3.2 Schema evolution

-- Add a new column
ALTER TABLE customers ADD COLUMN email VARCHAR;

-- Rename a column
ALTER TABLE customers RENAME COLUMN login TO username;

Cleaning up

To clean up the resources, navigate to CloudFormation and delete the stack that you created.

Conclusion

This solution demonstrates an integration between Amazon EMR Trino and Amazon S3 Tables using the Apache Iceberg REST catalog specification. In this post, we showed you how to create and query S3 Tables from Trino on Amazon EMR. The architecture delivers several advantages for modern data platforms:

Operational simplicity: S3 Tables eliminates the complexity of managing Iceberg table metadata, compaction schedules, and snapshot lifecycle policies. The service handles these operations automatically, allowing data teams to focus on analytics rather than infrastructure maintenance.

Performance at scale: The architecture is designed for large-scale workloads. Trino distributes query execution across the cluster while Iceberg’s metadata layer helps the engine locate only the relevant data files. Features like partition pruning, predicate pushdown, and columnar file formats can help improve performance for both interactive and batch workloads.

Cost efficiency: This architecture separates compute and storage, so you can scale each independently based on workload requirements. S3 Tables automatically compacts small files to help reduce storage overhead, and Amazon EMR clusters can scale dynamically so you pay for compute only when needed.

Open standards and portability: By using Apache Iceberg’s open table format and REST catalog specification, this solution avoids vendor lock-in. Other Iceberg-compatible engines can access tables created in S3 Tables including Apache Spark, Apache Flink, and Dremio, providing flexibility in tool selection.

Fine-grained access control: Integration with IAM and resource-based policies provides access control at the table bucket, namespace, and table level. For fine-grained access at the column and row level, you can integrate with AWS Lake Formation. AWS Signature Version 4 authentication supports secure communication between Trino and S3 Tables.

ACID transactions: Iceberg’s transaction model guarantees atomicity, consistency, isolation, and durability for all table operations. This supports reliable concurrent reads and writes, making the platform suitable for production workloads requiring data consistency.

This architectural pattern is particularly well-suited for organizations building modern data lakehouses, migrating from traditional data warehouses, or consolidating multiple analytics platforms. The combination of the managed compute of Amazon EMR, Trino’s versatile query engine, and the automated table management of S3 Tables creates a strong foundation for data-driven decision making.

To learn more about the services and features discussed in this post, see the following resources:

About the authors

Accelerate Apache Spark debugging on Amazon EMR with AWS DevOps Agent

2026-09-01 Kalyan Janaki

Post Syndicated from Kalyan Janaki original https://aws.amazon.com/blogs/big-data/accelerate-apache-spark-debugging-on-amazon-emr-with-aws-devops-agent/

When an Apache Spark job fails on Amazon EMR, the root cause can hide in executor logs, memory profiles, or application code. As data pipelines grow in complexity, correlating logs, metrics, and traces across multiple services requires significant operational effort. AWS DevOps Agent handles this investigation autonomously while keeping operators in the loop to review findings and approve fixes. From a single chat prompt, it produces a root cause and mitigation plan, often without any human involvement beyond the initial question.

The native AWS API tools in AWS DevOps Agent don’t extend into Spark-internal artifacts. Sometimes those tools can’t reach the evidence that pins down the root cause: a Spark History Server event log, executor Python worker memory, or a line of code that allocated too much. In these cases, AWS DevOps Agent can describe symptoms (“the executor exited with code 1”) but can’t identify the actual antipattern that caused them.

This post shows how to extend AWS DevOps Agent to investigate failures in Apache Spark workloads on Amazon EMR. You register the Apache Spark Troubleshooting Agent for Amazon EMR, a managed Model Context Protocol (MCP) server hosted by AWS, as a custom capability provider in your AWS DevOps Agent space. You route the traffic over AWS PrivateLink so MCP calls never traverse the public internet. Then you watch a single agent chat session investigate a deliberately failing Spark job, from Amazon CloudWatch alarm to line-numbered root cause, in about two minutes.

Prerequisites

Before you begin, make sure you have the following:

An AWS account with permissions to deploy AWS CloudFormation stacks that create AWS Identity and Access Management (IAM) roles and Amazon Virtual Private Cloud (Amazon VPC) resources.
The latest version of the AWS Command Line Interface (AWS CLI), Boto3, and Botocore, installed and configured with credentials for the same account and AWS Region.
This walkthrough assumes you are familiar with AWS DevOps Agent and know how to trigger an investigation. For an introduction, see Getting started with AWS DevOps Agent.

How AWS DevOps Agent discovers custom tools through MCP

Model Context Protocol (MCP) is an open standard that defines how AI agents discover and invoke external tools. AWS DevOps Agent supports connecting to custom MCP servers, which means you can expose new capabilities to it without modifying the agent itself. When you connect an MCP server to AWS DevOps Agent, the agent automatically discovers the available tools, understands their schemas, and calls them as part of its investigation workflow. You build and connect the MCP server, and the agent handles the rest.

MCP tools sit alongside the agent’s built-in AWS API tools. During a single investigation, the agent can interleave calls to cloudwatch.describe-alarms, emr-serverless.get-job-run, and a custom MCP tool such as analyze_spark_workload. The agent picks the right one for each subtask. You augment the agent’s reach without replacing what it already does.

For this integration, you don’t build an MCP server. The Apache Spark Troubleshooting Agent for Amazon EMR is itself a managed MCP server, hosted by AWS at a regional endpoint. Your job is to register that endpoint with AWS DevOps Agent and authorize the agent to call it. This requires a network path from the agent to the endpoint, plus an IAM role for AWS Signature Version 4 request signing.

Why Spark internals visibility matters

The actual root cause for a Spark failure usually lives somewhere none of those APIs (such as Amazon CloudWatch Logs Insights, AWS CloudTrail, or Amazon EMR step-status calls) can reach:

The Apache Spark Troubleshooting Agent for Amazon EMR reads the following sources.

The Spark History Server event log is a per-job archive in Amazon Simple Storage Service (Amazon S3) with stage timings, task-level metrics, executor utilization, shuffle read/write volumes, and garbage-collection pauses. Amazon EMR exposes this data through the Spark UI on Amazon EMR Serverless, Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2), and Amazon EMR on Amazon Elastic Kubernetes Service (Amazon EKS), but interpreting signals like data skew, executor memory pressure, or stages that take significantly longer than expected requires familiarity with Spark internals.

The Spark query plan — the logical and physical plan the driver compiled. Without it, you can’t identify antipatterns such as unnecessary data repartitioning or missing broadcast hints that trigger expensive shuffles.
The application source code in Amazon S3 — the .py or .jar code artifact the job ran. Without it, you can’t quote the offending line of a mapPartitions user-defined function or an inefficient collect().
The Python worker process telemetry — the PySpark worker is a separate Python subprocess outside the Java Virtual Machine’s (JVM) managed memory. When it crashes from spark.executor.pyspark.memory exhaustion, the JVM driver sees a generic “executor exited unexpectedly” message. The actual cause is invisible to standard JVM-level logs.

When the agent invokes analyze_spark_workload during an investigation, it returns a structured analysis with the antipattern identified at the line level, the offending stage isolated, and a concrete fix: both code changes and configuration changes.

Integrating AWS DevOps Agent with Apache Spark Troubleshooting MCP

This section explains how AWS DevOps Agent connects to the Apache Spark Troubleshooting Agent through a private MCP endpoint and orchestrates the investigation workflow.

How it works

Architecture diagram showing AWS DevOps Agent connecting to the Apache Spark Troubleshooting Agent over AWS PrivateLink

Figure 1: Integration architecture between AWS DevOps Agent and the Apache Spark Troubleshooting Agent for Amazon EMR over AWS PrivateLink

You submit an investigation prompt in AWS DevOps Agent.
AWS DevOps Agent sends a SigV4-signed MCP call into your Amazon VPC through the AWS DevOps Agent private connection.
The private connection forwards the request to the Interface VPC Endpoint.
The endpoint routes the request over AWS PrivateLink to the Apache Spark Troubleshooting Agent for Amazon EMR, which AWS manages.
The MCP service reads from your data sources (Amazon EMR, Amazon S3, Amazon CloudWatch Logs) using the same IAM role AWS DevOps Agent assumed for the call.
When a CloudWatch alarm transitions to ALARM state (for example, a failed-jobs alarm for your Amazon EMR Serverless application), AWS DevOps Agent automatically triggers an investigation without manual intervention.
AWS DevOps Agent decides which tools to call based on the prompt. For a Spark failure, that includes the Apache Spark Troubleshooting MCP server you registered as a capability provider.
Each MCP request is signed with AWS Signature Version 4 using the IAM role assigned to the capability provider. The request travels from AWS DevOps Agent into your Amazon VPC through the private connection. This private connection is a managed VPC Lattice resource gateway you created during setup.
From the resource gateway, the request flows to the Interface VPC Endpoint for the Amazon SageMaker Unified Studio MCP service, then on to the Apache Spark Troubleshooting Agent. The traffic stays entirely on the AWS network.
The MCP server reads the inputs it needs from your AWS account using the IAM role that you assigned to the capability provider during MCP server registration. This role grants access to the Spark History Server event log and application source code in Amazon S3, the driver and executor stdout streams in Amazon CloudWatch Logs, and the job-run metadata from Amazon EMR Serverless.
The MCP server returns its diagnostic findings to AWS DevOps Agent. The agent then analyzes the results, identifies the root cause, and presents recommended fixes both code-level and configuration-level in your chat.

Setting up the demo

As part of this demo, this post includes a sample AWS CloudFormation template, tested in the us-east-1 Region, that provisions the following resources for the walkthrough:

A dedicated Amazon Virtual Private Cloud (Amazon VPC) with two private subnets in Availability Zones supported by the Apache Spark Troubleshooting Agent for Amazon EMR.
An Interface VPC Endpoint for the Apache Spark Troubleshooting Agent for Amazon EMR.
An IAM role that AWS DevOps Agent assumes to invoke the Apache Spark Troubleshooting MCP server with AWS Signature Version 4.
A deliberately failing PySpark workload running on Amazon EMR Serverless, including the Amazon EMR Serverless application, the Spark execution role, and the demo logs stored in Amazon S3 bucket.
An Amazon CloudWatch alarm that fires when the demo job fails. This alarm is used as the trigger for the agent investigation later in this section.

Step 1: Clone the repository

Clone the git repository for the CloudFormation template, PySpark script, and Parquet data.

git clone https://github.com/aws-samples/sample-aws-data-processing-and-analytics.git

Step 2: Deploy the AWS CloudFormation stack

Deploy the template using the following AWS CLI command.

cd sample-aws-data-processing-and-analytics/blogs/devops-agent-spark-mcp-integration

aws cloudformation create-stack \
  --stack-name spark-troubleshooting-demo \
  --template-body file://cloudformation/spark-troubleshooting-devops-agent-blog.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region us-east-1

The stack reaches CREATE_COMPLETE in approximately 4–6 minutes. When it does, capture the following stack outputs, which you paste into the AWS DevOps Agent console in the next two steps:

DemoVpcId — the VPC ID for the AWS DevOps Agent private connection.
DemoSubnetIds — the two subnet IDs for the AWS DevOps Agent private connection.
SMUSVpcEndpointSecurityGroupId — the security group ID.
TroubleshootingRoleArn — the IAM role Amazon Resource Name (ARN).
MCPEndpointURL — the MCP endpoint URL to register.
FailedJobsAlarmName — the CloudWatch alarm name to reference in your investigation prompt.
DemoBucket — the S3 bucket name where you copy the demo script and Parquet data.

To retrieve all outputs at once, use the following AWS CLI command.

aws cloudformation describe-stacks \
  --region us-east-1 \
  --stack-name spark-troubleshooting-demo \
  --query "Stacks[0].Outputs" --output table

# Get your bucket name from the stack outputs
DEMO_BUCKET=$(aws cloudformation describe-stacks --stack-name spark-troubleshooting-demo --region us-east-1 --query 'Stacks[0].Outputs[?OutputKey==`DemoBucket`].OutputValue' --output text)

# Copy the script
cd sample-aws-data-processing-and-analytics/blogs/devops-agent-spark-mcp-integration
aws s3 cp scripts/customer_events_aggregator.py s3://$DEMO_BUCKET/customer_events_aggregator.py

# Copy the Parquet data
aws s3 cp data/ s3://$DEMO_BUCKET/data/ --recursive

Step 3: Create an agent space

The agent space defines which AWS account and Region the agent monitors, which IAM role it assumes, and which capability providers, including MCP servers, it can call.

Follow the steps in Creating an Agent Space in the AWS DevOps Agent User Guide. When completing those steps, use the following values:

Parameter	Value
Name	data-pipeline-troubleshooting
Region	us-east-1
Agent Space role	Choose Auto-create a new DevOps Agent role — the console generates a DevOpsAgentRole-AgentSpace* role with AIOpsAssistantPolicy attached
Optional integrations	Not required

After the agent space reaches Active status, proceed to create the private connection.

Step 4: Create the AWS DevOps Agent private connection

AWS DevOps Agent uses the private connection to reach into your Amazon VPC. Follow the steps in Connecting to privately hosted tools in the AWS DevOps Agent User Guide. You can use either the console or the AWS CLI command documented under Create a private connection.

When completing those steps, use the following values from your CloudFormation stack outputs:

Parameter	Value
Name	A descriptive name (for example, spark-private)
VPC	DemoVpcId from your stack outputs
Subnets	Both subnet IDs from DemoSubnetIds
Security group	SMUSVpcEndpointSecurityGroupId
TCP port ranges (Advanced configuration)	443
Host address (Service target details)	sagemaker-unified-studio-mcp.us-east-1.api.aws
DNS resolution	In VPC (private DNS)
Certificate public key	None

After the connection reaches Active status, proceed to Step 5.

Step 5: Register the Apache Spark Troubleshooting MCP server as a capability provider

With the private connection in place, register the MCP server as a capability provider. Follow the steps in Registering an MCP server at the account level in the AWS DevOps Agent User Guide.

When completing those steps, use the following values:

Parameter	Value
Name	spark-troubleshooting
Endpoint URL	MCPEndpointURL from your stack outputs
Connect to endpoint using a private connection	Selected

Step 6: Add the MCP server to the agent space

With the MCP server registered, you need a workspace where investigations run. The agent space defines which AWS account and Region the agent monitors, which IAM role it assumes, and which capability providers, including MCP servers, it can call.

In the MCP Server section, choose Add.

MCP Server section of the agent space detail page with an Add button

Figure 2: MCP Server section of the agent space detail page

In the Add a capability dialog, locate spark-troubleshooting in the list of registered MCP servers and choose Add.

Add a capability dialog listing the spark-troubleshooting MCP server

Figure 3: Add a capability with the spark-troubleshooting MCP server listed

On the Select MCP server tools page, both tools that the Apache Spark Troubleshooting Agent for Amazon EMR publishes are listed: analyze_spark_workload and analyze_spark_history_server_endpoint. Select both checkboxes, then choose Save.

Select MCP server tools page with both Spark troubleshooting tools checked

Figure 4: The Select MCP server tools page with both Spark troubleshooting tools selected

The agent space connects to the MCP server, lists its tools, and displays 2 Available / 2 Connected. Both tools are now part of your agent’s catalog.

MCP Server section showing spark-troubleshooting connected with two available and two connected tools

Figure 5: MCP Server section showing spark-troubleshooting connected with both tools available

Seeing it in action

To see the integration end to end, you submit a PySpark job, watch the CloudWatch alarm move to ALARM, and then ask AWS DevOps Agent to investigate using the alarm name.

The failing workload

The CloudFormation template provisioned an Amazon EMR Serverless application called analytics-events-platform and configured a sample PySpark job, customer_events_aggregator.py. The script simulates a common Python-side memory bug: a mapPartitions user-defined function accumulates 11 copies of every input row in an in-memory Python list before yielding results, while the job runs with spark.executor.pyspark.memory=256m. The Python worker process exceeds the 256 MB cap, the kernel kills it, Spark retries four times, and the stage is marked failed.

Submit the failing job

Run the DemoSubmitJobCommand from your stack outputs in your terminal. It looks like this:

aws emr-serverless start-job-run \
  --region us-east-1 \
  --application-id <DemoApplicationId> \
  --execution-role-arn <DemoExecutionRoleArn> \
  --name daily-customer-events-rollup \
  --job-driver '{"sparkSubmit":{"entryPoint":"s3://<DemoBucket>/customer_events_aggregator.py","entryPointArguments":["<DemoBucket>"],"sparkSubmitParameters":"--conf spark.executor.cores=2 --conf spark.executor.memory=1g --conf spark.executor.pyspark.memory=256m --conf spark.executor.instances=2"}}' \
  --configuration-overrides '{"monitoringConfiguration":{"s3MonitoringConfiguration":{"logUri":"s3://<DemoBucket>/logs/"}}}'

The command returns a jobRunId. Note it down. You will see it later in the agent’s investigation.

The job goes through PENDING to SCHEDULED to RUNNING to FAILED and reaches FAILED state in roughly four minutes.

Watch the CloudWatch alarm fire

The CloudFormation template also created a CloudWatch alarm named <DemoApplicationId>-FailedJobs (the exact name is in the FailedJobsAlarmName stack output). The alarm watches the FailedJobs metric in the AWS/EMRServerless namespace, scoped to your demo application, and flips to ALARM within a minute or two of the job failing.

Open the Amazon CloudWatch console, choose Alarms in the left navigation pane, and confirm the alarm is in In alarm state.

Amazon CloudWatch console alarm detail page showing the FailedJobs alarm in alarm state

Figure 6: The Amazon CloudWatch alarm detail page showing the FailedJobs alarm in the In alarm state

Ask AWS DevOps Agent to investigate

Open your AWS DevOps Agent space.
In the left navigation pane, choose Operator Access, then choose Incidents.
Choose Start an investigation.
Paste the following prompt, replacing <FailedJobsAlarmName> with the value from your stack outputs:

CloudWatch alarm in us-east-1 just went into ALARM state. Investigate why and recommend a fix

AWS DevOps Agent Start an investigation panel with the alarm prompt entered

Figure 7: AWS DevOps Agent Start an investigation panel with the Amazon CloudWatch alarm investigation prompt

The agent’s investigation chains together native AWS API tools and the Apache Spark Troubleshooting MCP tool you registered:

use_aws cloudwatch describe-alarms — fetches the alarm definition and reads its metric dimensions, identifying that the alarm is scoped to Amazon EMR Serverless application <DemoApplicationId>.
use_aws emr-serverless list-job-runs — finds the most recent FAILED job run on that application.
use_aws emr-serverless get-job-run — pulls the FAILED run’s metadata and last-known error.
spark-troubleshooting analyze_spark_workload — invokes the Apache Spark Troubleshooting Agent for Amazon EMR through the MCP capability provider, passing the application ID and job run ID. This is where the deep analysis happens.

Review the root cause and fix

When the investigation completes, AWS DevOps Agent presents the results across two tabs: Investigation timeline and Root cause.

The Investigation timeline shows every step the agent took: skills loaded, native AWS API calls made, and the moment it called the analyze_spark_workload MCP tool to analyze the failed Spark job. Each entry is expandable so you can audit the inputs and outputs.

Investigation timeline listing the agent tool calls and the MCP invocation

Figure 8: Investigation timeline tab showing the sequence of agent tool calls and the spark-troubleshooting MCP invocation

The Root cause tab is where the answer lands. It is organized into three sections that mirror what an experienced engineer would write in an incident report:

Root cause tab showing impact, root causes, and key findings for the memory failure

Figure 9: The Root cause tab showing the impact summary, identified root causes, and key findings for the Spark memory exhaustion failure

Impact — what failed, when, and for how long. For our demo, this calls out that the daily-customer-events-rollup job on the analytics-events-platform application failed with a MemoryError and that the alarm transitioned to ALARM state at the time of the failure.
Root causes — the actual antipattern. The agent identifies that customer_events_aggregator.py combines three compounding issues: an expand_event function (line 23) that amplifies each input row 11×, a repartition(1) that funnels all data into a single partition on a single executor, and a collect() (line 31) that pulls the amplified dataset back to the driver. All three run with only 1 GB of executor memory.
Key findings — supporting facts behind the diagnosis, including the executor memory configuration, the application’s maximum capacity, and how the agent confirmed each fact from the analyzed artifacts.

Both the antipattern identification and the supporting evidence come from artifacts the agent could only reach through the MCP tool: the application source code in Amazon S3, the Spark History Server event log, and the query plan. Without the Apache Spark Troubleshooting Agent for Amazon EMR plugged in, AWS DevOps Agent would have stopped at “the executor exited with a memory error.”

Clean up

To avoid ongoing charges, delete the resources you created. Some resources are managed by the AWS DevOps Agent console and must be removed there first. Otherwise, the CloudFormation stack deletion fails.

In the AWS DevOps Agent console, open your data-pipeline-troubleshooting agent space, choose the MCP Server section, select spark-troubleshooting, and choose Remove.
From the Agent spaces list, select data-pipeline-troubleshooting and choose Delete.
In Capability Providers, select spark-troubleshooting and choose Deregister.
In Capability Providers → Private connections, select smus-spark-private and choose Delete.
Delete the AWS CloudFormation stack. This removes the Amazon VPC, the Interface VPC Endpoint, the security group, the IAM role, the Amazon EMR Serverless application, the Spark execution role, the Amazon CloudWatch alarm, and the demo logs bucket.

aws cloudformation delete-stack \
  --region us-east-1 \
  --stack-name spark-troubleshooting-demo

Conclusion

In this post, you connected the Apache Spark Troubleshooting Agent for Amazon EMR to AWS DevOps Agent as a custom MCP capability provider. You kept the traffic on the AWS network with AWS PrivateLink, and ran a failing PySpark job to see the integration end to end. A CloudWatch alarm fired, you asked the agent to investigate, and a single chat session returned the root cause along with code and configuration fixes.

You can extend this pattern beyond the demo scenario. Consider connecting the MCP server to agent spaces that monitor your production Amazon EMR environment. Any Spark job that writes a History Server event log becomes diagnosable through the same workflow.

To continue learning, explore the following resources:

AWS DevOps Agent documentation — learn how to create agent spaces, configure integrations, and manage investigations.
Apache Spark Troubleshooting Agent for Amazon EMR setup guide — detailed prerequisites and configuration options for the MCP server.
Connecting MCP servers to AWS DevOps Agent — register additional MCP servers to expand your agent’s capabilities.
Sample code on GitHub — clone the CloudFormation template, PySpark script, and Parquet data used in this walkthrough.

If you’ve already integrated the Apache Spark Troubleshooting Agent into your operational workflow, or if you’re exploring other MCP-based extensions for AWS DevOps Agent, we want to hear about your experience. Share your thoughts and questions in the comments.

About the authors

GPU-accelerated Apache Spark with Amazon EMR and NVIDIA RTX PRO 4500 on Amazon EC2 G7 instances runs up to 3.7x faster

2026-08-25 McCall Peltier

Post Syndicated from McCall Peltier original https://aws.amazon.com/blogs/big-data/gpu-accelerated-apache-spark-with-amazon-emr-and-nvidia-rtx-pro-4500-on-amazon-ec2-g7-instances-runs-up-to-3-7x-faster/

For years, Apache Spark has been the backbone of large-scale data processing. However, as datasets grow and artificial intelligence and machine learning (AI/ML) pipelines become more complex, modern workloads demand more computational power. Feature engineering for machine learning models, large-scale extract, transform, and load (ETL) transformations, and real-time analytics workloads are computationally intensive by nature. GPU-accelerated instances improve performance and transform jobs that once took hours into minutes, so you can iterate on models faster and reduce operational costs. You can process larger datasets in single batches, make decisions in real time, and achieve strong performance without over-provisioning infrastructure.

We’re excited to share the benchmarking results on Amazon EMR with Amazon Elastic Compute Cloud (Amazon EC2) G7 instances, powered by NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. For data engineers and data scientists running Apache Spark workloads, this means faster pipelines, shorter iteration cycles, and more time spent on insights.

Amazon EMR on EKS natively supports the NVIDIA cuDF plugin for Apache Spark. This support is the result of joint engineering between AWS and NVIDIA to qualify the cuDF plugin for Amazon EMR, co-optimize Spark execution paths for RTX PRO 4500, and validate performance at scale through shared TPC-DS benchmarking on Amazon EC2 G7 instances. Now, Apache Spark workloads on Amazon EMR on EKS run up to 3.7x faster with Amazon EC2 G7 GPU instances than with comparable CPU instances, and require no changes to existing Spark code.

In the TPC-DS 3 TB benchmark, at the 64 GB memory tier, EC2 G7 instances with RTX PRO finished in 4.7 minutes. If you run large-scale data processing pipelines, you can cut job run times by more than two-thirds while maintaining full compatibility with the applications you already have in production.

The use cases that benefit most are those where speed directly unlocks business value. In AI/ML feature engineering, faster Spark jobs mean data science teams can iterate on features more quickly, reducing the time from raw data to trained model. In complex ETL pipelines, like financial transactions, clickstream aggregation, or supply chain data consolidation, GPU acceleration compresses multi-hour batch windows into near-real-time processing. For real-time analytics, teams running fraud detection, personalization engines, or operational dashboards can process larger volumes of data within tighter latency windows, without redesigning their architecture.

Beyond data analytics, the G7 instances will support a broad range of AI and graphics workloads, including conversational AI, content generation, recommender systems, and video streaming and rendering. Built on the AWS Nitro System, they deliver the security and resource efficiency that production AI, analytics, and graphics workloads demand.

The following sections walk through the cluster configuration, benchmark methodology, and performance results.

Cluster configuration

We benchmarked four instance types to measure the real-world performance of G7 GPU instances against comparable CPU instances for Spark SQL performance. The g7.4xlarge also provides 80 Gbps network bandwidth (compared to 15–17 Gbps on the CPU baselines) and uses RapidsShuffleManager. However, CPU runs showed no evidence of being network- or shuffle-bound at this cluster scale. All tests used Amazon EMR on EKS 7.12.0 with Apache Spark 3.5.6 and cuDF plugin 26.04.2, running the full TPC-DS benchmark at 3 TB scale across 103 queries. Each experiment ran 5 iterations. We report the median. Data was stored as Parquet on Amazon Simple Storage Service (Amazon S3) (same-region gateway endpoint). All instances were launched in a single Availability Zone.

Instance specifications

All four instance types share the same compute footprint of 16 vCPUs and 64 GB system RAM. The g7.4xlarge additionally includes an NVIDIA RTX PRO 4500 Blackwell GPU with 32 GB of dedicated video memory (VRAM), which the cuDF plugin uses to accelerate Spark SQL operations. The baseline for all speedup and cost comparisons is m9gd.4xlarge (Graviton), the lowest-cost CPU instance in the group.

.	g7.4xlarge	m9gd.4xlarge	m8id.4xlarge	m8a.4xlarge
Architecture	x86_64	arm64 (Graviton)	x86_64	x86_64
vCPU	16	16	16	16
RAM	64 GB	64 GB	64 GB	64 GB
GPU	1× RTX PRO 4500 Blackwell (32 GB VRAM)	–	–	–
NVMe	875 GB	950 GB	950 GB	EBS only (GP3 16k IOPS and 2000 MB/s throughput to match NVMe
Network	80 Gbps	Up to 17 Gbps	Up to 15 Gbps	Up to 15 Gbps

The g7.4xlarge uses the RTX PRO 4500 Blackwell Server Edition GPU. The CPU baselines cover all three major architectures: m8id.4xlarge (Intel x86), m8a.4xlarge (AMD x86), and m9gd.4xlarge (Graviton arm64).

Spark configuration

All instances used eight executor nodes with the following configuration:

Configuration	GPU instances	CPU instances
Amazon EMR release	emr-7.12.0-spark-rapids-latest	emr-7.12.0-latest
executor.cores	14	14
executor.instances	8	8
executor.memory	20G	20G
executor.memoryOverhead	30G	30G
spark.plugins	com.nvidia.spark.SQLPlugin	–
rapids.memory.pinnedPool.size	8G	–
rapids.sql.concurrentGpuTasks	3	–
shuffle.manager	RapidsShuffleManager	default (sort)
sql.adaptive.enabled	true	true
io.compression.codec	zstd	zstd

CPU instances use the same 30 GB memoryOverhead as GPU to make sure that the memory comparison is apples-to-apples. This setting reserves off-heap memory for shuffle and caching on both sides.

For GPU instances, the cuDF plugin offloads Spark SQL operations to the GPU automatically. No code changes are required. The executor.memoryOverhead value is set higher on GPU instances to accommodate GPU memory management and the RAPIDS shuffle manager.

The cuDF plugin automatically falls back to CPU execution for unsupported operators and user-defined functions (UDFs). Your job still completes, but those stages run without GPU acceleration. To identify which operations run on GPU compared to CPU, set spark.rapids.sql.explain=NOT_ON_GPU in your Spark configuration. For a pre-migration assessment of your workloads, use the NVIDIA cuDF tool to estimate GPU acceleration potential before moving to G7 instances.

To tune settings like concurrentGpuTasks and pinnedPool.size, use the Spark History Server on Amazon EMR on EKS, which provides per-stage execution details to identify CPU fallback and shuffle bottlenecks.

Getting started

Reference the Using cuDF Accelerator for Apache Spark with Amazon EMR on EKS for detailed setup instructions.

Prerequisites

Before running GPU-accelerated Spark on Amazon EMR on EKS, make sure the following are in place:

Amazon EMR on EKS release version 6.9.0 or later (this post uses emr-7.12.0-spark-rapids-latest).

The -spark-rapids release variant ships the NVIDIA cuDF plugin pre-installed.

Amazon Elastic Kubernetes Service (Amazon EKS) cluster with a GPU-enabled node group using G7 instances.
Node AMI: AL2023_x86_64_NVIDIA (Amazon EKS optimized accelerated AMI).

NVIDIA device plugin installed in the cluster to expose GPUs to Kubernetes pods:

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.9.0/nvidia-device-plugin.yml

Amazon EMR on EKS virtual cluster registered to the EKS namespace.

To validate GPU availability on your nodes:

kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

Note: Getting started with GPU-accelerated Spark on Amazon EMR is straightforward. To use the latest cuDF plugin, overlay the latest version (for example, 26.04.2 as of May 2026) onto the Amazon EMR RAPIDS image using an initContainer technique. This replaces the bundled cuDF JAR with a newer version while preserving all other Amazon EMR dependencies. We recommend using the latest Amazon EMR release to get the most up-to-date cuDF plugin for better performance. In our benchmarks, upgrading from cuDF plugin 25.08.0 to 26.04.2 reduced runtime by 36–38 percent. Download the latest cuDF plugin JAR from the NVIDIA repository. AWS Support covers Amazon EMR. For issues specific to a cuDF JAR, file a GitHub issue or contact NVIDIA at [email protected].

Performance benchmarks and cost efficiency

We ran the full TPC-DS benchmark suite (103 queries) at 3 TB scale on 8-node clusters in us-east-1. The following table summarizes the results:

.	GPU instances	CPU instances
Cost per run	$2.06	$2.93–$3.18
Total time (103 queries)	281s (4.7 min)	1,010–1,043s (16.8–17.4 min)
Speedup compared to CPU instances	3.7×	baseline

Cost per run is the total cluster cost for the benchmark’s duration: Cluster $/hr × (median runtime ÷ 3,600). The hourly rate combines the EC2 On-Demand cost for all 8 nodes and the Amazon EMR on EKS charge for the vCPU and memory the Spark pods consume. Both are billed per second (one-minute minimum), so you pay only for what a job uses while it runs. All runs used Amazon EMR on EKS 7.12.0 in us-east-1, with 8 × 4xlarge nodes (128 vCPU) on both the GPU and CPU sides. The g7.4xlarge cluster runs at $26.35/hr (8 × $3.042 EC2 = $24.34, plus $2.01 for Amazon EMR on EKS) and finishes in 281 seconds, at $2.06 per run. The CPU clusters run at a lower hourly rate ($10.43–$10.99) but take 1,010–1,043 seconds, landing at $2.93–$3.18 per run. All prices reflect On-Demand pricing in us-east-1 as of May 2026. G7 instances are also eligible for EC2 Spot and Compute Savings Plans, which can further reduce costs for recurring batch workloads.

Cost-per-run calculations include EC2 and Amazon EMR charges only. They exclude the EKS control-plane fee, EBS volumes, S3 request and storage costs, and the driver pod.

Bar chart of total TPC-DS runtime by instance type, showing g7.4xlarge finishing far faster than the CPU instances

Figure 1: Total runtime by instance type for all 103 TPC-DS queries at 3 TB scale. The g7.4xlarge with GPU acceleration completed the benchmark in 4.7 minutes, 3.7× faster than CPU instances (16.8-17.4 minutes)

Bar chart of total cost per benchmark run by instance type, showing the g7.4xlarge GPU instance costing less than the CPU instances

Figure 2: Total cost per benchmark run, including both Amazon EC2 instance and Amazon EMR on EKS cost across all 8 nodes. Despite a ~2.5× higher hourly rate, the g7.4xlarge GPU instance costs up to 31% less per run than Graviton because it finishes the workload 3.7× faster

Where GPU acceleration excels

GPU acceleration completed the 103-query power run in 281s compared to 1,032s on CPU, an overall 3.7× speedup that saves 750 seconds per run. GPU was faster on 102 of 103 query executions.

GPU acceleration delivers the largest gains on the long-running, compute- and shuffle-heavy queries where kernel throughput outweighs launch overhead. The biggest absolute time savings:

Query	CPU time	GPU time	Speedup	Time saved
q24 (part 1+2)	81.6s	15.9s	~5.1×	65.7s
q23 (part 1+2)	79.4s	16.4s	~4.9×	63.1s
q93	63.7s	5.6s	11.4×	58.1s
q76	30.4s	3.4s	9.0×	27.0s
q64	35.4s	8.5s	4.2×	26.9s
q50	27.6s	3.5s	7.9×	24.0s

Speedup distribution across all 103 executions:

Speedup band	Queries
≥5×	15
4–5×	13
3–4×	21
2–3×	26
1–2×	27
<1× (CPU faster)	1

Median per-query speedup 2.94× (geomean 2.84×). The heaviest wins (q50, q76, q93) are aggregation- and shuffle-join-intensive queries that convert cleanly to GpuHashAggregate and GpuBroadcastHashJoin.

Where CPU wins

With RAPIDS 26.04.2, the following query showcases a workload pattern where CPU was faster:

Query	CPU time	GPU time	Ratio	Root cause
q16	0.96s	1.44s	CPU 1.5x faster	Trivial/near-empty scan. Sub-second runtime where GPU kernel-launch overhead is not amortized

Choosing the right instance

Instance	Best for	Summary
g7.4xlarge (RTX PRO GPU)	Fastest and most cost-effective	Up to 3.7× faster than comparable CPU instances and up to 31% cheaper per run. Completes in 4.7 min compared to 17.2 min. Best choice for both speed and cost efficiency.
CPU instances (m8a / m8id / m9gd)	Flexibility, availability, and always-on workloads	Multiple architecture options deliver similar Spark SQL performance. Choose CPU when GPUs are unavailable, when clusters need to remain running continuously (for example, overnight jobs ready for next-day analysis), or when workloads cannot use GPU acceleration. CPU instances offer broad availability and predictable capacity without startup delays.

G7 instances require a G-instance vCPU service quota in your account (default is often 0 for GPU types). Request a quota increase through the Service Quotas console, or use On-Demand Capacity Reservations (ODCRs) to guarantee availability for recurring batch jobs.

Based on these benchmark results, consider evaluating GPU acceleration for your own Apache Spark workloads. Start by identifying compute-intensive operations in your current pipelines, particularly those involving large-scale aggregations, joins, or machine learning feature engineering that could benefit from the performance improvements demonstrated here.

Conclusion

Amazon EMR on EKS with NVIDIA RTX PRO 4500 together provide a meaningful step forward for teams running data-intensive Spark workloads at scale. Whether you’re building ML pipelines that demand rapid feature iteration, running complex ETL transformations across massive datasets, or powering real-time analytics that can’t afford to wait on slow batch jobs, GPU-accelerated Spark on G7 delivers the performance and speed to do more. As data and AI workloads continue to evolve, GPU-accelerated analytics on Amazon EMR is becoming the foundation for data teams. Get started with GPU-accelerated Spark on Amazon EMR on EKS today by visiting Amazon EMR documentation to launch your first G7-powered cluster and see the performance gains for yourself.

About the authors

Introducing Apache Spark troubleshooting agent for Amazon EMR on EKS

2026-08-03 Vara Bonthu

Post Syndicated from Vara Bonthu original https://aws.amazon.com/blogs/big-data/introducing-apache-spark-troubleshooting-agent-for-amazon-emr-on-eks/

Debugging a failed Apache Spark application on Amazon EMR on EKS often means correlating signals from several places at once. These signals include Spark driver and executor pod logs, Spark event logs, and container termination signals that surface as pod exit codes rather than clear Spark errors. For example, a single out-of-memory failure can appear as a Kubernetes exit code 137 with no obvious link back to the line of code or configuration that caused it. This cross-system investigation can extend a single incident’s mean-time-to-resolution (MTTR) to days and requires deep Spark and Kubernetes expertise.

We recently announced Amazon EMR on EKS now supports Apache Spark troubleshooting agent extending the Apache Spark troubleshooting agent to support Amazon EMR on EKS. The agent already helps data engineers diagnose Spark failures on Amazon EMR on EC2, Amazon EMR Serverless, and AWS Glue using natural language prompts. With this launch, you can now point the same workflow at a failed Amazon EMR on EKS job run. From a single natural language prompt, the agent automatically retrieves your Spark logs from Amazon Simple Storage Service (Amazon S3) or Amazon CloudWatch (depending on your job’s logging configuration) along with Spark History Server Event log data, identifies the root cause, and recommends a fix when the failure is code-related. This can help reduce incident MTTR from days to minutes. Amazon EMR on EKS customers can use the agent at no additional cost. You only pay for your existing Amazon EMR on EKS resources.

In this post, we show you how to set up the agent for Amazon EMR on EKS and walk through troubleshooting a failed job run. We demonstrate the workflow from both the Amazon EMR console and an AI assistant that supports the Model Context Protocol (MCP), an open standard for connecting AI assistants to external tools and data.

How the troubleshooting agent works on Amazon EMR on EKS

The troubleshooting agent exposes a single interface to diagnose failed Spark applications across Amazon EMR on EKS, Amazon EMR on EC2, Amazon EMR Serverless, AWS Glue, and Amazon SageMaker notebooks. Instead of navigating different consoles, APIs, and log locations for each service, you describe your failed job in natural language, and the agent handles the rest. You can reach the agent from the Amazon EMR console or from MCP-compatible AI assistants, such as Kiro CLI, Kiro IDE, or Claude Code. We walk through both later in this post.

The troubleshooting agent runs as a fully managed MCP server, so you do not need to deploy or maintain a local MCP server. It uses a single-tenant design to keep your application data and code isolated. Operations are read-only and governed by AWS Identity and Access Management (IAM) permissions. The agent can only access the resources and actions your IAM role grants. Tool calls are automatically logged to AWS CloudTrail for complete auditability.

Architecture of the Spark troubleshooting agent running as a managed MCP server with read-only IAM access and CloudTrail logging

What’s specific to Amazon EMR on EKS is how the agent gathers its inputs. On Amazon EMR on EKS, your Spark driver and executor logs can be delivered to Amazon S3, Amazon CloudWatch Logs, or both, depending on your job’s monitoring configuration. The agent handles both sources automatically:

Driver and executor pod logs in Amazon S3 – When your job is configured with S3 monitoring, the agent reads the Spark event logs and the per-container stderr/stdout logs from your S3 log location, including discovering executor pod logs.
Driver and executor container logs in Amazon CloudWatch – When your job is configured with CloudWatch monitoring, the agent reads the driver and executor container log streams directly from your CloudWatch log group.
Spark History Server (SHS) data through the Amazon EMR Persistent UI – For the richer SHS signals (query plans, executor timelines, stage metrics, and configurations), the agent connects to the Amazon EMR Persistent UI for your job run, the same mechanism used for Amazon EMR on EC2.

Drawing on years of AWS experience running millions of Spark applications at scale, the agent extracts the relevant features and signals from these sources, work that would otherwise require manual correlation across Amazon S3, Amazon CloudWatch, and the Spark UI. It then uses a large language model on Amazon Bedrock, grounded in a managed knowledge base of Spark and AWS troubleshooting expertise through Retrieval Augmented Generation (RAG), to produce a root cause analysis and, when the failure is code-related, a code recommendation.

The large language model (LLM), the knowledge base, and the retrieval that connects them are fully managed as part of the agent. There’s nothing for you to provision, host, or tune. This managed inference is provided at no additional cost for Amazon EMR on EKS. You pay only for the AWS resources you already use to run your Spark applications and to validate recommended changes.

The agent extracting signals from Amazon S3 and Amazon CloudWatch and using an Amazon Bedrock model with a knowledge base to produce a root cause analysis

Getting started

You can use the agent from either the Amazon EMR console or an MCP client. Both rely on setting up a single IAM role. The following sections walk through creating that role and then troubleshooting a failed job run with each method.

Set up IAM permissions

The IAM role grants the agent read access to the diagnostic sources it analyzes, such as your Amazon EMR on EKS job runs, the Amazon EMR Persistent UI, and your Spark logs in Amazon S3 and Amazon CloudWatch. Creating this role is the only setup required for the console experience. The MCP client path has a few additional prerequisites, covered later in the section on troubleshooting from an MCP client.

To run the commands in this section, you need the AWS Command Line Interface (AWS CLI) (version 2.30.0 or later) installed and configured with your AWS credentials. For instructions, see Setting up the AWS CLI.

Step 1: Create the IAM role

The agent uses your IAM role to authorize operations at the AWS service level. It can only access what your role allows. Create a role your account can assume, then attach a policy granting the permissions the agent needs for Amazon EMR on EKS.

First, set some variables for the commands that follow. ACCOUNT_ID is derived from your configured credentials. Set REGION to the AWS Region where you run your Amazon EMR on EKS workloads:

ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
REGION=us-east-2   # replace with your Region

Create a trust policy that allows your account to assume the role, and create the role:

cat > mcp-trust-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowAccountToAssumeRole",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::${ACCOUNT_ID}:root" },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

aws iam create-role \
  --role-name SparkTroubleshootingMCPRole \
  --assume-role-policy-document file://mcp-trust-policy.json

Step 2: Attach Amazon EMR on EKS permissions

Create and attach a policy granting the agent read access to your Amazon EMR on EKS job runs, the Amazon EMR Persistent UI, and your S3 and CloudWatch logs. Replace amzn-s3-demo-logging-bucket with the name of your logging bucket and replace my_log_group_name and my_log_stream_prefix with your CloudWatch log group name and log stream prefix, respectively.

cat > emr-eks-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EMREKSReadAccess",
      "Effect": "Allow",
      "Action": [
        "emr-containers:DescribeJobRun",
        "emr-containers:DescribeVirtualCluster",
        "emr-containers:ListJobRuns",
        "emr-containers:ListVirtualClusters"
      ],
      "Resource": ["*"]
    },
    {
      "Sid": "EMREKSPersistentApp",
      "Effect": "Allow",
      "Action": [
        "elasticmapreduce:CreatePersistentAppUI",
        "elasticmapreduce:DescribePersistentAppUI",
        "elasticmapreduce:GetPersistentAppUIPresignedURL"
      ],
      "Resource": ["*"]
    },
    {
      "Sid": "EMREKSS3LogAccess",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:ListBucket"],
      "Resource":[
        "arn:aws:s3:::amzn-s3-demo-logging-bucket",
        "arn:aws:s3:::amzn-s3-demo-logging-bucket/*"
      ]
    },
    {
      "Sid": "EMREKSCloudWatchLogAccess",
      "Effect": "Allow",
      "Action": [
        "logs:GetLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams"
      ],
      "Resource": [
        "arn:aws:logs:*:*:log-group:my_log_group_name:log-stream:my_log_stream_prefix/*"
      ]
    }
  ]
}
EOF

aws iam put-role-policy \
  --role-name SparkTroubleshootingMCPRole \
  --policy-name EMREKSTroubleshootingAccess \
  --policy-document file://emr-eks-policy.json

Note: If you prefer an automated setup, an AWS CloudFormation template that creates this role with the required permissions is available in the setup documentation. The previous CLI steps give you the same result with finer control over each permission.

Troubleshooting a failed Amazon EMR on EKS job run

You can reach the troubleshooting agent two ways: directly from the Amazon EMR console, or from an MCP-compatible AI assistant such as Claude Code. We walk through both, using two different failures to show the range of what the agent diagnoses.

Option 1: Troubleshoot from the Amazon EMR console

The console offers the fastest path. Once you’ve created the IAM role in the Set up IAM permissions section, no additional setup is required. Here we troubleshoot a job that failed with a driver out-of-memory error. The application generates a large dataset and calls collect() to pull it back to the driver, exceeding the configured spark.driver.maxResultSize of 512 MiB.

Open the Amazon EMR console, choose Virtual clusters (under Amazon EMR on EKS), and select the virtual cluster that ran your job.
In the Jobs list, find your failed job run and choose its Failed status. This opens a popover with a Troubleshoot with AI button.

The failed job run popover in the Amazon EMR console with the Troubleshoot with AI button

Choose Troubleshoot with AI. The agent analyzes the job and returns its findings directly on the console, namely the analysis insights, a root cause, and a recommendation. For this job, it identifies that the collect() operation on line 24 attempts to materialize the full result set on the driver, exceeding the spark.driver.maxResultSize safety limit. This fails the job before an actual driver out-of-memory crash. Because the failure stems from the application code, the agent also returns a code recommendation: a before-and-after diff that replaces the collect() call with a distributed write to the destination path. Executors then persist their partitions in parallel instead of funneling the data through the driver.

Agent results in the console showing the root cause and a before-and-after code recommendation for the collect() failure

Option 2: Troubleshoot from an MCP client (Claude Code)

You can also use the agent from MCP-compatible AI assistants. This option requires a one-time setup to connect the assistant to the agent’s MCP servers, and it unlocks a conversational workflow where the agent chains from analysis into a concrete code fix. In this walkthrough, we use Claude Code.

Prerequisites

In addition to the IAM role from the Set up IAM permissions section, the MCP client path requires:

Python 3.10 or higher.
The uv package manager. For instructions, see Installing uv.
Claude Code installed. For instructions, see Install Claude Code. You can also use another MCP-compatible AI assistant such as Kiro CLI or Kiro IDE.

Configure an AWS CLI profile

Configure a profile that assumes the IAM role you created, so the MCP servers call AWS with the agent’s permissions:

export IAM_ROLE=arn:aws:iam::${ACCOUNT_ID}:role/SparkTroubleshootingMCPRole
export SMUS_MCP_REGION=${REGION}

aws configure set profile.smus-mcp-profile.role_arn ${IAM_ROLE}
aws configure set profile.smus-mcp-profile.source_profile default
aws configure set profile.smus-mcp-profile.region ${SMUS_MCP_REGION}

Add the MCP servers

The troubleshooting agent provides two tools through two MCP servers: analyze_spark_workload (workload analysis and root cause) and spark_code_recommendation (code fixes). Add both to your assistant.

For Claude Code:

claude mcp add sagemaker-unified-studio-mcp-troubleshooting \
    -- uvx mcp-proxy-for-aws@latest \
    https://sagemaker-unified-studio-mcp.${SMUS_MCP_REGION}.api.aws/spark-troubleshooting/mcp \
    --service sagemaker-unified-studio-mcp --profile smus-mcp-profile \
    --region ${SMUS_MCP_REGION} --read-timeout 180

claude mcp add sagemaker-unified-studio-mcp-code-rec \
    -- uvx mcp-proxy-for-aws@latest \
    https://sagemaker-unified-studio-mcp.${SMUS_MCP_REGION}.api.aws/spark-code-recommendation/mcp \
    --service sagemaker-unified-studio-mcp --profile smus-mcp-profile \
    --region ${SMUS_MCP_REGION} --read-timeout 180

Verify your setup by running the /mcp command in Claude Code to confirm the sagemaker-unified-studio-mcp-troubleshooting and sagemaker-unified-studio-mcp-code-rec servers are connected and their tools are available.

For Kiro CLI:

# Add the Spark Troubleshooting MCP server
kiro-cli-chat mcp add \
    --name "sagemaker-unified-studio-mcp-troubleshooting" \
    --command "uvx" \
    --args "[\"mcp-proxy-for-aws@latest\",\"https://sagemaker-unified-studio-mcp.${SMUS_MCP_REGION}.api.aws/spark-troubleshooting/mcp\", \"--service\", \"sagemaker-unified-studio-mcp\", \"--profile\", \"smus-mcp-profile\", \"--region\", \"${SMUS_MCP_REGION}\", \"--read-timeout\", \"180\"]" \
    --timeout 180000 \
    --scope global

# Add the Spark Code Recommendation MCP server
kiro-cli-chat mcp add \
    --name "sagemaker-unified-studio-mcp-code-rec" \
    --command "uvx" \
    --args "[\"mcp-proxy-for-aws@latest\",\"https://sagemaker-unified-studio-mcp.${SMUS_MCP_REGION}.api.aws/spark-code-recommendation/mcp\", \"--service\", \"sagemaker-unified-studio-mcp\", \"--profile\", \"smus-mcp-profile\", \"--region\", \"${SMUS_MCP_REGION}\", \"--read-timeout\", \"180\"]" \
    --timeout 180000 \
    --scope global

Verify with the /tools command in Kiro CLI to confirm the analyze_spark_workload and spark_code_recommendation tools are available.

Run the agent

For this walkthrough, we troubleshoot a different failure to show how the agent chains from analysis into a concrete code fix. The job is a small PySpark application that reads a CSV file into a DataFrame and registers it as a temporary view named people. It runs a Spark SQL query to uppercase the Name column before displaying the results. The job run failed because the query calls UPPERX, a function that doesn’t exist in Spark SQL (it’s a typo for the built-in UPPER).

From the Claude Code terminal (or MCP-compatible assistants), describe your failed job run in natural language, providing the virtual cluster ID and job run ID:

Debug my EMR on EKS job with job run id <jr-id> and virtual cluster id <vc-id> in <region>

The agent invokes the analyze_spark_workload tool, which automatically:

Calls the Amazon EMR on EKS API to retrieve your job run’s configuration and determine where its logs are stored.
Retrieves your Spark logs from Amazon S3 or Amazon CloudWatch, depending on your job’s logging configuration.
Connects to the Amazon EMR Persistent UI to extract Spark UI features such as the execution plan, stage metrics, and executor timelines.
Analyzes the correlated signals and returns a root cause explanation.

For this job, the agent returns:

Root cause: SQL function error. Your Spark SQL query references a function UPPERX that doesn’t exist in an available function catalog (system.builtin, system.session, or spark_catalog.default). Category: SQL_ERROR. The job failed because the function name can’t be resolved. UPPERX is almost certainly a typo for the built-in UPPER function.

Because the failure is code-related, the agent then chains into the spark_code_recommendation tool, which produces a concrete before-and-after fix:

  df.createOrReplaceTempView("people")

- result = spark.sql("SELECT UPPERX(Name) FROM people")
+ result = spark.sql("SELECT UPPER(Name) FROM people")
  result.show()

  spark.stop()

The two tools work together. analyze_spark_workload identifies the root cause, and when the failure stems from the application code, spark_code_recommendation returns the exact edit to make. You review the recommendation and apply it with full control over the change. The agent only provides the analysis and recommendations.

Supported failure categories

The troubleshooting agent diagnoses a wide range of Apache Spark failures on Amazon EMR on EKS, including:

Out-of-memory and resource exhaustion – Driver and executor out-of-memory errors, including driver-side failures from operations like collect() and executor terminations that surface as Kubernetes pod exit codes (such as exit code 137).
Data skew and shuffle issues – Uneven partitioning and shuffle failures that concentrate work on a few executors.
Configuration errors – Misconfigured Spark settings that lead to failures or inefficiency.
Code-level issues – Problems such as incorrect API usage, unbounded collect() calls, and user-defined function (UDF) errors, for which the agent can recommend code fixes.

Code recommendations are supported for PySpark workloads on Amazon EMR on EKS, Amazon EMR on EC2, Amazon EMR Serverless, and AWS Glue.

Conclusion

With support for Amazon EMR on EKS, the Apache Spark troubleshooting agent gives platform and data engineering teams a shared workflow for investigating failed Spark applications. By bringing together Spark and Kubernetes diagnostic signals, the agent can reduce manual investigation and repeated handoffs between teams, helping engineers identify likely causes and corrective actions faster.

There’s no additional charge for using the troubleshooting agent, including the large language model used through Amazon Bedrock. You pay only for the AWS resources used to run your Spark applications and validate recommended changes.

To get started:

Follow the setup guide in the Apache Spark troubleshooting agent documentation.
Configure the MCP servers in Kiro CLI, Kiro IDE, or Claude Code, and try troubleshooting a failed Amazon EMR on EKS job run.

About the authors

Lowering AWS KMS decrypt API costs in EMR Spark jobs

2026-07-30 Navaneedha Krishnan Jagathesan

Post Syndicated from Navaneedha Krishnan Jagathesan original https://aws.amazon.com/blogs/big-data/lowering-aws-kms-decrypt-api-costs-in-emr-spark-jobs/

Modern organizations processing vast amounts of data on Amazon EMR with Apache Spark face a growing cost challenge. As the number of encrypted Amazon Simple Storage Service (Amazon S3) objects grows, AWS Key Management Service (AWS KMS) decrypt API calls multiply rapidly, driving up operational costs. Consider a retail organization processing hundreds of terabytes of customer transaction data daily in S3 encrypted with AWS KMS. Each Spark task accessing an encrypted S3 object triggers an AWS KMS decrypt API call. At scale, these calls accumulate into significant and often unexpected cost increases. This is especially true for workloads that require key auditability and cannot switch to S3 Bucket Keys. S3 Bucket Keys reduce AWS KMS request costs by decreasing the number of calls from S3 to AWS KMS. However, S3 Bucket Keys limit per-object key auditability in AWS CloudTrail, which might not meet the compliance requirements of some organizations.

This post introduces practical techniques to reduce AWS KMS decrypt costs. You can reduce API call volume and lower costs without compromising encryption. It covers three techniques: optimizing file formats (including Apache Iceberg), aggregating data, and using AWS Glue Data Catalog partition indexes.

Optimization techniques

The following sections describe three techniques you can apply independently or together to reduce the number of AWS KMS decrypt API calls.

Use data aggregation

Data aggregation reduces redundant AWS KMS decrypt API calls by consolidating smaller files into larger blocks. When Spark reads many small files from S3, each file triggers its own decrypt call. By combining multiple small files into fewer, larger files, you can reduce the total number of AWS KMS API invocations. This technique is effective for read-heavy workloads that involve numerous small files stored on S3.

You can use AWS CloudTrail to monitor changes in API call frequency and validate the effectiveness of data aggregation in reducing costs.

Step 1: Benchmark baseline performance

Before applying optimizations, establish baseline metrics to quantify improvements.

from pyspark.sql import SparkSession
import time

spark = SparkSession.builder.appName("Baseline Job").getOrCreate()
start_time = time.time()
data = spark.read.format("csv").load("s3://amzn-s3-demo-bucket/data/")
end_time = time.time()
load_time = end_time - start_time
print(f"Load time: {load_time:.2f} seconds")
data_count = data.count()
print(f"AWS KMS calls triggered: {data_count} rows processed")

The following figure shows the number of AWS KMS Decrypt API calls captured in AWS CloudTrail. Use these baseline metrics to compare against optimized results in subsequent steps.

Amazon Athena console displaying CloudTrail log query results with a KMS Decrypt API call events triggered during the baseline Spark job reading unoptimized CSV files from S3

AWS CloudTrail log showing baseline AWS KMS Decrypt API call count

Step 2: Aggregate files using Spark

Consolidating smaller files into fewer, larger files stored in S3 minimizes redundant decrypt API calls.

consolidated_data = data.coalesce(10)
consolidated_data.write.mode("overwrite").parquet("s3://amzn-s3-demo-bucket/optimized-data/")

Step 3: Rerun the job with optimized files

Read the aggregated data created in Step 2 and compare the AWS KMS Decrypt API call count against the baseline metrics from Step 1.

optimized_data = spark.read.parquet("s3://amzn-s3-demo-bucket/optimized-data/")
optimized_data.count()

The following figure shows the AWS CloudTrail logs after reading the aggregated data.

Amazon Athena console displaying CloudTrail log query results with a reduced number of KMS Decrypt API call events after reading aggregated Parquet files

AWS CloudTrail logs in Amazon Athena showing AWS KMS Decrypt API call count after data aggregation

CloudTrail metrics comparison

Track the number of API calls and observe the direct impact of data aggregation on reducing AWS KMS decrypt API calls for the same amount of data.

The following figure compares the AWS KMS Decrypt API call count before and after data aggregation for the same dataset.

Comparison chart showing AWS KMS Decrypt API call count for the same dataset, with a significant reduction after consolidating small CSV files into fewer aggregated Parquet files

AWS KMS Decrypt API call comparison before and after data aggregation

Aggregating small files into fewer large files reduces decrypt calls and shortens load time.

Optimize file formats and compression

Selecting appropriate file formats and applying compression minimizes the amount of data read from S3 and the number of AWS KMS decrypt operations.

Columnar formats (Parquet/ORC)

Columnar file formats like Parquet and ORC let Spark read only the required columns for analysis, which improves performance for analytical queries. For example, you can convert raw CSV data to Parquet to benefit from better I/O efficiency and query optimization.

df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load("s3://emr-kms-demo/data/")

# Set compression for Parquet files
spark.conf.set("spark.sql.parquet.compression.codec", "snappy")

df.write.format("parquet").save("s3://amzn-s3-demo-bucket/parquet-data/")

Iceberg format

Apache Iceberg is a modern table format designed for large-scale analytic datasets. It supports schema evolution, snapshot isolation, and time travel, making it an excellent choice for data lakes on S3. When used with PySpark, Apache Iceberg simplifies data management by automatically optimizing file layouts, handling partitions, and integrating with Spark catalogs.

The following PySpark example uses Iceberg with Amazon EMR and S3:

pyspark \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.4.2 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
  --conf spark.sql.catalog.spark_catalog.type=hadoop \
  --conf spark.sql.catalog.spark_catalog.warehouse=s3://amzn-s3-demo-bucket/iceberg-warehouse \
  --conf spark.sql.defaultCatalog=spark_catalog

df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load("s3://amzn-s3-demo-bucket/data/")

spark.conf.set("spark.sql.parquet.compression.codec", "snappy")

# Write data to Iceberg table
df.writeTo("iceberg_from_emr_data").using("iceberg").create()

# Read from Iceberg table
spark.read.table("iceberg_from_emr_data").show()

Compression

Using compression reduces data size and speeds up reads and writes between S3 and Spark. Note: ZSTD is the recommended and default compression codec for Iceberg, offering better compression ratios. For this demonstration, we use Snappy to illustrate the concept.

spark.conf.set("spark.sql.parquet.compression.codec", "snappy")

Compression not only minimizes I/O and network overhead but also accelerates job execution in distributed Spark environments.

CloudTrail comparison on API calls

The following figure shows the reduction in AWS KMS Decrypt API calls when using optimized file formats with compression.

Comparison showing AWS KMS Decrypt API call count for Parquet files with Snappy compression

AWS KMS Decrypt API calls on optimized file formats with compression

The following table illustrates the reduction in AWS KMS decrypt calls when moving from raw, uncompressed CSV data to optimized Parquet files with compression enabled.

Comparison showing AWS KMS Decrypt API call count for Uncompressed CSV vs Parquet files with Snappy compression

AWS KMS Decrypt API call comparison for CSV and compressed Parquet with Snappy

AWS Glue Data Catalog partition index

Partitioning data helps Spark jobs retrieve subsets of relevant data, reducing scan ranges, and decrypt operations. Using AWS Glue Data Catalog partition indexes reduces scanning overhead and the number of AWS KMS API calls.

Without a partition index, when Spark queries a partitioned table, AWS Glue Data Catalog returns all partitions by calling the GetPartitions API. Spark then reads every S3 object across all returned partitions. Because each S3 object is individually encrypted, Spark must call the AWS KMS Decrypt API once per object. More objects mean more decrypt calls and higher costs. With a partition index, AWS Glue performs server-side partition filtering, returning only matching partitions.

Step 1: Baseline query without partition index

Using an Amazon EMR Spark job:

spark.sql("SELECT * FROM default.`kms-demoevents` WHERE year='2000' AND month='04'").count()

Then check CloudTrail for the kms:Decrypt call count.

The following figure shows the AWS KMS Decrypt API call count when running the baseline query without a partition index. Spark scans all partitions, resulting in a higher number of decrypt calls.

Amazon Athena console displaying CloudTrail log query results showing the total number of AWS KMS Decrypt API calls triggered when querying without a partition index

AWS KMS Decrypt API call count without a partition index

Step 2: Add partition index and rerun the baseline query from Step 1

In the AWS Management Console or through the AWS Command Line Interface (AWS CLI), create partition indexes and rerun the same baseline query from Step 1. Create partition indexes on the year and month columns. Then check CloudTrail for the kms:Decrypt call count.

The following figure shows the AWS KMS Decrypt API call count after adding a partition index. With the partition index, AWS Glue filters partitions server-side, resulting in fewer S3 objects read and fewer decrypt calls.

Amazon Athena console displaying CloudTrail log query results showing a reduced number of AWS KMS Decrypt API calls after adding a partition index on year and month columns, compared to the baseline query without partition index

AWS KMS Decrypt API call count with a partition index

Conclusion

Optimizing EMR Spark jobs ensures cost-effective and efficient processing of encrypted data at scale. S3 Bucket Keys is the most effective way to reduce AWS KMS Decrypt API calls. The techniques covered in this post are additional optimizations that you can use together with S3 Bucket Keys for further cost reduction. You can also use them independently when S3 Bucket Keys cannot be used because of per-object auditability requirements in CloudTrail. Start implementing these strategies today to improve your Spark workload efficiency and achieve cost savings.

We welcome your feedback. If you have questions or suggestions about this post, leave a comment below.

About the author

Accelerate Spark on EMR Serverless with larger workers and shuffle-optimized disks

2026-07-29 Karthik Prabhakar

Post Syndicated from Karthik Prabhakar original https://aws.amazon.com/blogs/big-data/accelerate-spark-on-emr-serverless-with-larger-workers-and-shuffle-optimized-disks/

With Amazon EMR Serverless, you can run open source big data frameworks such as Apache Spark and Apache Hive without managing clusters or infrastructure. Customers are increasingly choosing EMR Serverless for their analytics workloads because of the simplicity of a fully managed, serverless experience. As adoption grows, teams want to bring their most demanding jobs to Serverless too. These jobs include large-scale joins, shuffle-heavy ETL, and memory-intensive analytics that previously required carefully sized clusters. Customers migrating these heavyweight workloads from their Spark clusters often need the same compute shapes on EMR Serverless to achieve the same price-performance and make migration easier.

Today, we’re excited to announce a new 32 vCPU / 244 GB worker configuration on Amazon EMR Serverless, giving you the headroom to run your most intensive workloads without leaving the serverless experience.

Overview of larger workers

An EMR Serverless application uses workers to run your Spark tasks, and you can choose a worker size that matches your workload. The new 32 vCPU worker offers a large compute and memory footprint (32 vCPUs and 244 GB of memory) that supports attaching up to 2,000 GB of shuffle-optimized disk. This combination benefits three common workload patterns:

Shuffle-intensive workloads – Wide transformations such as join, groupBy, sortBy, and repartition redistribute large amounts of data across the cluster. Larger workers keep more shuffle data local to each executor and read and write shuffle blocks on higher-throughput disk, reducing remote fetches and shuffle wait time.
I/O-heavy workloads – Queries that scan large datasets or spill intermediate data to disk are limited by disk throughput and IOPS. The shuffle-optimized disk raises the ceiling on both sides. Large workers also improve network bandwidth.
Memory-intensive workloads – Higher per-executor memory (244 GB versus 30 GB) lets more data be cached and processed in memory without spilling, which helps with data skew and caching.

Benchmark setup

We compared the recommended large-worker configuration against common small-worker defaults at identical total compute (192 vCPUs).

Large workers – 6 executors × (32 vCPU / 244 GB / 2,000 GB shuffle-optimized disk).
Small workers – 48 executors × (4 vCPU / 30 GB / 200 GB standard disk).

The two configurations differ in both worker shape and disk class. We compare them as paired configurations, since shuffle-optimized disk is the intended disk type for the 32 vCPU worker. The reported gains reflect this combined effect.

Both configurations set spark.dynamicAllocation.enabled=false and spark.scheduler.minRegisteredResourcesRatio=1 to reduce variance from worker launch times. Requiring full registration ensures each query starts only after the cluster is ready. We ran every query with 3 iterations and reported the median. Both benchmarks ran on EMR release emr-7.13.0 in a virtual private cloud (VPC) with an Amazon Simple Storage Service (Amazon S3) gateway endpoint attached to the private subnets.

Benchmark results

This post presents benchmark results comparing the new 32 vCPU workers against the existing 4 vCPU standard workers using the industry-standard TPC-DS and TPC-H benchmarks. Across 126 queries (104 TPC-DS and 22 TPC-H), large workers delivered an average of 29% faster query execution and 29% lower query-attributed cost, with zero regressions and improvements peaking at 45–55% on shuffle-heavy, multi-table join queries. Both the larger executor shape and the shuffle-optimized disk contribute to these gains. The disk advantage is most pronounced on the shuffle and I/O-heavy queries where the largest improvements appear.

The following table summarizes the results across both benchmarks. Large workers won every query on both performance and cost.

Note: The benchmark results in this post are derived from the TPC-DS and TPC-H benchmark specifications. TPC-DS and TPC-H are trademarks of the Transaction Processing Performance Council.

Benchmark	Queries tested	Avg performance improvement	Avg cost improvement
TPC-DS (3 TB)	104	26.7%	27.4%
TPC-H (1 TB)	22	38.5%	37.2%
Combined	126	28.8%	29.1%

TPC-DS 3 TB benchmark

TPC-DS is an industry-standard decision support benchmark that models complex analytical workloads with multi-table joins, subqueries, and aggregations. We ran 104 queries from the TPC-DS v2.4 suite against a 3 TB partitioned Parquet dataset, with 3 iterations per query for statistical confidence.

The test environment was as follows:

Dataset: 3 TB partitioned Parquet (24 TPC-DS tables).
Queries: 104 (full suite minus 4 incompatible with the dataset schema).
Iterations: 3 per query (one query per start-job-run, a fresh Spark application each time).
Networking: VPC with an Amazon S3 gateway endpoint.
EMR release: emr-7.13.0.

The following chart shows the top 10 and bottom 5 queries by performance improvement. All 104 queries show a positive improvement, with the largest gains on shuffle-heavy queries such as q58 (45%), q21 (43%), and q12 (42%).

Bar chart of per-query performance improvement for the top 10 and bottom 5 TPC-DS queries

Performance improvement for the top 10 and bottom 5 TPC-DS queries, 32 vCPU compared to 4 vCPU standard workers

Why large workers are faster for TPC-DS

TPC-DS queries are characterized by complex multi-table joins that generate large shuffle operations. With 6 large executors instead of 48 small ones, the shuffle-optimized disk provides significantly higher random I/O throughput for reading and writing shuffle blocks. In addition, fewer executors mean less network coordination during shuffle. Each executor fetches shuffle data from only 5 remote sources instead of 47. This increases the share of shuffle data read locally, which improves performance.

TPC-H 1 TB benchmark

TPC-H is a decision support benchmark that focuses on ad hoc analytical queries. We ran all 22 TPC-H queries against a 1 TB dataset, with 3 iterations per query. Each query was submitted as a separate start-job-run (a fresh Spark application) to simulate the realistic pattern of independent ad hoc queries arriving without session warmup.

The test environment was as follows:

Dataset: 1 TB partitioned Parquet.
Queries: 22.
Iterations: 3 per query (one query per start-job-run, a fresh Spark application each time).
Networking: VPC with an Amazon S3 gateway endpoint.
EMR release: emr-7.13.0.

Bar chart of per-query performance improvement across the 22 TPC-H queries

Performance improvement for TPC-H queries, 32 vCPU compared to 4 vCPU standard workers

Why large workers are faster for TPC-H

The shuffle-optimized disk accelerates this initial table-scan phase. The subsequent query execution benefits from higher per-executor memory (244 GB versus 30 GB), which lets more data be processed in memory without spilling to disk.

Cost calculation and improvement results

The following table compares the cost of running TPC-DS and TPC-H benchmarks on larger workers (32 vCPU / 244 GB / 2,000 GB shuffle-optimized disk) versus smaller workers (4 vCPU / 30 GB / 200 GB standard disk) on EMR Serverless. Both configurations use identical total compute (192 vCPUs).

TPC-DS 3 TB

Metric	Larger Workers (6 × 32 vCPU)	Smaller Workers (48 × 4 vCPU)
Runtime	2,780.8s	3,670.2s
Resource Billed	vCPU = 148.31 \| Memory = 1,130.87 \| Disk = 9,269.47	vCPU = 195.74\| Memory = 1,468.07 \| Disk = 9787.15
Cost	$15.37	$19.87
Total vCPU	192	192
Total Disk	12 TB (shuffle-optimized)	9.6 TB (standard)
Improvement	27.4% lower cost	Baseline

TPC-H 1 TB

Metric	Larger Workers (6 × 32 vCPU)	Smaller Workers (48 × 4 vCPU)
Runtime	940.7s	1,529.5s
Resource Billed	vCPU = 50.17 \| Memory = 382.54 \| Disk = 3,135.60	vCPU = 81.57 \| Memory = 611.81 \| Disk = 4,078.72
Cost	$5.20	$8.28
Total vCPU	192	192
Total Disk	12 TB (shuffle-optimized)	9.6 TB (standard)
Improvement	37.2% lower cost	Baseline

Notes:

Runtime represents the sum of median query execution times across all queries in the benchmark (3 iterations per query, median reported).
Calculated cost is computed using EMR Serverless on-demand pricing: vCPU-hr ($0.052624), Memory GB-hr ($0.0057785), Storage GB-hr ($0.000111).
Illustrative TPC-DS run cost calculation:
- Large worker: (148.31 * $0.052624) + (1,130.87 * $0.0057785) + (9,269.47 * $0.000111) = $15.37.
- Smaller worker: (195.74 * $0.052624) + (1,468.07 * $0.0057785) + (9787.15 * $0.000111) = $19.87.
Cost is proportionally attributed to query execution time, excluding Spark initialization and shutdown overhead.
Both configurations use identical total vCPU (192) with Dynamic Resource Allocation disabled.
The improvement percentage represents the cost reduction achieved by larger workers relative to smaller workers.
Disk Sizing: The larger workers provisioned 25% more disk (12 TB versus 9.6 TB), yet the total cost came out 22.6–37.2% lower. Disk is by far the lowest-priced billing dimension on EMR Serverless ($0.000111 per GB-hour, versus 52x that for memory and 474x for vCPU) and made up only $1.03 of the $15.37 TPC-DS total. Many customers under-provision disk to trim this smallest line item, and it backfires. Shuffles slow down, jobs run longer, and every extra second is billed on the costlier vCPU and memory dimensions. On large workers (8+ vCPUs), shuffle-optimized disks scale IOPS and throughput with capacity, which helps move shuffle data faster. Size disk as a performance lever, not a cost lever.

When to use large workers

To determine whether the 32 vCPU worker with shuffle-optimized disk will benefit your Spark applications, consider the following:

Check the Stages tab of the Spark History Server for your EMR Serverless application and review the Shuffle Read and Shuffle Write columns. The larger the shuffle volume relative to the number of executors, the more a job benefits from keeping shuffle data local on high-throughput disk. Jobs that shuffle tens of gigabytes or more per executor are strong candidates.
Check the Stages tab for the Spill (memory) and Spill (disk) columns and the Executors tab for peak JVM memory. If data is spilling to disk or peak memory is close to the configured executor memory, the higher memory of the large worker can remove the spill and improve performance.

When not to use large workers

Large workers are not the best fit for every workload:

I/O-bound jobs – For workloads whose runtime is dominated by reading and writing data (rather than shuffle or memory pressure), a larger number of smaller workers (for example, 8 or 16 vCPU) with the same disk sizes can deliver better aggregate throughput. Spreading the work across more executors increases read/write parallelism, although a few large workers can leave disk and network bandwidth underused.

Conclusion

In this post, we demonstrated that EMR Serverless 32 vCPU workers deliver performance and cost benefits for shuffle-intensive Spark workloads. Across 126 TPC-DS and TPC-H queries, larger workers achieved an average 29% faster execution and 29% lower cost.

We recommend evaluating the 32 vCPU worker with shuffle-optimized disk for your shuffle-intensive and I/O-heavy EMR Serverless Spark workloads. To get started, use the following configuration:

--conf spark.executor.cores=32
--conf spark.executor.instances=6
--conf spark.executor.memory=220g
--conf spark.emr-serverless.executor.disk=2000G
--conf spark.emr-serverless.executor.disk.type=SHUFFLE_OPTIMIZED

For more information about worker configurations, refer to Worker configurations in the Amazon EMR Serverless User Guide. We also recommend staying up to date with the latest EMR releases to take advantage of ongoing performance improvements.

About the authors

Automate Spark Scala migration to 4.x with AWS Spark Upgrade Agent

2026-07-28 Bezuayehu Wate

Post Syndicated from Bezuayehu Wate original https://aws.amazon.com/blogs/big-data/automate-spark-scala-migration-to-4-x-with-aws-spark-upgrade-agent/

If you’re a data worker responsible for managing Apache Spark 3.x workloads on Amazon EMR before, you’ve likely faced the challenge of migrating hundreds of jobs to Spark 4.0 without disrupting production pipelines. In this post, you will learn how to automate Spark 3.x to 4.0 migration using the AWS Spark Upgrade Agent covering API deprecations, behavioral changes, build configuration updates, and job validation. What once took months of manual effort can be completed in hours.

This is part 3 of a three-part series on how the AWS Spark Upgrade Agent can automate and simplify Spark upgrades.

Part 1 introduces the agent’s architecture and capabilities. Part 2 walks through a complete PySpark migration from Spark 3.5 to Spark 4.0 on Amazon EMR Serverless. This post walks through Scala migration from Spark 3.3 (Scala 2.12) to Spark 4.0 (Scala 2.13).

Apache Spark 4.0 on Amazon EMR 8.x delivers improvements like native merge_into() support, enhanced Adaptive Query Execution, improved Python UDF performance through Arrow-based serialization, and major Structured Streaming enhancements. For teams on Spark 2.4 or 3.x, the complexity lies in managing API deprecations, behavioral changes, build configuration updates, and re-validating hundreds of jobs while maintaining production pipelines.

Prerequisites

This post assumes you’ve completed the one-time AWS CloudFormation setup and proxy configuration detailed in the introduction post.

What is the Spark Upgrade Agent?

The AWS Spark Upgrade Agent is a fully managed remote server that automates Spark migration using a Model Context Protocol (MCP) interface for code analysis and transformation. For details, see the introduction post.

Architecture: How it works

The architecture follows a least-privilege security model:

Scoped IAM roles — AWS IAM roles are scoped to only MCP server calls, Amazon Simple Storage Service (Amazon S3) staging bucket access, and Amazon EMR job submission.
Local source code — Your source code stays local, with only minimal diagnostic information transmitted.
Encryption in transit — All data is encrypted in transit.
Audit trail — AWS CloudTrail records every tool invocation for full auditability.

Example use case: Enterprise-scale Spark migration

To illustrate the capabilities of the agent at scale, consider a real-world migration scenario from a large company. This company runs a data processing platform with thousands of Spark jobs across Scala, PySpark, and Spark SQL workloads, with a code base spanning Spark 3.3 and 3.5.

The data engineering team faces a migration across three workload types with different complexity profiles:

Spark SQL applications: The most portable, but still requiring validation of behavioral changes in the query optimizer and join strategies introduced in Spark 4.0.
PySpark workloads: Requiring updates to UDF serialization patterns, Arrow-based optimizations, and DataFrame API changes.
Scala applications: The most complex, involving build system updates (Maven and SBT), API deprecations, and recompilation against new Spark 4.0 JARs.

To tackle this, the team uses the Spark Upgrade Agent to migrate all three workload types to Spark 4.0 on Amazon EMR 8.x. For each workload, the agent is invoked directly from Kiro or VS Code with Cline, applying targeted transformations and immediately validating results against a live Amazon EMR 8.x Serverless application running Spark 4.0.

Migrations that would traditionally require months of manual engineering effort complete in a fraction of the time. The agent follows an iterative refinement loop: it performs local validation, then submits the job to a remote Amazon EMR cluster. This loop catches and resolves runtime failures automatically, reducing the need for manual debugging cycles. Build configuration files (pom.xml and build.sbt) are updated automatically by the agent, eliminating a common source of migration errors.

Solution walkthrough

The following sections walk through the complete migration workflow, from initial setup through advanced code transformations and validation.

1. Setup

This section lists what you need before starting. Some items, such as Amazon EMR Serverless applications, can be created during the walkthrough using the agent if they don’t already exist.

Must have before starting:

AWS Command Line Interface (AWS CLI) configuration: Your AWS CLI must be configured with a profile that has the necessary permissions. See Configuring the AWS CLI for details.
IAM role with Amazon EMR permissions: An AWS CloudFormation template is provided in the setup guide to provision the required IAM role. The role is scoped to the permissions needed for the upgrade process: calling the MCP server, reading and writing to the Amazon S3 staging bucket, and submitting Amazon EMR jobs.
Amazon S3 staging bucket for artifacts: Used to store code artifacts and Amazon EMR job outputs during the validation phase.
Integrated development environment (IDE) installation: Kiro or VS Code with MCP support (Cline extension). Either IDE can interact with the Spark Upgrade Agent through natural language prompts. Consult the setup guide for Kiro and Cline’s documentation to use the MCP server with Cline.
One-click MCP server installation: The dataprocessing-mcp server is installed and configured as described in the setup guide.
Amazon EMR Serverless applications: An Amazon EMR Serverless application is required for the validation workflow:
- Target application (Spark 4.0): An Amazon EMR Serverless application configured with release label emr-spark-8.0.0, used to validate migrated jobs against Spark 4.0 on Amazon EMR 8.0.

1.1 Infrastructure setup (AWS CloudFormation)

Two AWS CloudFormation stacks create the required resources: an AWS IAM role, an Amazon S3 staging bucket, an Amazon EMR Serverless application (Spark 4.0), and its execution role.

Stack 1: AWS IAM role and Amazon S3 staging bucket

The spark-upgrade-mcp-setup template creates the AWS IAM role and Amazon S3 staging bucket required by the upgrade agent. Choose the Launch Stack button for your Region. For additional Regions, see the full Region list.

Region	Launch
US East (N. Virginia)	Launch Stack
US East (Ohio)	Launch Stack
US West (Oregon)	Launch Stack
Europe (Ireland)	Launch Stack

After deployment, open the AWS CloudFormation Outputs tab, copy the ExportCommand value, and run it in your terminal. This sets SMUS_MCP_REGION, IAM_ROLE, and STAGING_BUCKET_PATH automatically.

The following figure shows the Outputs tab with the ExportCommand value.

AWS CloudFormation console Outputs tab with the ExportCommand value ready to copy

Outputs tab of the AWS CloudFormation stack showing the ExportCommand value

# Sets SMUS_MCP_REGION, IAM_ROLE, and STAGING_BUCKET_PATH
export SMUS_MCP_REGION=<YOUR-REGION> && export IAM_ROLE=arn:aws:iam::<YOUR-ACCOUNT-ID>:role/spark-upgrade-role-* && export STAGING_BUCKET_PATH=<amzn-s3-demo-bucket>

Then configure the AWS CLI profile:

aws configure set profile.spark-upgrade-profile.role_arn ${IAM_ROLE}
aws configure set profile.spark-upgrade-profile.source_profile default
aws configure set profile.spark-upgrade-profile.region ${SMUS_MCP_REGION}

Stack 2: Amazon EMR Serverless target application and execution role

The emr-serverless-target-setup template creates an Amazon EMR Serverless application configured with Spark 4.0 (release label emr-spark-8.0.0) and a shared execution role used for job submission during the validation phase. Deploy it as follows:

git clone https://github.com/aws-samples/sample-amazon-emr-spark4-examples
cd sample-amazon-emr-spark4-examples/scala3/demo_1_spark_change_focus

The Scala sample lives at sample-amazon-emr-spark4-examples/scala3/demo_1_spark_change_focus. The CloudFormation template lives at resources/cloudformation/.

Deploy the CloudFormation template to create the target Amazon EMR Serverless application and a shared execution role:

aws cloudformation deploy \
  --template-file resources/cloudformation/emr-serverless-target-setup.yaml \
  --stack-name spark-emr-serverless-upgrade \
  --region ${SMUS_MCP_REGION} \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
  StagingBucketName=${STAGING_BUCKET_PATH} \
  TargetReleaseLabel=emr-spark-8.0.0 \
  TargetApplicationName=spark-upgrade-target

This creates an Amazon EMR Serverless target application (Spark 4.0) for upgrade validation, with a shared execution role. The application auto-stops after 15 minutes of idle time, so there is no cost when not in use. To upgrade between different Spark versions, override the SourceReleaseLabel and TargetReleaseLabel parameters with the Amazon EMR release labels that you want.

After the stack completes, note the outputs:

aws cloudformation describe-stacks \
  --stack-name spark-emr-serverless-upgrade \
  --region ${SMUS_MCP_REGION} \
  --query "Stacks[0].Outputs" --output table

This gives you the TargetApplicationId and ExecutionRoleArn needed for the upgrade prompt. Make a note of them.

2. Upgrade

This section covers a complete end-to-end upgrade using a representative ecommerce pipeline, a Scala application that processes order events, applies transformations, and writes results using merge-style upsert patterns. The same workflow applies to Scala and Spark SQL workloads covered in subsequent sections.

Step 1: Clone the sample project

Start by cloning the sample project from the AWS samples repository:

git clone https://github.com/aws-samples/sample-amazon-emr-spark4-examples
cd sample-amazon-emr-spark4-examples/scala3/demo_1_spark_change_focus

The repository includes representative PySpark, Scala, and Spark SQL applications designed to demonstrate common Spark 3.x patterns and their Spark 4.0 equivalents.

Step 2: Open in your IDE and connect to the MCP server

Open the project in Kiro or VS Code with the Cline extension. Verify that the dataprocessing-mcp server is active and connected. You can see it listed as an available MCP server in your IDE’s MCP panel. If you haven’t completed the one-time setup, follow the setup guide before proceeding.

Step 3: Start the upgrade with a natural language prompt

Once connected, initiate the upgrade by entering the following request in the agent interface:

Use the dataprocessing-mcp server to upgrade my local project at <path-to-your-project>.
Upgrade my Spark application from Amazon EMR Serverless version 6.9.0 to Amazon EMR Serverless version 8.0.0.
Use Amazon EMR Serverless app-id <your-app-id> for validation.
Store artifacts at s3://amzn-s3-demo-bucket/spark4-upgrade/

The agent responds by invoking generate_spark_upgrade_plan, analyzing your project structure, identifying incompatible patterns, and presenting a prioritized upgrade plan before proceeding with any code changes.

After you confirm the plan, the agent proceeds autonomously through the remaining phases:

Build configuration update — update_build_configuration rewrites pom.xml, build.sbt, or requirements.txt to target Spark 4.0 dependencies.
Environment validation — Java and Python environments are checked and updated as needed.
Code transformation — fix_upgrade_failure applies targeted fixes for each identified incompatibility, iterating until the project compiles cleanly.
Remote validation — run_validation_job submits the upgraded application to your Amazon EMR Serverless target application and monitors execution through check_job_status.
Data quality check (optional) — get_data_quality_summary compares output between the Spark 3.5 baseline and the Spark 4.0 run, confirming correctness before sign-off.

With the sample Scala ecommerce pipeline cloned from the sample-amazon-emr-spark4-examples repository and your IDE connected to the MCP server, you submitted a natural language prompt. This triggered the agent to analyze the project structure and generate a prioritized five-step upgrade plan, all before making any code changes.

Now that the upgrade plan is in place, the following sections walk through the specific code transformations the agent applies for a Scala workload.

Scala workload migration

This section covers the complete migration of a representative Scala Spark application from Amazon EMR Serverless 6.9.0 (Spark 3.3, Scala 2.12) to Amazon EMR Serverless 8.0.0 (Spark 4.0, Scala 2.13), using the demo_1_spark_change_focus sample from the AWSSpark4AutoUpgradeDemo repository.

Sample project: Ecommerce product change focus pipeline

The sample application processes product catalog change events from Amazon S3, applies enrichment transformations, and writes aggregated results back to Amazon S3. It represents a common pattern in ecommerce data platforms: incremental processing of catalog updates with downstream aggregation. In this example, we use VS Code with Cline, but you can also use Kiro or any other MCP-enabled IDE.

Project structure:

demo_1_spark_change_focus/
├── build.sbt
├── project/
│   ├── build.properties
│   └── plugins.sbt
└── src/
    └── main/
        └── scala/
            └── job_script.scala

The following figure shows the project structure as it appears in the IDE, with the build configuration and Scala source files.

IDE file tree showing the project layout with build.sbt, the project folder, and the Scala source file

Project structure with build.sbt and Scala source files in the IDE

Step 1: Invoke the upgrade agent

Enter the following prompt:

Use the dataprocessing-mcp server to upgrade my local project at
/Users/<your-path>/AWSSpark4AutoUpgradeDemo/resources/scala/demo_1_spark_change_focus/
from Amazon EMR Serverless version 6.9.0 to Amazon EMR Serverless version 8.0.0.
Use Amazon EMR Serverless app-id <your-app-id> for validation.
Store artifacts at s3://amzn-s3-demo-bucket/spark4-upgrade/scala/

The agent reads your project files, identifies the current Spark and Scala versions, and generates a structured upgrade plan (the analysis ID is logged for traceability). The following figure shows the agent generating the upgrade plan.

Agent output listing the detected Spark 3.3 and Scala 2.12 versions and the generated upgrade plan

Agent generating the upgrade plan with Spark and Scala version analysis

When prompted, choose Yes, proceed with the upgrade as planned, then choose Approve on the subsequent permission dialog to allow the agent to update the build configuration.

Step 2: Build configuration update — build.sbt

The first action of the agent is to invoke update_build_configuration, rewriting build.sbt to target Spark 4.0 and Scala 2.13, as shown in the following figure.

Agent output showing the build.sbt dependency changes from Spark 3.3 to Spark 4.0.1

Agent output showing build.sbt updates from Spark 3.3 to Spark 4.0

The build configuration update is generated. The following are the recommended changes for build.sbt.

Changes summary:

Scala version: Updated from 2.12.15 to 2.13.16 (required for Spark 4.0).
Spark dependencies: Updated from 3.3 to 4.0.
- spark-core: 3.3 → 4.0.
- spark-sql: 3.3 → 4.0.
Assembly settings: Added configuration for creating uber JARs with proper merge strategies.
Dependency exclusions: Added rules to exclude provided dependencies (Spark, Scala, Hadoop) from assembly.

The following figure shows the updated build.sbt and plugins.sbt files after the configuration changes are saved.

The updated build.sbt and plugins.sbt files open in the editor after the configuration changes

Updated build.sbt and plugins.sbt files saved after configuration changes

Step 3: Java environment check

Amazon EMR Serverless 8.0.0 runs on Java 17. The agent invokes check_and_update_build_environment to verify your local Java Development Kit (JDK) and upgrade it from Java 11 to Java 17, as shown in the following figure.

Agent output verifying the local Java version and recommending an upgrade to JDK 17

Agent verifying Java environment and recommending JDK 17 for Amazon EMR 8.0

Step 4: Scala source code transformations

After updating the build configuration, the agent compiles the project and applies fix_upgrade_failure iteratively to resolve Scala 2.13 and Spark 4.0 breaking changes. Scala 2.13 removed several deprecated collection methods that were available in 2.12. The compilation failed with errors related to the Scala 2.13 syntax change. The .to[Set] syntax needs to be updated to .to(Set) for Scala 2.13. The agent used the fix_upgrade_failure tool to resolve the compilation errors. The following are the key transformations applied to job_script.scala.

The following figure shows the agent applying Scala 2.13 source code transformations to resolve the compilation errors.

Agent output showing the Scala 2.13 source code edits applied to job_script.scala

Agent applying Scala 2.13 source code transformations to resolve compilation errors

// Disable ANSI (American National Standards Institute) (SQL compliance mode)(ANSI) mode to handle overflow and malformed cast operations
spark.conf.set("spark.sql.ansi.enabled", "false")

df.createOrReplaceTempView("airports")
// With ANSI mode disabled, overflow values will be handled gracefully
var new_df = spark.sql("SELECT *, CAST(build_time AS SMALLINT) as numeric_build_time FROM airports")
new_df.show()
new_df.createOrReplaceTempView("airports")

// Migration change: The to[Collection] method was replaced by the to(Collection) method.
val airports_in_us: Set[String] = spark.sql("SELECT name FROM airports WHERE country='USA'").collect().map(_.getString(0)).to(Set)
println(airports_in_us)
val airports_in_us_java: java.util.Set[String] = airports_in_us.asJava

// With ANSI mode disabled, malformed CAST operations will return null instead of failing
new_df = spark.sql("SELECT *, CAST(code AS INT) as numeric_code FROM airports")
new_df.show()
new_df.write
  .mode("overwrite")
  .parquet(outputPath)

Before

val airports_in_us: Set[String] = spark.sql("SELECT name FROM airports WHERE country='USA'").collect().map(_.getString(0)).to[Set]

After

val airports_in_us: Set[String] = spark.sql("SELECT name FROM airports WHERE country='USA'").collect().map(_.getString(0)).to(Set)

Code change explanation:

Scala 2.13 changed the collection conversion API. The .to[Collection] syntax was replaced with .to(Collection) using parentheses instead of square brackets.
Updated collection conversion from .to[Set] to .to(Set) to comply with Scala 2.13+ syntax requirements.
Changed import scala.collection.JavaConverters._ to import scala.jdk.CollectionConverters._ and updated .to[Set] to .to(Set).
Renamed the object from Spark3_3_Job to Spark4_0_Job. Updated the Parquet config keys from spark.sql.legacy.parquet.int96RebaseModeInRead/Write to spark.sql.parquet.int96RebaseModeInRead/Write.
Added spark.conf.set("spark.sql.ansi.enabled", "false") to handle overflow and malformed cast operations gracefully.
The output path was updated to s3://xxxxxxxxx/output.

After the compilation succeeds, the agent builds the assembly JAR. Choose Save to create a report for the build result.

Step 5: Runtime validation

Provide the following information to run the validation job on Amazon EMR Serverless.

Amazon EMR Serverless application ID (target application running Spark 4.0 on Amazon EMR 8.0.0):

To create an Amazon EMR application, follow the Amazon EMR documentation, or provide a prompt for the agent to create one for you.
Format: 00xxxxxxxxxxxxxxxxxxxxxxxxxx.

Execution role Amazon Resource Name (ARN) (IAM role for the job):

Set up the execution role following the IAM role guide.
Format: arn:aws:iam::123456789012:role/YourRoleName.

Amazon S3 staging path (for uploading the JAR and storing results):

Format: s3://amzn-s3-demo-bucket/path/.

AWS profile (the AWS profile to use for CLI commands, found in your mcp_settings.json file):

Example: default, dev, pro.

After you submit this information, the agent uploads the JAR to Amazon S3 and submits the validation job with the following arguments:

{
  "analysis_id": "a8869720-e005-41b1-89f3-620e1c5663c0",
  "application_type": "EMR-Serverless",
  "compute_id": "xxxxxxxxxxxxxx",
  "compute_run_config": {
    "executionRoleArn": "arn:aws:iam::xxxxxxxx:role/data-processing-mcp-role",
    "jobDriver": {
      "sparkSubmit": {
        "entryPoint": "s3://xxxxx/xxxxxxx/xxxxxxxxx-xxxxxxx-xxxxxxxx/xxx-job-assembly-1.0.jar",
        "entryPointArguments": [],
        "sparkSubmitParameters": "--class Spark4_0_Job --conf spark.executor.cores=4 --conf spark.executor.memory=16g --conf spark.driver.cores=4 --conf spark.driver.memory=16g --conf spark.executor.instances=2"
      }
    },
    "configurationOverrides": {
      "monitoringConfiguration": {
        "cloudWatchLoggingConfiguration": {
          "enabled": true,
          "logGroupName": "/aws/emr-serverless"
        }
      }
    }
  },
  "enable_data_quality_check": false,
  "s3_staging_path": "s3://xxxxxxx/xxxxxxxxx/",
  "is_source_version_run": false,
  "spark_metadata_file_put_presigned_url": null,
  "static_artifacts_file_paths": null
}

The agent monitors the job status upon approval.

Agent output showing the Amazon EMR Serverless validation job completing with a success status

Amazon EMR Serverless job validation output showing successful completion

Once you receive a success response, the agent proceeds to generate the upgrade summary for your Spark project.

{"success":true,"message":"EMR SERVERLESS job completed successfully","compute_run_id":"xxxxxxxxxx","compute_id":"xxxxxxxxxx","status":"SUCCESS","application_type":"EMR-Serverless","next_action":"Proceed to next step in the plan"}

After the job succeeds, you can optionally cross-check the results by navigating to your Amazon S3 bucket to verify the output files were written successfully, or by reviewing the job run details in the Amazon EMR Serverless console to confirm execution status and logs. Choose Approve to allow the agent to save the summary to a local file and upload it to Amazon S3.

You can also request ANSI mode enablement through the agent interface, and it applies the necessary updates accordingly.

Relevant Spark changes:

Migration rule: Spark 4.0 enables ANSI mode by default. To handle type conversion errors gracefully while keeping ANSI mode enabled, use TRY_CAST instead of CAST.
Change description: Enabled ANSI mode and replaced CAST with TRY_CAST for operations that might fail, specifically timestamp-to-smallint overflow and string-to-int malformed value conversions.

Applied changes:

Code diff — src/main/scala/job_script.scala:
- Changed spark.conf.set("spark.sql.ansi.enabled", "false") to spark.conf.set("spark.sql.ansi.enabled", "true").
- Replaced CAST(build_time AS SMALLINT) with TRY_CAST(build_time AS SMALLINT).
- Replaced CAST(code AS INT) with TRY_CAST(code AS INT).

The agent compiles the change and follows the previous steps to run the job on Amazon EMR Serverless.

Result: SUCCESS

In this Scala workload migration section, the agent automatically upgraded the ecommerce pipeline from Spark 3.3 and Scala 2.12 on Amazon EMR 6.9.0 to Spark 4.0 and Scala 2.13 on Amazon EMR 8.0.0. It rewrote build.sbt and plugins.sbt, upgraded the JDK from 11 to 17, and applied Scala 2.13 syntax fixes (.to(Set), CollectionConverters), Parquet config key updates, and ANSI mode handling with TRY_CAST replacements. The upgraded JAR was compiled, submitted to Amazon EMR Serverless, and validated with a SUCCESS status, completing the full migration without manual code edits.

Clean up

To avoid ongoing charges, delete the resources created during this walkthrough. Start by emptying the Amazon S3 staging bucket, then delete both AWS CloudFormation stacks in reverse order:

Empty the Amazon S3 staging bucket.

aws s3 rm s3://${STAGING_BUCKET_PATH} --recursive

Delete the Amazon EMR Serverless application stack.

aws cloudformation delete-stack --stack-name spark-emr-serverless-upgrade

Delete the MCP setup stack (IAM role and Amazon S3 bucket).

aws cloudformation delete-stack --stack-name spark-upgrade-mcp-setup

Conclusion

The AWS Spark Upgrade Agent transforms what has traditionally been a months-long, error-prone migration process into an automated, IDE-driven workflow that completes in hours. By combining intelligent code analysis, targeted transformations, and an iterative local-to-remote validation loop, the agent handles the complexity of upgrading Scala workloads from Spark 3.x to Spark 4.0 on Amazon EMR 8.x. The demo_1_spark_change_focus walkthrough demonstrates the ability of the agent to automatically update build configurations, apply Scala 2.13 syntax changes, handle Spark 4.0 breaking changes like ANSI mode defaults, and validate results against live Amazon EMR clusters, all through natural language prompts in your IDE. For teams managing large-scale Spark estates, this approach eliminates manual debugging cycles, reduces migration risk, and unlocks the performance gains of Spark 4.0 without the traditional engineering overhead.

Next steps:

If you’re new to the Spark Upgrade Agent, start with the introduction post for a lighter-weight introduction before tackling Scala workloads.
For a complete PySpark implementation and demo, refer to Upgrade PySpark from Spark 3.5 to Spark 4.0 with AWS Spark Upgrade Agent.
When you are ready for production, review the security model in the Architecture section and the IAM role setup guide to confirm your least-privilege configuration before running against production workloads.

Useful resources:

Have questions or feedback? Share your migration experience in the AWS re:Post community or open an issue in the sample repository. We’d love to hear how the agent performs on your workloads.

About the authors

How Mapfre USA modernized fraud claims with Amazon EMR Serverless

2026-07-15 Lijan Kuniyil

Post Syndicated from Lijan Kuniyil original https://aws.amazon.com/blogs/architecture/how-mapfre-usa-modernized-fraud-claims-with-amazon-emr-serverless/

Insurance fraud remains a significant challenge for the insurance industry because fraudulent claims can increase loss costs, reduce trust, and consume investigation capacity that could otherwise be focused on serving customers. Traditional fraud detection approaches typically rely on rules-based controls, manual investigation triggers, historical claim patterns, and structured-data-only analysis. These approaches are useful for known fraud patterns, but they can struggle to detect sophisticated fraud rings or hidden relationships across claimants, policies, vehicles, providers, addresses, and prior suspicious activities.

Mapfre USA is the number one auto and home insurer in Massachusetts, serving customers in 11 states nationwide. Our coverage includes auto, home, motorcycle, watercraft, business insurance, and more. As part of Mapfre Group, we’re a worldwide leader serving over 31.1 million customers in more than 100 countries with a team of 31,000 employees. In collaboration with AWS and Neo4j, Mapfre USA modernized its fraud prevention capabilities by combining graph-based features with machine learning (ML) models deployed on AWS. This initiative, focused initially on Massachusetts Auto insurance and later expanded to Home (HO), has delivered significant business impact, exceeding $5 million Net Present Value (NPV), with realized savings already outperforming projections.

In this post, we share how Mapfre USA designed and implemented this solution, highlight the technical architecture running on AWS specifically on the Mapfre Data Platform called Atenea, and explore lessons learned that can apply to other industries facing complex fraud challenges.

Business challenge

Fraudulent claims aren’t always isolated events. They often involve hidden networks of policyholders, vehicles, providers, and prior suspicious activities. Detecting these complex relationships requires going beyond traditional structured data analysis.

Mapfre set out with a clear goal:

Goal: Improve fraud detection accuracy and claims handling efficiency.
Key KPI: Identify fraudulent claims missed by traditional methods.
Approach: Develop several ML models that use both traditional structured data and graph-based features derived from claim relationships.
Deployment: Integrate seamlessly with Guidewire Claims, so front-line adjusters automatically receive fraud alerts with explanations.

Each flagged claim exposure generates a Guidewire activity showing the top three model drivers, helping investigators understand why the claim was identified and act quickly.

Technical solution on AWS (Atenea Data Platform)

The fraud detection platform is built on a modern data architecture on AWS, designed to scale efficiently and provide long-term governance.

At its core, the solution uses Apache Iceberg tables stored on Amazon Simple Storage Service (Amazon S3), with metadata managed through the AWS Glue Data Catalog and access governed through AWS Lake Formation as part of the Atenea lakehouse governance model. The platform feature store is implemented through feature-store-managed Iceberg tables that manage model features, predictions, and Guidewire activities. The implementation is structured across three logical layers:

Silver Layer – Iceberg tables that contain source data from each of the sources, used as the initial consumption point of the platform.
Gold Layer – Iceberg tables storing intermediate data, such as unified Guidewire activity logs, Auto features, and Home features.
Platinum Layer – Feature Store-managed Iceberg tables containing encoded features and model predictions, making them reusable across models and ensuring strong metadata governance.

Processing pipelines are executed on Amazon EMR Serverless, with orchestration managed by Apache Airflow operators running on Amazon Managed Workflows for Apache Airflow (Amazon MWAA). This provides elastic, cost-efficient compute for both batch processing and fast-time scoring, while keeping orchestration, monitoring, and recovery centralized.

For graph enrichment, the platform connects to Neo4j using a dedicated driver, enabling advanced network-based features like suspicious claim linkages, provider fraud ratios, and centrality metrics.

This architecture supports efficient, reliable, and transparent production execution through repeatable Airflow orchestration, environment-based continuous integration and delivery (CI/CD) promotion, centralized monitoring, failure notifications, retry mechanisms, dead-letter queue handling for Guidewire integration, and controlled secret management. At the same time, the layered lakehouse design keeps the platform flexible enough to evolve with new business needs and fraud detection use cases.

Architecture diagram showing the Mapfre USA fraud detection platform on AWS, including data ingestion, graph enrichment, model scoring, and Guidewire integration

The data sources are policy, claims, vehicles, and notes (from AS400 and Guidewire), which include structured data and derived features capturing entity relationships (graph data).

The following list describes the architecture overview:

Data ingestion – Claim batch data uploaded to Amazon S3. Data gets standardized and materialized in Iceberg tables within the Silver layer.
Graph enrichment – Data processed to update Neo4j graph database hosted on AWS.
Model training and scoring – Batch scoring for several ML models.
Model orchestration – Unified orchestration for ingestion, training, and inference using Apache Airflow operators. CI/CD pipelines for promotion across environments.
Execution platform – Amazon EMR Serverless for cost-efficient Spark processing. Migration to Apache Iceberg plus AWS Glue Data Catalog for scalable metadata handling.
Integration with claims systems – Fraud predictions automatically create Guidewire activities, enriched with a description for investigators.
Secrets and security – AWS Secrets Manager securely stores credentials and tokens for Guidewire API integration, with environment-specific and Region-specific access controls.
Monitoring and reliability – Amazon CloudWatch and Amazon Simple Notification Service (Amazon SNS) provide visibility into pipeline health and notify teams on failures. Data quality checks are executed at key stages of the pipeline to validate data availability, schema consistency, completeness, and business-rule expectations before outputs are consumed by models or sent to Guidewire.

Guidewire integration with MLOps on AWS

One of the most important parts of Mapfre’s solution was closing the loop between ML predictions and the claims handling system. This required a resilient integration between the Atenea data platform on AWS and Guidewire Claims.

The following describes the integration flow:

When an ML use case finishes scoring, the results are written as JSON files into the S3 path: <bucket_name>/guidewire/.
An S3 event notification triggers an AWS Lambda function.
This Lambda function:
- Reads the JSON file.
- Calls the Guidewire Predictive Model API.
- Because Guidewire doesn’t support batch requests, the Lambda function sends each JSON payload individually. This keeps the integration compatible with Guidewire and isolates failures at the individual activity level, but it increases the number of API calls and makes retry, throttling, DLQ handling, and monitoring controls important.
If successful, the API responds with HTTP 201 (activity created).
- If not, the Lambda function retries up to two times.
- Failed requests are sent to an Amazon Simple Queue Service (Amazon SQS) dead-letter queue (DLQ), and an Amazon SNS notification is published for monitoring.
Secrets are stored in AWS Secrets Manager and injected as Lambda environment variables, along with AWS Region-specific URLs for token retrieval and API endpoints.
The following JSON shows the example structure for Guidewire integration:

{
  "method": "createPredictiveActivity",
  "params": [
    {
      "claimNumber": "AUXXXXXXX",
      "exposureNumber": 1,
      "subject": "Fraud alert from ML model",
      "description": "Claim flagged as potential fraud based on graph + ML features",
      "shortSubject": "ML_Fraud_Flag",
      "priority": "high",
      "availableForClosedClaim": true,
      "autoCloseOnExposureClosure": false,
      "targetDays": 4,
      "escalationDays": 6
    }
  ]
}

Diagram showing the Guidewire integration flow with AWS Lambda, Amazon SQS dead-letter queue, and AWS Secrets Manager

Key benefits of this integration:

Real-time actionability – Fraud predictions automatically create Guidewire activities for front-line adjusters.
Resilience – Built-in retries, DLQ handling, and Amazon SNS alerts make sure failed events aren’t lost.
Security – Secrets and tokens are managed using AWS Secrets Manager, with strict environment separation (dev, pre, pro).
Scalability – Any new MLOps use case writes results into the S3 output path, automatically flowing into Guidewire.

This integration shows that fraud models don’t exist in isolation but actively augment daily claim workflows in production. It connects Atenea’s MLOps pipelines on AWS directly with business decisioning systems, which is critical to realizing the fraud savings impact.

Data quality and resilience

For robustness, data quality checks are applied on ingestion pipelines and graph features. Automated validation detects anomalies early, monitoring dashboards track KPIs and model performance, and standardized recovery and promotion processes operate across environments.

Visualization and investigative tools

Neo4j Bloom supports SIU workflows by visually exploring entity relationships, such as a provider linked across multiple suspicious claims, accelerating fraud ring identification.

Conclusion

The fraud detection model in auto claims has enhanced Mapfre USA’s ability to identify fraudulent activity, driving significant savings and improving overall claims efficiency.

During the pilot phase alone, savings exceeded projections, and in production the initiative has proven a Net Present Value (NPV) of more than $5M. These results confirm the business case and highlight the strength of combining structured data with graph-based features to uncover fraud networks that traditional approaches miss.

The results have been compelling:

Accuracy gains – Detection improved by 50–135 percent compared to baseline methods.
Substantial realized value – Both during the pilot and in production.
Cross-functional success – The initiative brought together Claims, IT Data, Advanced Analytics, and Neo4j teams in an agile, collaborative model.

Beyond the financial outcomes, several lessons have emerged. First, cross-functional collaboration between groups like Claims, Data Engineering, Advanced Analytics, and technology partners like AWS and Neo4j was critical to success. Second, explainability proved essential. By presenting adjusters with the top model drivers directly in Guidewire, trust and adoption of the system increased substantially. Finally, building resilience into the architecture through monitoring, retries, and data quality processes helped the models operate reliably in production.

Looking ahead, the platform is well-positioned to expand beyond fraud detection. New use cases such as underwriting anomaly detection and customer entity resolution are already on the roadmap. With robust architecture built on AWS using Amazon EMR Serverless, Apache Iceberg on Amazon S3 supported by AWS Glue Data Catalog and Lake Formation, a custom-built Feature Store, and Neo4j, Mapfre now has a scalable foundation to continue driving innovation and business impact.

To learn more about Amazon EMR Serverless, see the Amazon EMR Serverless documentation.

Multi-cloud lakehouse architecture on AWS for Agentic AI, Part 1: Architecture and best practices

2026-07-13 Sakti Mishra

Post Syndicated from Sakti Mishra original https://aws.amazon.com/blogs/big-data/multi-cloud-lakehouse-architecture-on-aws-for-agentic-ai-part-1-architecture-and-best-practices/

Enterprise data architectures have become fundamentally distributed. Over the past decade, organizations have made deliberate investments across multiple platforms such as relational databases for transactional workloads, cloud data warehouses for analytics, object stores for unstructured data, and SaaS applications for domain-specific functions. Each was chosen to solve a specific problem, serve a specific team, or meet a specific performance requirement. The result is not accidental sprawl. It is a deeply heterogeneous data landscape shaped by intentional, workload-driven decisions. The challenge now is not consolidation, but interoperability: enabling these systems to function as a unified foundation for the next generation of AI-driven applications.

Agentic AI systems that autonomously reason, plan, and take action on behalf of users are moving rapidly from experimentation to enterprise production. These systems do not just retrieve information. They synthesize it, act on it, and learn from it. And unlike traditional analytics tools that can work with a well-scoped dataset, AI agents require something more demanding: unified, governed, and real-time access to all relevant enterprise data, regardless of where it lives.

This is the gap that matters most right now. Enterprises that have invested in building strong data capabilities across multiple providers are well-positioned, but only if those platforms can be accessed together, consistently, and with the governance controls that enterprise AI requires. Without a unified data foundation, AI agents operate with incomplete context, governance becomes inconsistent, and the promise of autonomous AI remains out of reach.

Solution approach

The following high-level architecture explains how you can onboard metadata catalogs and MCP servers to your context layer, which becomes the primary input for your AI agents.

Assuming your data products have a well-defined metadata catalog, you can take a unified-catalog-first approach, then build the context layer on top of it to let your AI agents discover all the context from one place. This helps bring in centralized governance and audit control, because every request gets routed through the centralized metadata catalog and context layer to simplify implementation of unified governance. In addition, this brings simplicity to enable business semantics, define attribute priorities, and define authoritative sources for the consumer use cases.

Architecture showing metadata catalogs and MCP servers onboarded to a context layer that feeds AI agents

If any of the data sources does not have a well-defined metadata catalog, you can define Model Context Protocol (MCP) servers on them, and then directly onboard them to the context layer. For example, if you have semi-structured or unstructured datasets for which you do not have a well-defined metadata catalog, or you want to onboard third-party data sources through REST APIs, then you can add their respective MCP server to the context layer directly. The following architecture explains the extended flow for it.

Extended architecture where data sources without a metadata catalog expose MCP servers directly to the context layer

In this series of posts, we demonstrate how you can unify the metadata catalog access across multiple providers, how you can enable AI agents to query the unified catalog, and how the context layer can be integrated to unify metadata from catalogs and MCP servers. We have divided the series into the following parts.

Part 1: Architecture approach with tradeoffs to unify a multi-cloud lakehouse architecture that can power Agentic AI (this post).
Part 2: Implementing an example solution to unify catalogs from multiple providers and deploy AI agents to query the unified data access layer.
Part 3: Integrate a context layer on top of the unified catalog for AI agents.
Part 4: Onboard additional data sources to the context layer through MCP servers and demonstrate the full solution.

This post focuses on explaining the architecture approach to build the open lakehouse architecture on AWS, unifying the metadata catalog across providers for the AI agents to access. In addition, it highlights the architecture trade-offs and best practices.

Use case

Every AI initiative launched on a fragmented data foundation is an initiative that will need to be rebuilt. Organizations that establish unified data access today are the ones that will scale Agentic AI with confidence tomorrow. Consider a large enterprise managing petabytes of data across a diverse set of environments:

On-premises: Network device telemetry, customer records, and operational databases.
Multiple cloud platforms: Marketing analytics, HR systems, and enterprise applications distributed across cloud providers.
Data platforms: Data science workloads, feature engineering pipelines, and finance and supply chain analytics running on specialized platforms.
SaaS applications: Salesforce, SAP, Zendesk, ITSM, and other business tools that each hold a critical piece of the enterprise data picture.

The business objective is to build a unified analytics and AI platform that can:

Query and analyze data across all environments without requiring full data migration.
Enforce consistent data governance and access control regardless of data location.
Power AI agents that can autonomously discover, query, and act on enterprise data.
Reduce total cost of ownership by eliminating redundant pipelines and storage.

This architecture directly addresses these needs by combining flexible data integration patterns, an open-table-format-based lakehouse architecture (with an example of Apache Iceberg), AI agent deployment to access unified metadata, and centralized governance.

Reference architecture

Before going deeper into a specific architecture, let’s revisit at a high level how the AWS open lakehouse architecture enables data ingestion and query or catalog federation to power analytics, machine learning development, and generative AI application development.

The following architecture diagram represents an end-to-end flow that includes:

Data ingestion to the data lake or data warehouse through Zero-ETL and batch or stream processing using AWS native services, or accessing data from Google Cloud Platform using AWS Interconnect – multicloud.
A centralized metadata catalog layer that includes data on AWS and metadata representation of non-AWS data sources using query or catalog federation.
A context layer that you can integrate to create a knowledge graph with ontology and business semantics that can enrich context for AI agents.
The consumption layer, which can include analytics, machine learning model development with Amazon SageMaker AI, and generative AI application development with Amazon Bedrock AgentCore, Amazon Quick, or other AWS and non-AWS AI applications.

End-to-end AWS open lakehouse architecture spanning ingestion, catalog, context, and consumption layers

Let’s look at an expanded version of this architecture that details the data ingestion and data consumption patterns to build a unified data access layer on AWS that spans multiple cloud and ISV providers.

Expanded technical architecture walkthrough

The following architecture demonstrates the comprehensive AWS approach for metadata catalog consolidation through flexible integration patterns, and it also highlights patterns for building a lakehouse on AWS. Built on the open standards of Apache Iceberg for storage and governance through AWS Lake Formation, it creates a unified data foundation that connects existing investments without requiring wholesale migration, and it makes enterprise data AI-ready from day one. This architecture delivers value at every layer: business teams query across platforms without data movement, IT teams manage governance through a single federated layer with the flexibility to federate or ingest per use case, and compliance teams enforce policies once across all sources with full lineage and audit coverage.

Expanded lakehouse architecture on AWS showing federation and ingestion patterns across multiple cloud and ISV providers

The following are the key components of the architecture.

Data access methods

This section provides options to access data that is not available in AWS Glue Data Catalog and not available on AWS.

1. Iceberg catalog federation (Reference points 2, 6.1, 6.2)

AWS Glue Data Catalog implements the Iceberg REST Catalog API specification, which enables seamless federation with Databricks, Snowflake, or other Iceberg-compatible catalogs set up with Amazon Simple Storage Service (Amazon S3) as the storage layer.
With the growing adoption of Apache Iceberg, catalog federation will become a common standard in the future and simplify metadata unification.

2. Query federation (Reference point 1.1)

Direct cross-cloud querying over the public internet to Google BigQuery, Azure SQL, Salesforce, and other platforms.
Real-time access to external data sources without replication, and seamless access with AWS analytics services.
Provides flexibility, because the catalog federation capability of the Iceberg REST catalog is limited to Iceberg tables only.

2.1. Secured private connectivity to Google Cloud Platform using AWS Interconnect for multi-cloud (Reference points 3.1, 3.2)

The default query federation approach makes the connection and transfers data over the public internet, which has its own latency implications depending on the target platform and the data volume transferred over the internet. During re:Invent 2025, AWS announced the public preview of AWS Interconnect – multicloud, which recently became generally available.

AWS Interconnect – multicloud is a managed service that provides private, high-speed, and secure network connections between Amazon Web Services (AWS) and other cloud providers, starting with Google Cloud Platform (GCP), with Microsoft Azure and Oracle Cloud Infrastructure (OCI) coming later in 2026. You can enable the integration with three steps: 1) specify the target cloud service provider, 2) select the destination Region on the other side, and 3) pick the required bandwidth.

The following architecture represents AWS and GCP integration with AWS Interconnect – multicloud.

High-level architecture of AWS and GCP integration through AWS Interconnect for multi-cloud

On the AWS side, you need an AWS Direct Connect gateway (a global construct that acts as a route reflector), which you can attach to your Amazon Virtual Private Cloud (Amazon VPC) through a virtual private gateway or AWS Transit Gateway, or AWS Cloud WAN. On the GCP side, you need a Google Cloud Router that you attach to your customer VPC. Interconnect – multicloud offers pre-cabled capacity pools at shared Interconnect points of presence (PoPs) in selected Regions, where both AWS and GCP routers are co-located and pre-wired.

Because Interconnect – multicloud primarily routes traffic within the VPC through a private network, to benefit from it you need to keep your query engine or jobs within a customer VPC.

2.2. High network bandwidth with on-premises systems (Reference point 4)

AWS Direct Connect for high-bandwidth, low-latency on-premises connectivity.

Data ingestion methods

This section focuses on ways you can use to onboard datasets (complete or subset) to a lakehouse on AWS.

1. Zero-ETL: Data movement to AWS with Zero-ETL ingestion (Reference points 5.1, 5.2)

AWS Zero-ETL capabilities for seamless data loading from AWS and non-AWS sources.
Flexibility to choose your target as an Amazon S3 based data lake or Amazon Redshift.

2. Extract, transform, load (ETL): Extract data from JDBC or SaaS sources and transform through a batch or stream pipeline (Reference points 3.1, 3.2)

Option to design batch and stream ingestion pipelines using AWS managed services with open source data processing engines such as Apache Spark and Apache Flink.
Flexibility to choose from multiple deployment options, including managed clusters on Amazon Elastic Compute Cloud (Amazon EC2), such as Amazon EMR on EC2; serverless options, such as Amazon EMR Serverless and AWS Glue; and managed clusters on Amazon Elastic Kubernetes Service (Amazon EKS), such as Spark or Flink on Amazon EKS or Amazon EMR on EKS.
Hundreds of connectors available as part of AWS Glue to extract data from JDBC and SaaS sources, and the flexibility to design custom connectors that can run on serverless Glue clusters.

The following architecture expands the flow 1.1 to 1.2 ingestion method that integrates AWS services to onboard data to the Amazon S3 raw layer and then takes it through an ETL pipeline for data cleansing and transformations. It also includes steps to onboard unstructured data to Amazon S3 using Amazon Bedrock Data Automation, and taking the lakehouse data for machine learning development with Amazon SageMaker AI.

Ingestion architecture integrating AWS services to load data into the Amazon S3 raw layer and process it through an ETL pipeline

You can also use AWS Interconnect – multicloud to run Spark jobs (Spark with Amazon EMR on EKS or open source Spark on any compute within a customer VPC) to ingest and transform data from Google Cloud with private connectivity.

3. Accessing data from Google Cloud over a private network

Refer to the preceding data access methods (3.1 and 3.2).

4. Onboarding data from AWS Outposts (S3 on Outposts) (Reference points 9.1 to 9.5)

Option to onboard S3 on AWS Outposts data to regional Amazon S3 through AWS DataSync (reference 9.1 to 9.3), which might be a better fit to sync files as-is through a scheduled batch or an event-driven approach.
Flexibility to transform the S3 on Outposts data using an Amazon EMR clusters on Outposts job, and then directly write the transformed output to a regional Amazon S3 bucket in the formats you want (including open table formats such as Apache Hudi, Apache Iceberg, and Delta Lake).

Lakehouse foundation with Apache Iceberg

By standardizing on Apache Iceberg, you’re not choosing AWS over your other platforms. You’re choosing interoperability and future flexibility. Your data becomes truly portable across any Iceberg-compatible engine.

Open table format: Industry-standard format supported across AWS, Databricks, Snowflake, and other platforms, which eliminates vendor lock-in.
ACID transactions: Reliability with full transactional consistency.
Time travel and schema evolution: Built-in versioning and flexible schema management.
Performance optimization: Advanced features such as hidden partitioning, partition evolution, and metadata management.

Note that lakehouse storage is not limited to the Apache Iceberg format, and you have the flexibility to include other open table formats (for example, Apache Hudi and Delta Lake) or file formats (for example, Apache Parquet and Apache Avro).

Unified governance and access control

AWS governance capabilities transform the lakehouse from a storage layer into a fully governed data platform. This delivers security, compliance, and data quality out of the box, applied consistently across all data sources including federated catalogs. A unified catalog consolidates metadata from AWS and non-AWS sources with generative AI-powered business glossary generation, while automated ML-powered classification identifies sensitive data (for example, PII, PHI, and financial data) across structured and unstructured datasets. AWS Identity and Access Management (AWS IAM) and AWS Lake Formation enforce fine-grained access control at the row, column, cell, and tag level, applied consistently across Amazon Athena, Amazon Redshift Spectrum, Amazon EMR, and federated sources. End-to-end data lineage tracking provides visual data flow graphs, impact analysis, and compliance audit trails. When AI agents explore metadata from the unified catalog and submit a query to Amazon Athena for execution, the Lake Formation fine-grained access control filters data based on the user interacting with the AI agent.

For the foundation model integrated into your AI agents, you can use Amazon Bedrock Guardrails, which implements customized safeguards to block harmful content and minimize hallucinations. Amazon Bedrock AgentCore provides fine-grained policy control over agent actions with real-time enforcement and managed authentication for agents accessing AWS and third-party services.

A comprehensive audit and compliance stack spans Amazon CloudWatch, AWS CloudTrail, AWS IAM, AWS Key Management Service (AWS KMS), AWS Audit Manager, and AWS PrivateLink. This stack makes sure every agent invocation is traceable, every key is managed, and every configuration is automatically mapped to frameworks including ISO, SOC, GDPR, and HIPAA.

When an end user interacts with the AI chat assistant, the layers of security and governance should go through the following.

Layer 1: Who can access?

Enable Active Directory and single sign-on integration for user authentication, and a combination of AWS IAM roles for AWS API-level authorization.

Layer 2: What can they see?

Integrate an agent profile to define what datasets each agent can access, because not all agents should have access to all datasets.
Enable fine-grained access control on the metadata layer using AWS Lake Formation that can filter rows and columns.
Enable data masking as applicable while the query responses are served through the query engine.

Layer 3: What can the agent do?

Control agent actions by restricting them to read-only, and apply restrictions to INSERT, UPDATE, and DELETE if the agents are supposed to query only.
Apply a limit on the number of rows that can be returned from the query, and apply a query scan limit to reduce cost.

Layer 4: What does the agent reveal?

Enable output filtering to make sure no PII is included.
Apply Amazon Bedrock Guardrails on large language model (LLM) responses to make sure the model does not produce anything inappropriate.
In addition, enable audit logging of all queries to make sure future audit and compliance needs can be met.

Comprehensive analytics ecosystem (Reference points 7.1, 7.2, 7.3)

AWS offers a complete analytics ecosystem that includes the following.

Amazon Athena: Serverless SQL queries with Iceberg v2 support, including provisioned capacity for consistent performance and workgroups for resource and cost management.
Amazon Redshift Spectrum: Federated queries across the data warehouse and Iceberg data lake.
Amazon Quick Sight: Enterprise visualization with governed access to all data.
AWS Glue and Amazon EMR: Distributed data processing capability for enterprise transformations.

AI-ready architecture (Reference points 8.1 to 8.4)

A consolidated lakehouse architecture helps you make data ready for AI agents that can access the data through readily available MCP servers or through the AWS SDK for Python (Boto3) for Amazon Athena or Amazon Redshift Spectrum. AI agents can integrate the AWS MCP Server to interact with AWS analytics services such as AWS Glue, Amazon Athena, and Amazon S3 Tables, a capability of Amazon S3, to query both data and metadata.

AI agents need context to understand how the catalog tables and their attributes are linked to each other, how users have queried them in the past, or what priorities are defined to understand which one is an authoritative source for a particular natural language question. To enable the AI agent with additional context, we can integrate the AWS Context service that was pre-announced recently at the AWS New York Summit 2026.

Governance integration: AI agents automatically inherit Lake Formation permissions, because the agent can submit the SQL query to be run through Amazon Athena or Amazon Redshift Spectrum. This makes sure they only access data that users are authorized to see. Amazon SageMaker Unified Studio data lineage tracks AI agent queries for full auditability.

The following diagram represents how the AI agent request flow looks.

AI agent request flow through the unified catalog, Lake Formation governance, and Amazon Athena

This architecture delivers value across every layer of the organization. Business teams gain faster time-to-insight by querying data across all platforms without waiting for data movement, while eliminating duplicate storage and reducing transfer costs through federation. The Apache Iceberg open table format ensures data portability and freedom from vendor lock-in. For IT and data teams, a single governance layer across all sources, including federated catalogs, reduces operational complexity, while the flexibility to choose between federation and ingestion for each use case, combined with the elastic AWS infrastructure and the petabyte-scale metadata architecture of Iceberg, delivers both agility and scalability. Data governance and compliance teams benefit from a single point of policy enforcement across all data regardless of location, complete lineage and access logs for audit and compliance reporting, automated sensitive data classification, and policies that are defined once and enforced everywhere, including across federated sources.

Architecture tradeoffs and best practices

The following are a few key trade-offs you need to consider while designing the solution.

Data ingestion and access methods

Use catalog federation (Iceberg REST) when:

The source platform supports the Iceberg REST API (Databricks, Snowflake Polaris).
Data is already in Iceberg format with Amazon S3 backed storage.
You want bidirectional discovery (AWS tables visible in Databricks or Snowflake too).

Use query federation (Amazon SageMaker Lakehouse architecture or AWS Glue connectors) when:

The source is BigQuery, SQL Server, or another non-Iceberg platform.
Data must stay in the source cloud (sovereignty, contractual, or latency reasons).
Real-time access is required without replication lag.

Use ingestion (Zero-ETL, AWS Glue, or Amazon EMR) when:

Data is accessed frequently with a low-latency requirement by AI agents or high-concurrency analytics.
The business decides to build a data lake and warehouse on AWS.
You need full governance, time travel, and performance optimization.

Use AWS Interconnect – multicloud when:

You need real-time or near-real-time query federation to GCP data sources (BigQuery, AlloyDB, Cloud Spanner) and latency or security requirements prohibit public internet routing.
You have high-volume, recurring data transfers between AWS and GCP where public internet egress costs or bandwidth variability are unacceptable.
Your organization has compliance or regulatory requirements mandating that data never traverse the public internet (HIPAA, PCI-DSS, or financial services regulations).
You need bidirectional connectivity, such as GCP workloads calling AWS APIs, or AWS workloads calling GCP APIs, both over private paths.

Choosing between federation and ingestion based on use case

Dimension	Federation (Query in Place)	Ingestion (Move to AWS)
Data freshness	Real-time or near-real-time	Dependent on ingestion frequency
Query performance	Subject to source system latency and network	Subject to data volume and operation, avoids cross-cloud network latency
Cost	Lower storage cost. Higher per-query cost for cross-cloud egress	Higher upfront ingestion cost. Lower ongoing query cost
Governance	Partial. Source system retains some control, and a unified catalog can simplify governance for consumers	Full. Lake Formation enforces all policies across all AWS analytics services
Data portability	Data remains in source	Data fully portable in open format
AI readiness	Limited. Agents depend on source availability	High. Agents query optimized, governed Iceberg tables
Operational complexity	Lower initial setup. Harder to debug cross-cloud issues	Higher initial setup. Simpler long-term operations

Integrating Amazon Bedrock AgentCore Gateway and Amazon Bedrock AgentCore Runtime based on use case

The following are key differences between AgentCore Gateway and AgentCore Runtime that are relevant for our use case.

Dimension	Amazon Bedrock AgentCore Gateway	Amazon Bedrock AgentCore Runtime
Timeout	5 minutes (hard limit)	15 min sync / 8 hours async
Statefulness	Stateless (per-request)	Stateful (session-based)
Best for	Lightweight API proxying	Long-running data processing
Your lakehouse queries	Will time out frequently	Handles multi-hour jobs

Because AgentCore Gateway has a 5-minute hard timeout limit, use AgentCore Runtime for data processing jobs.

AWS Glue ETL jobs can run for minutes to hours.
Amazon Redshift queries on large datasets routinely exceed 5 minutes.
Athena federated queries (especially cross-cloud through Interconnect) can be slow.
Iceberg table scans on multi-TB datasets take time.

You can use AgentCore Gateway if the scope is limited to Glue Data Catalog interactions to fetch metadata schema, because that won’t run for more than 5 minutes.

Design considerations for production implementation

In practice, there are multiple aspects to consider when deploying the solution for production. The following summarizes a few of the key issues you might encounter and approaches to address them.

Catalog federation: The metadata drift problem

One of the first surprises in production is metadata drift, the state where your federated catalog no longer reflects the actual schema of the source system, because the source system’s metadata changes are not reflected in the unified catalog. The agent continues to generate SQL against the stale schema, producing silent failures that are hard to trace.

The following are a few ways you can address the metadata drift issue.

Implement a catalog refresh schedule. Even a daily Glue crawler run against federated sources catches most drift before it causes agent failures.
Add schema validation as a pre-query step in your agent tool. Before running SQL, verify that the referenced columns exist in the current catalog metadata.
Instead of pulling metadata changes from the source in a scheduled manner, you can design an event-driven system, where the source system triggers a push event to run the schema change in the federated catalog.

Query federation: Latency is non-deterministic

Query federation works well for moderate data volumes, but latency becomes non-deterministic at scale. A query that returns in 3 seconds during testing can take more than 10 seconds in production when the source system is under load, the network path is congested, or the federated connector is cold-starting.

The following are a few approaches you can consider to improve the performance.

Set explicit query timeouts in your Athena execution context. Without them, a slow federated query will block your agent indefinitely.
Implement query result caching for frequently asked questions. Most business users ask the same questions repeatedly, and caching at the agent layer improves perceived performance.
For time-sensitive use cases, consider caching aggregated data in an AWS lakehouse on a schedule rather than querying live. This trades freshness for reliability.

AgentCore memory: Statefulness cost

AgentCore Memory enables stateful conversations, but in production, unbounded memory accumulation creates its own problems. An agent that remembers every conversation eventually starts surfacing stale context. For example, a user who asked about Q3 revenue six months ago gets that context injected into a Q1 query today.

The following are a few ways you can optimize cost and improve relevance.

Set explicit memory expiry (we use 30 days as shown in the implementation) and enforce it consistently.
Use session-scoped memory for transactional queries and long-term memory only for user preferences and recurring patterns.
Implement a memory review step in your LangGraph workflow. Before invoking the model, filter retrieved memories by recency and relevance score rather than injecting all of them.

LangGraph orchestration: When tool calls loop

The conditional routing of LangGraph is powerful, but in production we observed a failure mode where the agent enters a tool call loop. The model repeatedly calls the same tool with slightly different parameters, never reaching a satisfactory answer. This typically happens when the tool returns partial or ambiguous results and the model keeps trying to refine.

What we learned:

Add a maximum tool call counter in your LangGraph state. If the agent has called tools more than N times in a single session, force a graceful exit with a summary of what was found.
Return structured, unambiguous responses from your tools. Include row counts, column names, and explicit null indicators so the model can reason clearly about completeness.
Log every tool invocation with its input and output. This is the single most valuable debugging artifact when diagnosing agent misbehavior in production.

Handling hallucination risks in federated agent architectures

This is the most important section for teams moving from prototype to production. Hallucination in agentic AI systems that query real data is qualitatively different from hallucination in general-purpose LLMs, and it is more dangerous because the outputs look authoritative.

There are three distinct hallucination risk zones in a lakehouse AI agent:

SQL generation: The model generates SQL that is syntactically valid but semantically wrong. For example, when asked “What is our revenue growth this quarter?”, the model might generate a query that compares the wrong date ranges, uses the wrong aggregation function, or joins tables on incorrect keys, and then returns a confident, formatted answer with the wrong numbers.
Cross-source synthesis: When the agent queries multiple federated sources and synthesizes results, the risk compounds. The model may correctly retrieve customer counts from Amazon S3 and revenue figures from Snowflake, but incorrectly draw conclusions that aren’t supported by either dataset individually.
Memory-augmented reasoning: When long-term memory is active, the model may blend historical context with current query results in ways that are factually incorrect. For example, it might apply a business rule that was true six months ago but has since changed.

To improve, before any agent output informs a business decision, apply the following three-step validation framework:

Step 1: Source verification. Can you trace the answer back to a specific table, column, and row count? If the agent can’t show you the SQL and the row count, the answer is unverified.
Step 2: Reasonableness check. Does the answer fall within expected ranges? A sudden 10x spike in customer count is a signal to investigate.
Step 3: Cross-validation. For critical decisions, run the equivalent query directly in Athena or your BI tool and compare. Discrepancies reveal either a model reasoning error or a data quality issue. Resolve both before the answer is trusted.

These lessons don’t diminish the value of the architecture. They make it production-ready. The teams that move fastest with agentic AI are not the ones who skip these guardrails. They’re the ones who build them in from the start and spend less time firefighting in production.

Alternative to the unified catalog approach

In case you face technical and process challenges to unify catalogs across providers, you can let each data producer expose the metadata and data through MCP servers, as represented in the following diagram. In this approach, each producer takes the responsibility of maintaining the MCP servers and exposing them to the context layer. While this approach provides autonomy to data owners to operate independently and with flexibility, it also creates operational overhead to synchronize all metadata in a consistent way.

Alternative architecture where each data producer exposes its metadata and data through its own MCP server to the context layer

What’s next

In Part 2 of this series, we walk through the full implementation step by step, including hands-on scripts to:

Load example sales datasets into Databricks and marketing data to Snowflake as Iceberg tables, and federate them into AWS Glue Data Catalog through the Iceberg REST API.
Register Google BigQuery as a native federated data source in Amazon SageMaker, instead of a traditional AWS Lambda connector integration.
Create a customer master table as a native Iceberg table in Amazon S3.
Run a single SQL query in Amazon Athena that joins all four sources across two federation patterns, with no data movement.
Deploy an AI agent on Amazon Bedrock AgentCore that can autonomously query the same unified catalog using Amazon Athena and answer complex business questions in natural language queries. In addition, integrate AgentCore Memory to persist user context.

Conclusion

In this post, we summarized how you can unify data access across multiple cloud and ISV providers on AWS with the combination of catalog federation, query federation, and data movement to AWS. We then explained how AWS Glue Data Catalog and Lake Formation help provide unified catalog and access governance, and how AI agents hosted in Amazon Bedrock AgentCore can access it using MCP servers to explore the metadata context, convert user natural language queries to SQL, and use Amazon Athena to run the query across data sources to get the response to the end user. In addition, we provided an overview of different data ingestion methods to build a lakehouse architecture on AWS, including AWS Interconnect – multicloud and where it adds value.

We also provided architecture trade-offs and best practices to integrate the service capabilities. In the next post (Part 2), we will take a specific use case and provide a step-by-step implementation guide to unify the catalog and deploy the agent to Amazon Bedrock AgentCore.

About the author

How MAPFRE USA modernized fraud claims with Amazon EMR Serverless

2026-07-13 Lijan Kuniyil

Post Syndicated from Lijan Kuniyil original https://aws.amazon.com/blogs/architecture/how-mapfre-usa-modernized-fraud-claims-with-amazon-emr-serverless/

Insurance fraud remains a significant challenge for the insurance industry. Fraudulent claims can increase loss costs, reduce trust, and consume investigation capacity that could otherwise be focused on serving customers. Traditional fraud detection approaches typically rely on rules-based controls, manual investigation triggers, historical claim patterns, and structured-data-only analysis. These approaches are useful for known fraud patterns, but they can struggle to detect sophisticated fraud rings or hidden relationships across claimants, policies, vehicles, providers, addresses, and prior suspicious activities.

MAPFRE USA is a top-rated auto and home insurer in Massachusetts, serving customers in 11 states nationwide. Our coverage includes auto, home, motorcycle, watercraft, business insurance, and more. As part of MAPFRE Group, we’re a worldwide leader serving over 31.1 million customers in more than 100 countries with a team of 31,000 employees. In collaboration with AWS and Neo4j, MAPFRE USA modernized its fraud prevention capabilities by combining graph-based features with machine learning (ML) models deployed on AWS. This initiative focused initially on Massachusetts auto insurance and later expanded to home insurance. It has delivered significant business impact, exceeding $5 million in net present value (NPV) over five years, with realized savings already outperforming projections.

In this post, we share how MAPFRE USA designed and implemented this solution, highlight the technical architecture running on AWS, specifically the MAPFRE data platform called Atenea, and explore lessons learned that can apply to other industries facing complex fraud challenges.

Business challenge

MAPFRE set out with a clear goal:

Goal: Improve fraud detection accuracy and claims handling efficiency.
Key performance indicator (KPI): Identify fraudulent claims missed by traditional methods.
Approach: Develop several ML models using both traditional structured data and 54 graph-based features derived from claim relationships.
Deployment: Integrate with Guidewire Claims, so front-line adjusters automatically receive fraud alerts with explanations.

Each flagged claim exposure generates a Guidewire activity showing the top three model drivers, helping investigators understand why the claim was flagged and act quickly.

Technical solution on AWS (Atenea data platform)

The fraud detection platform is built on a modern data architecture on AWS, designed to scale efficiently and support long-term governance.

Silver layer: Iceberg tables that contain source data from each of the sources. Used as the initial consumption point of the platform.
Gold layer: Iceberg tables storing intermediate data, such as unified Guidewire activity logs, Auto features, and Home features.
Platinum layer: Feature Store-managed Iceberg tables containing encoded features and model predictions, making them reusable across models and ensuring strong metadata governance.

Processing pipelines are executed on Amazon EMR Serverless, with orchestration managed by Apache Airflow operators running on Amazon Managed Workflows for Apache Airflow (MWAA). This provides elastic, cost-efficient compute for both batch processing and fast-time scoring, while keeping orchestration, monitoring, and recovery centralized.

For graph enrichment, the platform connects to Neo4j using a dedicated driver, enabling advanced network-based features like suspicious claim linkages, provider fraud ratios, and centrality metrics.

This architecture supports efficient, reliable, and transparent production execution. It uses repeatable Airflow orchestration, environment-based continuous integration and continuous delivery (CI/CD) promotion, centralized monitoring, failure notifications, retry mechanisms, dead-letter queue handling for Guidewire integration, and controlled secret management. At the same time, the layered lakehouse design keeps the platform flexible enough to evolve with new business needs and fraud detection use cases.

Fraud detection architecture on AWS showing data ingestion to Amazon S3, the Silver, Gold, and Platinum Iceberg layers, Neo4j graph enrichment, Amazon EMR Serverless processing, and Guidewire integration

The data sources here are policy, claims, vehicles, and notes (from AS400 and Guidewire), which are structured data. Derived features that capture entity relationships make up the graph data.

Let’s go through the architecture overview:

Data ingestion – Claim batch data is uploaded to Amazon S3. The data is standardized and materialized in Iceberg tables within the Silver layer.
Graph enrichment – Data processed to update Neo4j graph database hosted on AWS.
Model training and scoring – Batch scoring for several ML models.
Model orchestration – Unified orchestration for ingestion, training, and inference using Apache Airflow operators. CI/CD pipelines for promotion across environments.
Execution platform – Amazon EMR Serverless for cost-efficient Spark processing. Migration to Apache Iceberg plus AWS Glue Data Catalog for scalable metadata handling.
Integration with claims systems – Fraud predictions automatically create Guidewire activities, enriched with a description for investigators.
Secrets and security – AWS Secrets Manager securely stores credentials and tokens for Guidewire API integration, with environment-specific and region-specific access controls.
Monitoring and reliability – Amazon CloudWatch and Amazon Simple Notification Service (Amazon SNS) provide visibility into pipeline health and notify teams on failures. Data quality checks are executed at key stages of the pipeline to validate data availability, schema consistency, completeness, and business-rule expectations before outputs are consumed by models or sent to Guidewire.

Guidewire integration with MLOps on AWS

One of the most important parts of MAPFRE’s solution was closing the loop between ML predictions and the claims handling system. This required a resilient integration between the Atenea data platform on AWS and Guidewire Claims.

Integration flow:

When an ML use case finishes scoring, the results are written as JSON files into the S3 path: <bucket_name>/guidewire/.
An S3 event notification triggers the AWS Lambda function LambdaXXXInvokeGuidewireAPI.
This Lambda function:
- Reads the JSON file.
- Calls the Guidewire Predictive Model API.
- Because Guidewire doesn’t support batch requests, the Lambda function sends each JSON payload individually. This keeps the integration compatible with Guidewire and isolates failures at the individual activity level, but it increases the number of API calls and makes retry, throttling, DLQ handling, and monitoring controls important.
If successful, the API responds with HTTP 201 (activity created).
- If not, the Lambda retries up to two times.
- Failed requests are sent to an SQS Dead-Letter Queue (DLQ) and an SNS notification is published to an SNS queue for monitoring.
Secrets are stored in AWS Secrets Manager and injected as Lambda environment variables, along with AWS Region-specific URLs for token retrieval and API endpoints.

Example JSON structure for Guidewire integration:

{
  "method": "createPredictiveActivity",
  "params": [
    {
      "claimNumber": "AUXXXXXXX",
      "exposureNumber": 1,
      "subject": "Fraud alert from ML model",
      "description": "Claim flagged as potential fraud based on graph + ML features",
      "shortSubject": "ML_Fraud_Flag",
      "priority": "high",
      "availableForClosedClaim": true,
      "autoCloseOnExposureClosure": false,
      "targetDays": 4,
      "escalationDays": 6
    }
  ]
}

Guidewire integration flow from Amazon S3 to an AWS Lambda function that calls the Guidewire API, with an SQS dead-letter queue and Amazon SNS for failures

Key benefits of this integration:

Real-time actionability – Fraud predictions automatically create Guidewire activities for front-line adjusters.
Resilience – Built-in retries, DLQ handling, and SNS alerts keep failed events from being lost.
Security – Secrets and tokens are managed using AWS Secrets Manager, with strict environment separation (dev, pre, pro).
Scalability – Any new MLOps use case writes results into the S3 output path, automatically flowing into Guidewire.

This integration shows that fraud models don’t just exist in isolation but actively augment daily claim workflows in production. It connects Atenea’s MLOps pipelines on AWS directly with business decisioning systems, which is critical to realizing the fraud savings impact.

Data quality and resilience

For robustness, we apply data quality checks on ingestion pipelines and graph features. Automated validation detects anomalies early, monitoring dashboards track KPIs and model performance, and standardized recovery and promotion processes run across environments.

Visualization and investigative tools

Neo4j Bloom supports Special Investigations Unit (SIU) workflows by visually exploring entity relationships, such as a provider linked across multiple suspicious claims, accelerating fraud ring identification.

Neo4j Bloom graph visualization showing a provider node linked across multiple suspicious insurance claims

Conclusion

The fraud detection model for auto claims has enhanced MAPFRE USA’s ability to identify fraudulent activity, driving significant savings and improving overall claims efficiency.

During the pilot phase alone, savings exceeded projections by over half a million dollars, and in production the initiative has proven an NPV of more than $5M at current business volumes. These results confirm the business case and highlight the strength of combining structured data with graph-based features to uncover fraud networks that traditional approaches miss.

The results have been compelling:

Accuracy gains – detection improved by 50–135 percent compared to baseline methods.
Realized value – In 2025, MA Auto and MA Home claim savings reached a combined total of $6.81M, with $6.59M from MA Auto and $225K from MA Home.
Proven return on investment (ROI) – the project delivered an NPV of $4.7M at approval, and results are already exceeding expectations.
Cross-functional success – the initiative brought together Claims, IT Data, Advanced Analytics, and Neo4j teams in an agile, collaborative model.

Beyond the financial outcomes, several lessons emerged. First, cross-functional collaboration between groups like Claims, Data Engineering, Advanced Analytics, and technology partners like AWS and Neo4j was critical to success. Second, explainability proved essential. By presenting adjusters with the top model drivers directly in Guidewire, we increased trust and adoption of the system substantially. Finally, building resilience into the architecture through monitoring, retries, and data quality processes helped the models operate reliably in production.

Looking ahead, the platform is well-positioned to expand beyond fraud detection. New use cases such as underwriting anomaly detection, customer entity resolution, and retention modeling are already on the roadmap. With a robust architecture built on AWS using Amazon EMR Serverless, Apache Iceberg on Amazon S3 supported by AWS Glue Data Catalog and AWS Lake Formation, a custom-built Feature Store, and Neo4j, MAPFRE now has a scalable foundation to continue driving innovation and business impact.

To start building a similar solution, open the Amazon EMR console and review the AWS Architecture Center for reference patterns you can adapt to your own fraud detection and analytics workloads.

About the authors

How BigBasket uses the Iceberg based lakehouse architecture on AWS to power lightning-fast grocery delivery across India

2026-07-06 Annie Mattoo

Post Syndicated from Annie Mattoo original https://aws.amazon.com/blogs/big-data/how-bigbasket-uses-the-iceberg-based-lakehouse-architecture-on-aws-to-power-lightning-fast-grocery-delivery-across-india/

Delivering fresh groceries to millions of customers across India in a few minutes demands a radically modern data architecture and resilient processes to help the business make faster decisions. This is what BigBasket was able to achieve by building a lakehouse architecture on AWS.

In this post, we demonstrate how BigBasket implemented the lakehouse architecture on AWS, including their architecture decisions, implementation approach, and the measurable business results you can expect from a similar modernization. Whether you’re facing scalability challenges or planning your own lakehouse implementation, this blueprint provides actionable insights you can adapt for your organization.

About BigBasket

BigBasket (Innovative Retail Concepts Private Limited) is India’s largest online supermarket, serving millions of customers across over 60 cities. Founded in 2011, the company offers groceries, fresh produce, household items, and personal care products through its mobile app and website, operating subscription services (BBDaily) and quick commerce (bbnow). For BigBasket, the ability to deliver groceries on time isn’t only a competitive advantage. It’s the foundation of customer trust, where every minute counts.

However, rapid business growth brought significant operational challenges:

Inability to consistently meet on-time delivery adherence because of high order volumes, extended travel times, and more, directly impacting key metrics like on-time rate (OTR)-10 mins and OTR-15 mins.
Struggling to meet on-time delivery targets because of picking inefficiency, high order volumes, and extended travel times, directly impacting key metrics like OTR-10 mins and OTR-15 mins.
Delays in stock availability impacting vendor fill-rates, inter-distribution center orders, and warehouse operations.
Inaccurate stock forecasting for top-selling stock keeping units (SKUs), assortment variety, event SKUs, store capacity, and buying cycles.
Lower dark store productivity across picking, stacking, order processing, and goods receipt notes (GRN).

Behind these business challenges lay a fundamental technology problem: the existing data infrastructure couldn’t keep pace. The company experienced rapid store growth, expanding 4x in a short timeframe, which exposed several limitations within their existing data architecture that needed attention.

Understanding the technical bottlenecks

BigBasket’s initial architecture relied heavily on a single data warehouse built on Amazon Redshift to meet all reporting and dashboarding needs. While this traditional approach had served them well initially, several important limitations emerged:

Stale data: Extract, transform, load (ETL) pipelines delivered only day-old (D-1) data, making near real-time analysis impossible for dashboard requirements.
Extended recovery times: Pipeline failure recovery processes took several hours, causing significant delays in data availability for business users.
Schema rigidity: Schema changes in source databases frequently triggered pipeline failures because of a lack of schema evolution support.
Scalability constraints: The infrastructure struggled to handle the sudden load increase from 13,000 to over 35,000 transactions for reports and dashboards with more than 1,000 dataset refreshes.
Cost implications: Increasing data volumes demanded additional compute resources, driving up costs.

Diagram of the scalability and cost limitations of BigBasket’s legacy Amazon Redshift data warehouse

It became clear that the existing data infrastructure wasn’t able to meet the evolving business requirements and a redesign of their data architecture is needed.

Why lakehouse architecture?

A modern data lakehouse architecture addresses these issues with near real-time data processing, flexible schema evolution, and scalable analytics, capabilities necessary for fast-moving commerce operations. The lakehouse approach combines the flexibility and cost-effectiveness of data lakes with the performance and governance features of data warehouses, combining the strengths of both. The design of a data lakehouse provides interoperability across storage systems for combined analytics activities.

Solution overview

BigBasket partnered with AWS to implement a comprehensive lakehouse architecture using a combination of AWS native services and open-source technologies.

The following diagram shows an elaborated view of Bigbasket’s modernized architecture on AWS.

Detailed lakehouse data flow across bronze, silver, and gold medallion layers on AWS

Data ingestion: Enabling continuous replication

AWS Database Migration Service (AWS DMS) ingests data from online transaction processing (OLTP) databases running on Amazon Relational Database Service (Amazon RDS) into the lakehouse on AWS.

This method continuously replicates data with minimal latency, so your analytics reflect near real-time business operations.

Storage and governance: Building a solid foundation

The lakehouse is built on Amazon Simple Storage Service (Amazon S3) and Amazon Redshift, which serve as the centralized data lake and warehouse following a medallion architecture.

The architecture persists all analytical data using Apache Iceberg as the open table format. Iceberg provides a robust foundation for large-scale analytics with the following capabilities:

ACID transactions: Guarantees data consistency and correctness across concurrent read and write operations.
Time travel: Supports querying historical table versions for auditing, troubleshooting, and recovery.
Schema evolution: Allows schema changes without disrupting existing queries or downstream pipelines.

The medallion architecture structures data across three logical layers within the lakehouse:

Bronze layer: Implements change data capture (CDC)-based source replication using AWS DMS. Raw change events flow into Amazon S3 as Apache Parquet files in their original format from source systems, preserving the complete change history. The data pipeline processes and deduplicates these events using Apache Spark on Amazon EMR to create and maintain Apache Iceberg tables that act as replicated source tables.
Silver layer: Represents the conformed data model, where data is cleansed, standardized, and validated with enforced quality checks. This layer contains core dimension and fact tables, modeled for analytical consistency and reuse across domains. Data is stored as Apache Iceberg tables on Amazon S3, making it reliable and performant for downstream analytics and transformations.
Gold layer: Provides business-ready data marts and wide tables optimized for reporting, dashboarding, and domain-specific use cases. These datasets are curated to align with business metrics and key performance indicators (KPIs) and are served from Amazon Redshift, using Iceberg-backed tables to deliver fast, scalable analytics for business intelligence (BI) tools and end users.

This layered approach maintains a clear separation of concerns across raw ingestion, analytical modeling, and business consumption, while supporting scalability and flexibility across the organization. AWS Lake Formation enforces fine-grained data access controls, and the AWS Glue Data Catalog centrally manages metadata across Amazon S3 and Amazon Redshift, ensuring consistent data discovery and governance across the analytics ecosystem.

Data processing: Flexibility and performance

For data processing and transformations, BigBasket uses Amazon EMR with Apache Spark and dbt, orchestrated by Apache Airflow running on Amazon Elastic Kubernetes Service (Amazon EKS) as the core compute layer of the lakehouse. Apache Spark on Amazon EMR handles large-scale distributed processing, including CDC deduplication, incremental transformations, and complex data reshaping. Apache Iceberg serves as the open table format, which provides several critical capabilities.

dbt is used to define and execute transformation logic using SQL, managing the build of data models such as staging, intermediate, and final tables on top of the raw data. dbt uses the dbt-Trino adapter to run these transformations using the Trino engine, materializing the results as Apache Iceberg tables in Amazon S3. This approach provides a simple, modular, and governed way to manage transformations while taking advantage of Iceberg’s transactional guarantees.

These features are necessary for production lakehouse implementations and help you avoid vendor lock-in while maintaining enterprise reliability.

Online analytical processing (OLAP) and analytics: Hybrid approach for cost optimization

The analytics layer uses a hybrid approach that you can adapt based on your query patterns:

Amazon Redshift: For querying of active, frequently accessed data from the Gold layer.
Amazon Athena: For ad-hoc queries on historical data.
Apache Trino: For federated queries across multiple data sources while powering dbt-driven transformations directly on Apache Iceberg tables.

This hybrid strategy optimizes costs by keeping frequently accessed data in Amazon Redshift while querying historical data directly from Iceberg tables in Amazon S3. Amazon Redshift data sharing supports a multi-warehouse architecture for cross-team collaboration, allowing different teams to access shared datasets without data duplication.

Orchestration: Managing complex workflows

Apache Airflow running on Amazon EKS orchestrates and schedules data pipelines across the entire environment, providing visibility and control over complex workflows. This gives you a unified view for monitoring and managing your data operations.

Machine learning integration

Amazon SageMaker AI powers machine learning workloads for predictive analytics and model training directly on lakehouse data, from demand forecasting to delivery optimization. This tight integration means your data scientists can work with the same governed data that powers your analytics.

Visualization: Making insights accessible

Amazon Quick Sight provides data visualization and business intelligence reporting capabilities, making insights accessible to business users across the organization without requiring technical expertise.

Special focus: Clickstream data processing

BigBasket implemented a sophisticated dual-path architecture for processing clickstream data from mobile apps and web interactions:

Real-time path: Data flows through Scala stream collectors on Amazon Elastic Compute Cloud (Amazon EC2) (behind Elastic Load Balancing) to Amazon Kinesis Data Streams and Amazon OpenSearch Service for immediate insights into customer behavior. This path is necessary when you need to react to user actions within seconds, for example detecting fraud or personalizing experiences in real time.
Batch path: The batch path validates data, stores it in Amazon S3, processes it through Amazon EMR, and loads it into Amazon Redshift for comprehensive historical analysis. This path handles data quality checks, enrichment, and aggregation for long-term analytics.

The trade-off between these approaches is latency versus completeness. Real-time processing gives you speed but may sacrifice some data quality checks, while batch processing provides accuracy but introduces delay. This dual approach achieves both immediate operational insights and deep analytical capabilities, letting you optimize for different use cases.

The following diagram shows how the clickstream data is handled and effectively processed today.

BigBasket’s dual-path clickstream processing architecture with real-time and batch paths on AWS

The results: measurable business impact

The data platform transformation achieved significant results across multiple dimensions:

Technical improvements

Near real-time data: Achieved near real-time data availability for dashboards within 3–5 minutes, replacing previously day-old data.
Rapid failure recovery: Pipeline failure re-runs now complete in minutes instead of hours.
Comprehensive governance: Full control over data governance with robust observability, lineage, data accuracy, and consistency.
Enhanced scalability: Successfully handling over 35,000 reports and dashboards with over 1,000 dataset refreshes.

Business outcomes

On-time delivery: Improved monitoring with real-time insights on low-performing stores.
Stock availability: Reduced operational issues with visibility into key bottlenecks.
Stock forecasting: Improved accuracy and availability of top-selling SKUs.
Dark store productivity: Enhanced productivity of warehouse executives across all operations.

Key takeaways: lessons for modern data platforms

BigBasket’s journey offers valuable insights for organizations facing similar challenges:

Quick commerce needs quick observability. In the fast-paced world of quick commerce, faster decision-making directly improves business metrics. Real-time data isn’t a luxury. It’s a necessity.
Embrace ELT for real-time needs. Shifting from traditional ETL to an extract, load, transform (ELT) pattern within a lakehouse architecture is important to unlock near real-time analytics capabilities.
A lakehouse delivers speed and governance. Modern lakehouse architectures don’t force trade-offs. You can achieve both fast data availability and comprehensive control, lineage, and accuracy.
Focus on operational resilience. Designing for rapid failure recovery (re-runs in minutes, not hours) is necessary for maintaining data availability and business trust, especially in customer-facing operations.
Incremental migration. You don’t need to rebuild everything. Evolve your current Amazon S3 data lake or reuse your existing investments in Amazon Redshift to build the data lakehouse capabilities.

The road ahead

BigBasket continues to innovate, now moving to adopt Amazon SageMaker Unified Studio to access all lakehouse components in a simplified manner across the enterprise. This next evolution will further streamline data access and accelerate insights across teams.

The company’s transformation demonstrates that with the right architecture and AWS services, organizations can turn data infrastructure challenges into competitive advantages, delivering not only better analytics but better customer experiences.

As you plan your own lakehouse implementation, use these patterns and lessons learned to accelerate your journey and avoid common pitfalls.

About the authors

Deploy modern data platforms in minutes with MDAA

2026-07-02 Sudeshna Dash

Post Syndicated from Sudeshna Dash original https://aws.amazon.com/blogs/big-data/deploy-modern-data-platforms-in-minutes-with-mdaa/

Modern Data Architecture Accelerator (MDAA) is an open source framework that replaces infrastructure code with concise YAML configuration, so your team can deploy a governed, production-ready data architecture, reducing deployment time from months to weeks (depending on complexity and team experience).

Organizations building modern data architecture on AWS face a critical challenge: deploying production-ready, governed infrastructure traditionally requires 6–12 months of custom development, thousands of lines of infrastructure code, and continuous remediation cycles to maintain security and compliance. Governance is often added incrementally, treated as an afterthought that creates compliance gaps and engineering rework.

MDAA addresses this by replacing infrastructure code with concise YAML configuration, achieving up to 97.6 percent code reduction (from approximately 1,800 lines of AWS CloudFormation to 45 lines of MDAA YAML) while embedding governance from the start. The complete Governed Lakehouse Starter Kit deploys 491 AWS resources across 12 stacks from approximately 450 lines of YAML configuration, representing a 66x verbosity ratio where each line automatically expands into production-ready infrastructure.

In this post, we explore how MDAA transforms data architecture development from months of manual coding to production-ready deployment through configuration-driven infrastructure and embedded governance, examine a real customer transformation, and provide a clear implementation pathway for your own data modernization journey.

Customer use case and challenge

A university system office needed to modernize its analytics architecture across 17 campuses while managing sensitive educational data. Their third-party dependency created bottlenecks that slowed feature implementation from weeks to months, and their IT team lacked the cloud skillsets to build modern infrastructure independently.

With MDAA, they achieved:

95 percent reduction in time-to-value for dashboard and feature implementation (from weeks to hours).
17 campuses integrated into a unified, secure architecture.
7.2TB of data and over 8,000 dashboards migrated successfully.
Significant cost savings by removing third-party dependencies and reducing license costs.
Enhanced security posture for external stakeholders accessing sensitive educational data.

The team used MDAA to implement a modernization strategy with continuous integration and continuous delivery (CI/CD) for automated deployment. The architecture now supports rapid response to stakeholder requests while maintaining strict data governance through AWS Lake Formation.

Their transformation demonstrates what becomes possible when governance is embedded from launch rather than added incrementally, moving from months-long manual development to weeks of production-ready deployment through configuration-driven infrastructure.

Solution: MDAA and its value propositions

MDAA’s capabilities stem from its modular, composable architecture. The accelerator provides over 40 pre-built modules that encapsulate AWS best practices for security, governance, and operational excellence. Organizations describe the outcomes they want in MDAA-specific YAML configuration files (not CloudFormation or Terraform YAML) and the accelerator automatically translates these configurations into AWS Cloud Development Kit (AWS CDK) constructs, which then deploy via CloudFormation with embedded governance.

Configuration over code. The MDAA framework takes a fundamentally different approach: describe the outcomes you want in YAML, and the accelerator deploys production-ready infrastructure with embedded governance. Consider deploying a governed data lake where fraud detection teams need write access to transaction data, while marketing analytics teams require read-only access to customer behavior data. Traditional approaches require over 1,800 lines of CloudFormation across Amazon Simple Storage Service (Amazon S3) buckets, AWS Key Management Service (AWS KMS) keys, AWS Identity and Access Management (IAM) policies, and Lake Formation permissions. With MDAA, the same governed data lake is expressed in 45 lines of configuration, a 97.6 percent reduction, while helping you apply encryption, least-privilege access, and cross-account governance as built-in defaults.

The configuration deploys multi-zone S3 storage with KMS encryption, Lake Formation permissions with tag-based access control (TBAC) enabled, Amazon SageMaker Unified Studio for data product discovery, and encrypted AWS Glue Data Catalog with automated crawlers. All permissions flow through Lake Formation rather than individual IAM policies.

Embedded governance from day one. Governance is declared in YAML and deployed alongside infrastructure from the first run. Fine-grained access controls, encrypted data catalogs, data quality validation, audit trails, and sensitive data classification are all part of the same configuration. MDAA’s Governed Lakehouse starter kit defines an entire governed data architecture in roughly 450 lines of YAML, which produces approximately 29,700 lines of CloudFormation across 12 stacks (a 98.5 percent reduction in infrastructure code).

Modular, composable architecture. Each module is purpose-built to handle a specific capability within the data architecture. Modules communicate through AWS Systems Manager Parameter Store, passing resource identifiers (Amazon Resource Names (ARNs), IDs, and names) between stacks. This approach removes hardcoded dependencies. A KMS key created in one module can be referenced by another through parameter resolution, with all dependencies resolved automatically at deployment time.

The diagram illustrates the deployed architecture and team-level access flow that MDAA generates from the 45-line configuration.

Progressive architecture patterns. MDAA provides four reference architecture patterns that align to progressive stages of data infrastructure maturity:

Basic Data Lake deploys a governed data lake with built-in security controls, data quality checks, centralized metadata management using AWS Lake Formation and AWS Glue.
Data Science Platform extends the data lake with Amazon SageMaker notebooks, feature stores, and machine learning (ML) pipelines so data science teams can experiment and train models on governed data.
SageMaker Unified Studio adds a single interface for analytics and ML collaboration, connecting data engineers, analysts, and data scientists in one workspace.
Generative AI Platform layers Amazon Bedrock and Retrieval Augmented Generation (RAG) capabilities on top of your existing data foundation, so teams can build generative AI applications grounded in enterprise data.

Each pattern builds the one before it. You can start with the Basic Data Lake and adopt additional patterns as your team’s needs grow. MDAA’s modular design means you add capabilities without rearchitecting what you already deployed.

The infrastructure is versioned through GitHub, repeatable across environments, and auditable through comprehensive AWS CloudTrail logging. Data engineers focus on data pipelines and business logic while MDAA manages infrastructure complexity and governance integration. This represents the fundamental shift: from writing infrastructure code to describing the outcomes you want through configuration, with governance embedded from the start.

Use case of MDAA: Governed data architecture

DataOps teams spend significant time on governance tasks, including permissions management, compliance validation, and access control, rather than building pipelines and analytics. These aren’t data problems, they’re governance problems that consume engineering capacity meant for higher-value work. MDAA addresses this at the architectural level. Governance is declared in YAML and deployed alongside infrastructure from the first run.

The following sections walk through how each governance module works in practice.

Amazon SageMaker Unified Studio provides a governed data catalog where data producers publish data products, and consumers discover and subscribe to them. Your deployment with MDAA includes a pre-configured domain, blueprints (managed and custom), projects, and environment profiles, all defined in a single configuration file:

# sagemaker.yaml --- 16 lines that deploy 114 CloudFormation resources
domains:
  domain1:
    dataAdminRole:
      id: ssm:/{{org}}/govern1/generated-role/data-admin/id
    description: SMUS Domain 1
    userAssignment: MANUAL

    tooling:
      vpcId: '{{context:vpc_id}}'
      subnetIds:
        - '{{context:private_subnet_id1}}'
        - '{{context:private_subnet_id2}}'

    groups:
      team1:
        ssoId: '{{context:team1-group-sso-id}}'
      team2:
        ssoId: '{{context:team2-group-sso-id}}'

Behind this configuration, MDAA deploys an Amazon SageMaker Unified Studio domain with dedicated KMS keys, execution and provisioning roles, and single sign-on group profiles for team access. Data producers tag and publish assets with metadata, ownership, and classification. Consumers browse a searchable catalog, see only authorized assets, and request access through a governed workflow. Cross-account and cross-business-unit data sharing flows through a subscription model, ensuring every access grant is tracked, auditable, and revocable.

Use case of MDAA: Restricting access to cardholder data using Lake Formation

AWS Lake Formation provides fine-grained access control at database and table levels, removing manual IAM policy management. MDAA deploys AWS Lake Formation with pre-configured settings that disable IAMAllowedPrincipals, the critical governance setting that ensures all permissions flow through centralized governance:

# lakeformation-settings.yaml --- 6 lines that deploy 25 CloudFormation resources
lakeFormationAdminRoles:
  - id: generated-role-id:data-admin
createCdkLFAdmin: true
createDataZoneAdminRole: true
iamAllowedPrincipalsDefault: false

That last flag is the single most important governance setting in the platform. Without it, an IAM principal with glue:GetTable can read tables in the catalog, bypassing the entire access control model. Most manual setups miss this or defer it.

With the data lake configuration, you declare roles and access policies in YAML where admins get full control, engineers get read access to curated data, extract, transform, and load (ETL) roles get scoped write access, and MDAA compiles them into the correct S3 bucket policies and Lake Formation registrations.

Use case of MDAA: Ensuring data integrity with AWS Glue Data Quality

AWS Glue Data Quality runs automated validation rulesets continuously as part of the pipeline, not as periodic batch checks. MDAA’s data quality module supports over 15 built-in rule types, from completeness and uniqueness checks to statistical thresholds and data freshness validation:

# data-quality.yaml
projectName: example-project

rulesets:
  customer-data-quality:
    description: Validate customer data completeness and uniqueness
    targetTable:
      databaseName: project:databaseName/customer-data
      tableName: customers
    ruleset:
      - ruleType: IsComplete
        column: customer_id
      - ruleType: Uniqueness
        column: email
        comparisonOperator: ">"
        threshold: 0.95
      - ruleType: RowCount
        comparisonOperator: ">"
        value: 100

Quality metrics flow into Amazon CloudWatch for real-time alerting. If anomalies are detected, automated workflows quarantine affected records and alert data engineering teams before issues reach downstream consumers.

Protecting metadata at rest: AWS Glue Data Catalog encryption

Table schemas, column names, and partition structures can reveal sensitive information about an organization’s data architecture, even without access to the underlying data. AWS Glue Catalog Encryption secures metadata at rest using AWS KMS-managed keys. MDAA configures catalog encryption by default, so schema definitions and connection passwords are encrypted from initial deployment without requiring manual key management setup. Access to catalog metadata follows the same Lake Formation governance controls applied to the data itself, so teams see only the schemas that they’re authorized to query.

Auditing every data access event: CloudTrail integration

Every data access event must be logged and attributable to a specific identity. Without a complete audit trail, demonstrating compliance during a regulatory review becomes a manual, error-prone process. AWS CloudTrail captures API-level activity across the data infrastructure, recording who accesses what data, when, and from which service. MDAA configures CloudTrail integration by default, so audit logging is active from initial deployment rather than added retroactively. Log data flows into a centralized, tamper-resistant store, giving compliance teams a single location to query access history across all business units and accounts.

Identifying sensitive data automatically: Macie integration

In large environments, sensitive information spreads across dozens of S3 buckets through pipelines, transforms, and ad hoc data drops, and self-reporting data owners consistently produce gaps. Amazon Macie uses machine learning to automatically discover and classify sensitive data in S3, surfacing findings at the object level without manual tagging. MDAA configures Macie across your S3 buckets during deployment, routing findings to Amazon EventBridge where automated workflows can alert owners or trigger remediation.

Together, these controls form a layered defense: Lake Formation governs access to cataloged data, Glue Data Quality validates integrity on arrival, and Macie identifies sensitive data that lands outside governed pipelines to reduce compliance risk.

Multi-account data mesh

MDAA provides extensive support for multi-account data mesh setups, with decentralized data ownership across business units and centralized governance. The data mesh starter kit supports cross-account data product publishing and consumption, allowing organizations to scale data sharing while maintaining consistent security and compliance controls.

Technical implementation

Ready to deploy your modern data architecture? Here are the resources to get started:

MDAA Implementation Guide provides detailed instructions for deploying all starter packages, including architecture patterns, configuration examples, security best practices, and troubleshooting guidance.

MDAA Hands-on Workshop offers step-by-step guided implementation with AWS experts. The workshop covers configuration management best practices, implementation patterns, hands-on labs with real-world scenarios, and cleanup instructions.

GitHub Repository and Documentation provide source code, module reference, and comprehensive documentation.

Organizations approach MDAA from different starting points. Some modernize existing data architectures, migrating from on-premises infrastructure or legacy cloud architectures. Others build new architectures for artificial intelligence and machine learning (AI/ML) initiatives or generative AI applications. Financial services organizations require PCI-DSS compliance from day one. Healthcare organizations need controls that can help support HIPAA. Each journey benefits from MDAA’s configuration-driven approach and embedded governance.

Conclusion

MDAA transforms data architecture development from months of manual coding to production-ready deployment. Configuration-driven infrastructure reduces development time by 40–60 percent while embedding governance from the start. The university system’s 95 percent reduction in time-to-value demonstrates the outcome: organizations deploy secure, compliant, governed data architectures in weeks rather than months.

Financial services organizations can deploy architectures to help them align with PCI-DSS compliance requirements using Lake Formation access controls, Glue Data Quality validation, SageMaker Unified Studio data discovery, comprehensive CloudTrail audit trails, and automated Macie data classification, all inherited from configuration rather than built manually.

Data architecture journeys need not follow six-month timelines with governance added incrementally. MDAA provides an alternative: describe the outcomes you want through YAML configuration, inherit pre-validated security controls, and deploy production-ready infrastructure with comprehensive governance from initial deployment.

Security and compliance is a shared responsibility between AWS and the customer. For more information, see the AWS Shared Responsibility Model.

Need help or have questions? Contact AWS ProServe for personalized guidance on selecting the right package and deployment strategy for your organization.

About the author

Modernizing financial analytics with Amazon SageMaker Unified Studio

2026-06-22 Umang Aggarwal

Post Syndicated from Umang Aggarwal original https://aws.amazon.com/blogs/architecture/modernizing-financial-analytics-with-amazon-sagemaker-unified-studio/

Avanse Financial Services is one of India’s leading education loan providers. Their Data Engineering Team had built a data lake on AWS using Amazon Simple Storage Service (Amazon S3), Amazon Athena, and AWS Glue for data ingestion and processing. However, their analytics and reporting layer ran on an external analytics application that wasn’t integrated with AWS. Data had to be copied from Amazon S3 into this external application before analysts could run any report, its license consumed a significant portion of their budget despite low utilization, and every integration with AWS services required custom-built pipelines.

After evaluating their options, Avanse migrated to a cloud-native lakehouse architecture using Amazon SageMaker Unified Studio, which unified their data engineering, analytics, and artificial intelligence (AI) workflows in a single governed environment on AWS. In this post, we walk through their migration journey so you can adapt their approach to your own environment.

Why Avanse chose to modernize

The separation between their AWS data lake and their external analytics application created five problems:

Daily data synchronization bottleneck. Every report required a 4-hour batch copy from Amazon S3 into the external analytics application before analysts could query it. Business decisions were based on data that was at least a day old.
Fixed licensing costs disconnected from usage. The external analytics application charged an annual fee regardless of how many queries analysts ran. Avanse needed usage-based pricing that matched what they actually consumed, not a fixed fee for capacity they weren’t using.
Limited auditability. The external analytics application ran on a shared server where different business units (risk, collections, portfolio management) shared the same resources. It lacked granular audit trails, making it difficult to trace who accessed what data and when, or to allocate costs per team.
No centralized data discovery. Although AWS Glue Data Catalog managed schema metadata for the data lake, the external analytics application couldn’t access it. Analysts working in that application relied on folder structures and manual documentation to find the right datasets, slowing onboarding and increasing the risk of using outdated data.
Disconnected from AWS services. The external analytics application couldn’t query data in Amazon S3 or use AWS Glue catalogs natively. Every data flow required connectors and custom-built pipelines, adding maintenance overhead.

Additionally, some datasets were stored on Network File System (NFS) storage outside of Amazon S3, creating another data silo that needed to be consolidated.

Avanse chose Amazon SageMaker Unified Studio because it addressed all five challenges: direct querying of data in Amazon S3 avoiding synchronization, usage-based compute through Amazon Athena and Amazon EMR Serverless, project-based isolation with per-project billing, lineage tracking with AWS IAM Identity Center, and native integration with their existing AWS services.

Solution overview

The core architectural change was moving from a two-application model to a single integrated stack:

Previous architecture: Avanse’s data ingestion and processing ran on AWS (Amazon S3, AWS Glue, Athena), but analytics and reporting ran on an external analytics application. Data had to be batch-copied from Amazon S3 into this external application daily before analysts could query it. Each system had its own access controls, and there was no shared catalog or lineage tracking between them.
New architecture: Analytics now run directly against data in Amazon S3 through Amazon SageMaker Unified Studio. There’s no data copy step. Analysts query the same data that the ingestion pipelines produce, using Athena for SQL and EMR Serverless for large-scale processing. Governance, access control, and lineage are centralized through IAM Identity Center and SageMaker Catalog.

The following diagram illustrates the target architecture. It follows a lakehouse pattern, storing data in open formats on Amazon S3 while maintaining ACID transaction support for the consistency financial regulators expect.

Three-layer lakehouse architecture for Avanse on AWS, showing the data layer with Amazon S3 and AWS Glue Data Catalog, the compute layer with Amazon SageMaker Unified Studio, AWS Glue ETL, AWS Lambda, Amazon EMR Serverless, Amazon SageMaker AI, and Amazon Bedrock, and the governance layer with AWS IAM Identity Center, SageMaker Catalog, and Amazon DataZone

The architecture has three layers:

Data layer – Amazon S3 stores data in open formats (Parquet, Delta Lake) with S3 Intelligent-Tiering for automatic cost optimization. AWS Glue Data Catalog maintains schema metadata, making data discoverable across tools.
Compute layer – Amazon SageMaker Unified Studio provides project-based workspaces organized by business function. Collections uses the built-in SQL Query Editor powered by Athena, Risk Reporting uses JupyterLab for interactive analysis, and MIS runs large-scale Spark jobs through Amazon EMR Serverless. AWS Glue ETL handles data transformations and AWS Lambda provides event-driven triggers for report generation. For machine learning (ML) workloads, Amazon SageMaker AI supports model training and deployment, with Amazon Bedrock available for generative AI capabilities such as enhancing risk narratives.
Governance layer – IAM Identity Center provides SSO and audit logging across workspaces. SageMaker Catalog serves as the business glossary with data lineage tracking and access controls. Amazon DataZone connects components through a common metadata layer.

Migration journey

Avanse followed a five-phase approach. The timelines can be adapted to your environment, but the systematic progression from validation through production deployment is key.

Phase 1: Technical validation (72-hour workshop)

Avanse started with a focused 72-hour workshop using isolated SageMaker environments where developers could experiment without impacting production. Their team tested SQL analytics against existing Athena tables and validated that Python and PySpark could replicate their existing analytics workflows.

The team confirmed that querying data directly in Amazon S3 addressed their synchronization bottleneck entirely. The 4-hour daily data copy was no longer necessary, which validated the migration approach.

Phase 2: Data migration and storage optimization

Avanse migrated datasets from NFS storage and legacy analytics formats into Amazon S3, consolidating the data into a single location. They implemented S3 Intelligent-Tiering, which automatically moves data between access tiers based on usage patterns, optimizing costs without impacting retrieval performance.

They replaced legacy analytics connectors with native Athena workgroups within SageMaker Unified Studio, avoiding data synchronization entirely. Source data remained in Amazon S3, queryable by both Athena SQL and SageMaker notebooks, establishing a single source of truth.

Phase 3: Compute modernization

Avanse moved from a shared analytics server to project-based isolation in SageMaker Unified Studio. Each business function (Risk Reporting, Collections, MIS) received its own project with dedicated compute spaces running JupyterLab. Project-specific IAM execution roles provided access controls and cost allocation per business unit.

A single browser-based URL with multi-factor authentication (MFA) now provides access to SQL analytics using the built-in query editor, ML development in JupyterLab notebooks, and big data processing through Amazon EMR Serverless. This replaced the need for local analytics client installations.

Phase 4: Governance implementation

Avanse deployed SageMaker Catalog as their central business data catalog. Analysts now discover approved datasets through semantic search rather than navigating folder structures or relying on manual documentation. They mapped technical Athena table names to business terms. For example, analysts search for “collection efficiency” and find the relevant tables with descriptions, schemas, and lineage.

Lineage capture traces each metric in risk reports back to source tables, transformations, and intermediate datasets. Every action (notebook execution, SQL query, data access) is tied to IAM Identity Center users, creating the comprehensive audit trail their compliance team needed.

Phase 5: Use case migration

Rather than attempting a big-bang migration, Avanse moved critical workflows one at a time:

Portfolio MIS (Monthly/Fortnightly): Previously required the daily 4-hour data copy from Amazon S3 into the external analytics application before report generation could begin. Avanse avoided the data synchronization step entirely and now generates MIS reports by querying existing Athena tables directly in Amazon S3. Because the source data was already on AWS, there was no need to involve the external application for this activity. Report generation dropped from hours to under 30 minutes.
Collection Efficiency and Bounce Calculation: Ported complex legacy analytics procedures for calculating metrics like collection efficiency and bounce rates to event-driven processing using AWS Glue ETL, AWS Lambda, and PySpark jobs for high-volume data aggregation. The serverless execution model charges only for compute time consumed.
EDW Risk Reporting: Large-scale regulatory joins of Enterprise Data Warehouse assets previously ran as legacy scheduled procedures. These now run as SQL queries in the SageMaker Unified Studio query editor, where analysts execute them on-demand or schedule them through Athena workgroups. The distributed query engine handles complex multi-table joins spanning millions of rows.
Scorecard Generation: Model building shifted from the external analytics application to SageMaker AI workflows. Data scientists use JupyterLab with Python libraries and deploy models directly to SageMaker endpoints, avoiding data movement between separate environments.

Overcoming technical challenges

One technical challenge was code migration. Avanse’s analytics code base contained years of accumulated proprietary scripts and procedures. Direct line-by-line translation was not practical. Instead, they took a pragmatic approach: basic data transformations moved to SQL in Athena, complex business logic was rewritten in PySpark for scalability, and statistical procedures were replaced with Python libraries like pandas and scikit-learn. The approach was to focus on what the code accomplishes, then implement it using cloud-native patterns.

The other technical challenge was performance validation. The team needed to confirm that querying data in Amazon S3 would deliver acceptable performance compared to the external analytics application’s in-memory processing. Queries against Parquet-formatted data in Amazon S3 using Athena delivered comparable performance for standard reporting workloads, while avoiding the 4-hour daily data synchronization step entirely. For large-scale regulatory joins spanning millions of rows, Amazon EMR Serverless provided distributed Spark processing that completed in minutes rather than the hours required in the external application.

Key outcomes

Area	Result
Licensing costs	Avoided external analytics application fees entirely
Storage costs	Reduced through S3 Intelligent-Tiering, which automatically moves data between access tiers based on usage patterns
Report generation	From over 4 hours (including data synchronization from Amazon S3 to the external analytics application) to under 30 minutes with direct Amazon S3 querying
Compliance audits	From weeks of manual investigation to days with automated lineage reports
Compute costs	Usage-based serverless model replaced always-on external analytics infrastructure
Collaboration	Unified browser-based environment for data scientists, analysts, and engineers

“By adopting SageMaker Unified Studio, we as the Data Team eliminated legacy licensing costs, reduced storage and compute expenses with a serverless, usage-based model, and accelerated our periodic report generation. At the same time, we transformed compliance and collaboration by cutting audit timelines while unifying our teams in a single, efficient data environment.” – Komal Thakkar, AVP – Lead, Data Engineering, Avanse Financial Services

Best practices

Based on their experience, Avanse recommends:

Start with a workshop. Validate your specific use cases in a 72-hour technical validation before committing to full migration.
Migrate use cases, not code. Focus on what your analytics accomplish, then implement using cloud-native patterns rather than translating legacy scripts line by line.
Invest in governance early. Implement the data catalog and lineage tracking from day one.
Embrace project-based isolation. Organize around business functions for clear cost allocation and security boundaries.
Document business logic. Use migration as an opportunity to capture undocumented knowledge in the business glossary and dataset descriptions.

Conclusion

Avanse’s migration from an external analytics application to Amazon SageMaker Unified Studio consolidated their analytics stack into a single integrated environment on AWS. By querying data directly in Amazon S3 instead of copying it into the external application, they alleviated their biggest operational bottleneck. Project-based isolation replaced a shared server model, giving each business unit independent compute and clear cost visibility. And centralized governance through SageMaker Catalog and IAM Identity Center gave their compliance team the audit trails they had been missing.

The serverless, usage-based model means Avanse no longer pays for idle capacity. The lakehouse architecture supports new analytics patterns as they emerge, and native integration with AWS services, including generative AI through Amazon Bedrock, positions them to adopt new capabilities as their needs evolve.

Next steps

Start your analytics modernization journey by scheduling a 72-hour technical validation workshop. Contact your AWS account team to discuss your migration approach.

For more information, see:

Upgrade PySpark from Spark 3.5 to Spark 4.0 with AWS Spark Upgrade Agent

2026-06-09 Prasad Nadig

Post Syndicated from Prasad Nadig original https://aws.amazon.com/blogs/big-data/upgrade-pyspark-from-spark-3-5-to-spark-4-0-with-aws-spark-upgrade-agent/

Upgrading Apache Spark applications across major versions means tracking down breaking changes, manually debugging failures from log files, and running repeated test cycles. This process can stretch across weeks for complex code bases.

In this post, we walk through a hands-on PySpark migration from Spark 3.5 to Spark 4.0 on Amazon EMR Serverless, using the AWS Spark Upgrade Agent. You’ll see how the agent iteratively validates your application on a live Amazon EMR Serverless application, automatically diagnosing and resolving failures from Amazon CloudWatch logs until the job succeeds. By the end, you have a multi-pipeline PySpark application running on Spark 4.0 with four distinct breaking changes resolved. The fixes include configuration key removals, codec renames, and stricter charset validation, all driven through natural language interaction in the Integrated Development Environment (IDE).

This is part 2 of a three-part series on how the AWS Spark Upgrade Agent can automate and simplify Spark upgrades.

In Part 1, we introduced the agent’s architecture and capabilities. This post walks through a complete PySpark migration from Spark 3.5 to Spark 4.0 on Amazon EMR Serverless.

In the sections that follow, you will set up the prerequisites and infrastructure, explore the sample application, run the iterative validation workflow on EMR Serverless, review data quality results, and generate a comprehensive upgrade summary.

Note: Because this upgrade is performed using the AWS Spark Upgrade Agent Model Context Protocol (MCP) server, an agentic artificial intelligence (AI) system, the agent might take different paths to reach the same successful outcome. The workflow demonstrated here represents one successful upgrade path. The key takeaway is the end-to-end workflow: generating an upgrade plan, iteratively validating on Amazon EMR Serverless, and producing a comprehensive upgrade summary.

1. Prerequisites and setup

This section covers the tools, infrastructure, and IDE configuration you need before starting the upgrade. To follow along, you need an AWS account with an AWS Identity and Access Management (AWS IAM) user or role that has permissions to deploy AWS CloudFormation stacks, create AWS IAM roles and policies, and create Amazon EMR Serverless applications. Intermediate knowledge of AWS Command Line Interface (AWS CLI), AWS CloudFormation, and Python is helpful.

1.1 Install Kiro CLI and local tools

In this post, we use Kiro CLI to demonstrate the upgrade workflow. You can use an MCP-compatible IDE or framework. Examples include VS Code with Cline, Cursor, Windsurf, and Claude Desktop, among others. To follow along with Kiro CLI, install it on your workstation. For more details on the installation and setup, refer to Setup for Upgrade Agent:

curl -fsSL https://cli.kiro.dev/install | bash

Run the following command and use your builder ID to log in:

kiro-cli login --use-device-flow

With the Kiro CLI installed and logged in, rather than installing the remaining tools manually, use Kiro CLI to set up and verify your prerequisites with the following prompt:

kiro-cli chat
> Install AWS CLI, Python 3.10, and uv on my system if they are not already installed

Kiro CLI output showing successful installation of AWS CLI, Python, and uv

Output of AWS CLI and local tools install step.

These tools are needed for the upgrade workflow:

AWS CLI: Configured with a profile that has permissions to assume the AWS Identity and Access Management (AWS IAM) role created following.
Python 3.10+: Required to match the EMR 8.0 runtime.
uv package manager: Required by the MCP Proxy for AWS.

1.2 Infrastructure setup (AWS CloudFormation)

Two AWS CloudFormation stacks create the required resources: an AWS IAM role, an Amazon Simple Storage Service (Amazon S3) staging bucket, an Amazon EMR Serverless application (Spark 4.0.1), and its execution role.

Stack 1 – AWS IAM role and Amazon S3 staging bucket:

#	Region	Launch
1	US East (N. Virginia)	Launch Stack
2	US East (Ohio)	Launch Stack
3	US West (Oregon)	Launch Stack
4	Europe (Ireland)	Launch Stack

CloudFormation Outputs tab showing ExportCommand with SMUS_MCP_REGION, IAM_ROLE, and STAGING_BUCKET_PATH values

Outputs tab of the CloudFormation stack.

# Sets SMUS_MCP_REGION, IAM_ROLE, and STAGING_BUCKET_PATH
export SMUS_MCP_REGION=<YOUR-REGION> && export IAM_ROLE=arn:aws:iam::<YOUR-ACCOUNT-ID>:role/spark-upgrade-role-* && export STAGING_BUCKET_PATH=<YOUR-BUCKET>

Then configure the AWS CLI profile:

aws configure set profile.spark-upgrade-profile.role_arn ${IAM_ROLE}
aws configure set profile.spark-upgrade-profile.source_profile default
aws configure set profile.spark-upgrade-profile.region ${SMUS_MCP_REGION}

Stack 2 – Amazon EMR Serverless target application and execution role:

git clone https://github.com/aws-samples/sample-amazon-emr-spark4-examples
cd sample-amazon-emr-spark4-examples/pyspark/AWSSpark4AutoUpgradeDemo

The PySpark sample lives at resources/global_logistics_platform/. The AWS CloudFormation template lives at resources/cloudformation/.

Deploy the AWS CloudFormation template to create the source and target Amazon EMR Serverless applications and a shared execution role:

aws cloudformation deploy \
  --template-file resources/cloudformation/emr-serverless-target-setup.yaml \
  --stack-name spark-emr-serverless-upgrade \
  --region ${SMUS_MCP_REGION} \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
    StagingBucketName=${STAGING_BUCKET_PATH} \
    SourceReleaseLabel=emr-7.0.0 \
    TargetReleaseLabel=emr-spark-8.0-preview \
    SourceApplicationName=spark-upgrade-source \
    TargetApplicationName=spark-upgrade-target

This creates two Amazon EMR Serverless applications: a source (Spark 3.5.0) for data quality baseline and a target (Spark 4.0.1) for upgrade validation, with a shared execution role. Both applications auto-stop after 15 minutes of idle time, so there is no cost when not in use. To upgrade between different Spark versions, override SourceReleaseLabel and TargetReleaseLabel with your target Amazon EMR release labels.

After the stack completes deployment, note the outputs:

aws cloudformation describe-stacks \
  --stack-name spark-emr-serverless-upgrade \
  --region ${SMUS_MCP_REGION} \
  --query "Stacks[0].Outputs" --output table

This gives you the SourceApplicationId, TargetApplicationId, and ExecutionRoleArn needed for the upgrade prompt. Make a note of them.

1.3 IDE and MCP server configuration

Configure the spark-upgrade MCP server. For Kiro CLI:

kiro-cli-chat mcp add \
    --name "spark-upgrade" \
    --command "uvx" \
    --args '[
      "mcp-proxy-for-aws@latest",
      "https://sagemaker-unified-studio-mcp.'${SMUS_MCP_REGION}'.api.aws/spark-upgrade/mcp",
      "--service", "sagemaker-unified-studio-mcp",
      "--profile", "spark-upgrade-profile",
      "--region", "'${SMUS_MCP_REGION}'",
      "--read-timeout", "180"
    ]' \
    --timeout 180000 \
    --scope global

For other MCP clients, refer to your IDE’s MCP configuration documentation and use the same server parameters shown previously.

Verify the connection: Start Kiro CLI and confirm the spark-upgrade tools are loaded:

$ kiro-cli chat
...
spark-upgrade (MCP):
- generate_spark_upgrade_plan          * not trusted
- update_build_configuration           * not trusted
- fix_upgrade_failure                  * not trusted
- run_validation_job                   * not trusted
- check_job_status                     * not trusted
...

Tip: After Kiro CLI and the MCP server are configured, you can ask the agent to verify your setup. For example: “Check if I have AWS CLI, Python 3.10+, and uv installed, and confirm the spark-upgrade MCP server is connected.”

Kiro CLI output confirming spark-upgrade MCP server connection and tool availability

Output showing the status of each tool, AWS CLI, and MCP server.

Tip: Trust mode vs. confirm mode: When running the upgrade agent in Kiro CLI, you have two options:

Trust mode: Type t when prompted to approve a tool. The agent auto-approves subsequent uses of that tool without asking for confirmation. You can also use /tools trust-all to trust every tool at once for a fully autonomous experience.

Confirm mode: Type y for each individual tool invocation. This lets you review, verify, and approve every action before the agent runs it. If this is your first time using the agent, use confirm mode for full visibility.

2. Hands-on PySpark upgrade from Spark 3.5 to Spark 4.0

This section walks through the complete migration of a representative PySpark application from Amazon EMR Serverless 7.0.0 (Spark 3.5.0) to EMR Serverless with the emr-spark-8.0-preview release label (Spark 4.0.1), using the global_logistics_platform sample.

2.1 Sample project: global logistics platform

The sample application is a multi-domain PySpark data processing application with three pipelines:

Fleet management: Processes vehicle telemetry data (GPS tracking, fuel consumption, driver behavior scoring) using window functions, lag/lead operations, and statistical aggregations. Writes Parquet with lz4raw compression.
International shipping: Handles cross-border shipment documents with multi-language address standardization using character encoding functions (encode/decode with charsets like Shift_JIS, GB2312, EUC-KR), and processes carrier manifests with ISO-8859-1 encoding.
Historical compliance: Processes regulatory audit records spanning centuries (including pre-1582 Julian calendar dates), requiring legacy datetime rebasing for Parquet writes.

Project structure:

global_logistics_platform/
├── main.py                          # Orchestrator - runs all 3 pipelines
├── src/
│   ├── utils/
│   │   └── spark_config.py          # Spark session config & logging
│   └── domain/                      # Application code that needs migration
│       ├── fleet_management/
│       │   └── telemetry_processor.py
│       ├── international_shipping/
│       │   └── shipment_processor.py
│       └── historical_compliance/
│           └── compliance_processor.py
└── data/                             # Sample dataset for the workflow
    └── sample/
        ├── fleet_telemetry.csv
        ├── international_shipments.csv
        └── compliance_records.csv

2.2 The four Spark 4.0 incompatibilities

Before diving into the upgrade, here are the four specific breaking changes present in this code base that the agent discovers and resolves entirely through runtime validation:

#	Incompatibility	File(s)
1	Legacy Parquet configuration key removed: `spark.sql.legacy.parquet.datetimeRebaseModeInWrite` removed in Spark 4.0. Must use `spark.sql.parquet.datetimeRebaseModeInWrite`.	`spark_config.py`
2	Parquet compression codec rename: `lz4raw` codec renamed to `lz4_raw` in Spark 4.0.	`telemetry_processor.py`
3	Stricter charset encoding validation: Spark 4.0 tightened `encode()` behavior. Encoding CJK (Chinese, Japanese, Korean) characters to `ISO-8859-1` now throws `MALFORMED_CHARACTER_CODING`. In Spark 3.x this silently replaced unmappable chars with `?`. Restored via `spark.sql.legacy.codingErrorAction`.	`spark_config.py`
4	Character encoding restrictions: `encode()`/`decode()` in Spark 4.0 supports US-ASCII, ISO-8859-1, UTF-8, UTF-16BE, UTF-16LE, UTF-16, and UTF-32. Code uses `Shift_JIS`, `GB2312`, `EUC-KR`.	`shipment_processor.py`

The agent resolves each of these through iterative runtime validation on EMR Serverless: submitting the job, diagnosing failures from Amazon CloudWatch logs, applying fixes, and resubmitting until the job succeeds.

Architecture diagram showing the iterative validation workflow between the IDE, MCP server, and Amazon EMR Serverless

2.3 Step 1: Invoke the upgrade agent

Open the project in Kiro CLI and enter the following prompt:

Upgrade my Spark application in the current directory from EMR serverless version 7.0.0 to EMR serverless version 8.0.0.
Use Amazon EMR Serverless target app-id <YOUR-TARGET-APP-ID> and execution role
<YOUR-EXECUTION-ROLE-ARN> for validation.
Use source Amazon EMR Serverless app-id <YOUR-SOURCE-APP-ID> for data quality baseline.
Store artifacts at s3://${STAGING_BUCKET_PATH}/spark4-upgrade/python/
Enable data quality validation

Tip: The SourceApplicationId, TargetApplicationId, and ExecutionRoleArn are in the Outputs of the spark-emr-serverless-upgrade AWS CloudFormation stack you deployed in Section 1.2.

The agent invokes generate_spark_upgrade_plan, scans the project structure, identifies the Spark version mapping (EMR 7.0.0 → Spark 3.5.0, EMR 8.0.0 → Spark 4.0.1), and produces a structured upgrade plan with an Analysis ID for traceability.

The agent presents the plan and asks for confirmation. Type y to approve the tool invocation, or t to trust that tool for the rest of the session.

You have an option to save the plan as a local JSON file for future reference or to resume the upgrade at a later point, so go ahead and ask Kiro to save it locally. Provide the AWS CLI profile that you have configured on your system. Use the following prompt to provide these inputs:

Yes I would like to save the plan to a local file and use spark-upgrade-profile

2.4 Step 2: Build and package

The agent validates the Python project compiles successfully, then packages it for Amazon EMR Serverless deployment:

Runs py_compile on each .py file to verify syntax.
Creates src.zip containing the src/ directory (preserving the import structure used by from src.utils import ...).
Uploads src.zip, main.py, and sample input data to the Amazon S3 staging path.

# What the agent does behind the scenes:
zip -r src.zip src/
aws s3 cp main.py s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/source/main.py
aws s3 cp src.zip s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/source/src.zip
aws s3 cp data/sample/ s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/input/ --recursive

No external dependencies (no requirements.txt), so no virtual environment is needed. If your project has external dependencies in a requirements.txt, the agent will package them into a virtual environment archive and include it in the EMR Serverless submission parameters.

2.5 Step 3: Data quality baseline on source application

Before migrating the code, the agent establishes a data quality baseline by running the original (pre-upgrade) code on the source Amazon EMR Serverless application (Spark 3.5.0 / EMR 7.0.0). This captures the expected output that the upgraded application must match.

The agent submits the job to the source application with data quality check enabled:

{
  "executionRoleArn": "arn:aws:iam::<YOUR-ACCOUNT-ID>:role/<YOUR-EXECUTION-ROLE>",
  "jobDriver": {
    "sparkSubmit": {
      "entryPoint": "s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/source/main.py",
      "entryPointArguments": [
        "s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/input/",
        "s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/output/source/"
      ],
      "sparkSubmitParameters": "--py-files s3://<YOUR-BUCKET>/spark4-upgrade/python/<ANALYSIS-ID>/source/src.zip"
    }
  },
  "configurationOverrides": {
    "monitoringConfiguration": {
      "cloudWatchLoggingConfiguration": {
        "enabled": true,
        "logGroupName": "/aws/emr-serverless"
      }
    }
  }
}

The agent monitors the source run via check_job_status until it completes successfully. This baseline output is stored for comparison after the target validation succeeds.

2.6 Step 4: Iterative runtime validation on target application

This is the core of the upgrade. The agent submits the unmodified application to the target Amazon EMR Serverless application (Spark 4.0.1), and every incompatibility is discovered, diagnosed, and fixed through runtime failures. The agent drives the entire fix cycle by submitting to EMR, reading errors from Amazon CloudWatch logs, applying fixes, rebuilding, and resubmitting.

The agent presents the proposed Amazon EMR Serverless job configuration for your review before each submission. Type y to approve.

2.6.1 Fix 1: Legacy Parquet configuration key removed (iteration 1)

The first submission fails immediately at SparkSession initialization:

org.apache.spark.sql.AnalysisException:
The SQL config 'spark.sql.legacy.parquet.datetimeRebaseModeInWrite' was removed
in the version 4.0.0. Use 'spark.sql.parquet.datetimeRebaseModeInWrite' instead.

The Historical Compliance pipeline configures spark.sql.legacy.parquet.datetimeRebaseModeInWrite for handling pre-1582 Julian calendar dates. Spark 4.0 removed the legacy. prefix from this configuration key.

The agent calls fix_upgrade_failure, which identifies the migration rule and recommends the fix:

File: src/utils/spark_config.py

# Before
.config("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "LEGACY")

# After
.config("spark.sql.parquet.datetimeRebaseModeInWrite", "LEGACY")

After applying the fix, the agent rebuilds src.zip, re-uploads to Amazon S3, and resubmits the job.

2.6.2 Fix 2: Parquet compression codec rename (iteration 2)

The resubmitted job fails with a new error, which confirms progress:

pyspark.errors.exceptions.captured.IllegalArgumentException:
[CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION]
The codec lz4raw is not available.
Available codecs are brotli, uncompressed, lzo, snappy, lz4_raw, none, zstd, lz4, gzip.
SQLSTATE: 56038

The Fleet Management pipeline’s telemetry_processor.py uses lz4raw as the Parquet compression codec. Spark 4.0 renamed this to lz4_raw (with an underscore).

The recommended fix:

File: src/domain/fleet_management/telemetry_processor.py

# Before
.option("compression", "lz4raw")

# After
.option("compression", "lz4_raw")

The agent applies the change, rebuilds, and resubmits.

2.6.3 Fix 3: Stricter charset encoding validation (iteration 3)

The next submission surfaces a different failure:

org.apache.spark.SparkRuntimeException:
[MALFORMED_CHARACTER_CODING]
Invalid value found when performing `encode` with ISO-8859-1
SQLSTATE: 22000

The International Shipping pipeline’s process_carrier_manifests() method uses encode(..., 'ISO-8859-1') on data containing CJK (Chinese, Japanese, Korean) characters. Although ISO-8859-1 is in Spark 4.0’s supported charset list, it is a single-byte encoding that cannot represent CJK characters. In Spark 3.x, the Java charset encoder silently replaced unmappable characters with ?. Spark 4.0 tightened this behavior to throw MALFORMED_CHARACTER_CODING for unmappable characters.

The agent identifies the migration rule and adds a legacy compatibility configuration:

File: src/utils/spark_config.py

# Added to SparkSession builder
.config("spark.sql.legacy.codingErrorAction", "true")

This restores the Spark 3.x behavior where unmappable characters are silently replaced instead of throwing errors.

With the configuration added, the agent rebuilds and resubmits.

2.6.4 Fix 4: Character encoding restrictions (iteration 4)

The fourth submission fails with yet another encoding error:

org.apache.spark.SparkIllegalArgumentException:
[INVALID_PARAMETER_VALUE.CHARSET]
The value of parameter(s) `charset` in `encode` is invalid:
expects one of the iso-8859-1, us-ascii, utf-16, utf-16be, utf-16le, utf-32, utf-8,
but got Shift_JIS. SQLSTATE: 22023

The International Shipping pipeline’s standardize_addresses_with_charset() method uses Shift_JIS, GB2312, and EUC-KR charsets in encode()/decode() calls. Spark 4.0 restricts these functions to seven standard charsets. These regional charsets are not in the supported list.

The agent replaces the unsupported charsets with UTF-8:

File: src/domain/international_shipping/shipment_processor.py

Before (Spark 3.5.0):

df = df.withColumn(
    "shipper_address_normalized",
    when(col("origin_country") == "JP",
         expr("decode(encode(shipper_address, 'Shift_JIS'), 'UTF-8')"))
    .when(col("origin_country") == "CN",
         expr("decode(encode(shipper_address, 'GB2312'), 'UTF-8')"))
    .when(col("origin_country") == "KR",
         expr("decode(encode(shipper_address, 'EUC-KR'), 'UTF-8')"))
    .otherwise(col("shipper_address"))
)

After (Spark 4.0.1):

df = df.withColumn(
    "shipper_address_normalized",
    when(col("origin_country") == "JP",
         expr("decode(encode(shipper_address, 'UTF-8'), 'UTF-8')"))
    .when(col("origin_country") == "CN",
         expr("decode(encode(shipper_address, 'UTF-8'), 'UTF-8')"))
    .when(col("origin_country") == "KR",
         expr("decode(encode(shipper_address, 'UTF-8'), 'UTF-8')"))
    .otherwise(col("shipper_address"))
)

The same transformation is applied to consignee_address_normalized.

The agent rebuilds and resubmits one final time.

2.6.5 Final submission: success

The fifth submission completes successfully:

{"success": true, "message": "EMR SERVERLESS job completed successfully",
"compute_run_id": "<JOB-RUN-ID>", "status": "SUCCESS",
"application_type": "EMR-Serverless"}

The three pipelines (Fleet Management, International Shipping, and Historical Compliance) complete on EMR Serverless with the emr-spark-8.0-preview release label (Spark 4.0.1).

2.7 Summary of the iterative runtime validation

The runtime validation loop is the core value of the upgrade agent. Here’s the complete iteration history:

Table showing the four validation iterations with error types and fixes applied

Each iteration follows the same cycle:

Diagram showing the submit, diagnose, fix, rebuild, and resubmit cycle

Failures that would normally require manual log analysis, root cause investigation, and code patching are resolved automatically by the agent in this workflow.

3. Data quality validation

With both the source baseline (Section 2.5) and the upgraded target run (Section 2.6) completed successfully, the agent performs data quality validation to verify the migration hasn’t changed your application’s output. This is the key advantage of including the source application in your upgrade prompt: the agent can compare outputs from both Spark versions side by side.

3.1 Data quality comparison

The agent invokes get_data_quality_summary to compare the outputs across four dimensions:

Schema validation: Confirms column names, data types, and column ordering match between source and target outputs.
Row count validation: Verifies no data loss or duplication during migration.
Nullability validation: Detects changes in null handling.
Statistical summary validation: Compares numeric and string column distributions (min, max, mean, count, distinct values).

The agent presents the comparison results:

Data quality summary showing schema, row count, and nullability checks passing with a statistical mismatch in shipper_address

The preceding image shows the data quality summary.

Three of four checks pass cleanly. The statistical summary validation detects a mismatch in the shipper_address column of the customs_declarations output: the max and min summary values differ between source and target.

3.2 Understanding and resolving the mismatch

This mismatch is a direct consequence of Fix 4 (Section 2.6.4). The original code ran addresses through a Shift_JIS/GB2312/EUC-KR → UTF-8 roundtrip that produced garbled text, because the intermediate regional charset corrupted multi-byte UTF-8 characters. The upgraded code uses UTF-8 → UTF-8, preserving addresses faithfully. The mismatch reflects improved data quality, not a regression.

Schema, row counts, and nullability matched exactly: the difference is limited to string values that were previously garbled. No further action is needed. The upgraded application is production-ready.

Expected behavior: Character encoding migrations might change string values, although they preserve semantic meaning. When data quality validation reports mismatches, trace each one back to a specific code change. If the mismatch is explained by a required migration fix (as here), verify the new behavior is correct and document it. If a mismatch cannot be explained, investigate before promoting to production.

4. Upgrade summary

After the agent completes the entire upgrade workflow, it produces a comprehensive upgrade summary following a structured template. This summary lets you review the job configuration updates, code modifications with diffs and file references, relevant migration rules applied, and data quality validation status.

Here is the summary the agent produced for this upgrade:

Upgrade plan

Compile and build project with current Spark 3.5.0: validated that Python files compile successfully.
Run baseline validation on source EMR Serverless (00g4vhvt1lhtrs09) with Spark 3.5.0: established data quality baseline.
Run target validation on target EMR Serverless (00g4vhvt3np1bj09) with Spark 4.0.1: fixed 4 issues iteratively across 4 validation attempts.
Compare data quality between source and target runs: detected expected mismatch in shipper_address.
Generate and persist upgrade summary.

Upgrade result

Upgrade completed with data validation enabled. Data validation detected an expected mismatch in the shipper_address column because of the charset encoding migration from unsupported charsets (Shift_JIS, GB2312, EUC-KR) to UTF-8.

Dependency changes

No external dependencies were changed in this project (no requirements.txt).

Job configuration changes

Parquet datetime rebase configuration key renamed.
- Change: spark.sql.legacy.parquet.datetimeRebaseModeInWrite → spark.sql.parquet.datetimeRebaseModeInWrite.
- Migration rule: In Spark 4.0, the legacy datetime rebasing SQL configurations with the prefix spark.sql.legacy are removed. The SQL configuration spark.sql.legacy.parquet.datetimeRebaseModeInWrite was removed in the version 4.0.0. Use spark.sql.parquet.datetimeRebaseModeInWrite instead.
Legacy coding error action enabled.
- Change: Added spark.sql.legacy.codingErrorAction set to true.
- Migration rule: In Spark 4.0, the encode() and decode() functions raise MALFORMED_CHARACTER_CODING error when handling unmappable characters. In Spark 3.5 and earlier versions, these characters are replaced with garbled text. To restore the previous behavior, set spark.sql.legacy.codingErrorAction to true.

Code changes

Validation attempt 1: Legacy Parquet configuration key.
- Validation run: EMR-Serverless job_run_id 00g4vm14v118vg0b.
- Error: The SQL config 'spark.sql.legacy.parquet.datetimeRebaseModeInWrite' was removed in the version 4.0.0.
- Applied changes: src/utils/spark_config.py: Changed .config("spark.sql.legacy.parquet.datetimeRebaseModeInWrite", "LEGACY") to .config("spark.sql.parquet.datetimeRebaseModeInWrite", "LEGACY").
Validation attempt 2: Parquet compression codec.
- Validation run: EMR-Serverless job_run_id 00g4vm5pm1hig00b.
- Error: [CODEC_NOT_AVAILABLE.WITH_AVAILABLE_CODECS_SUGGESTION] The codec lz4raw is not available.
- Applied changes: src/domain/fleet_management/telemetry_processor.py: Changed .option("compression", "lz4raw") to .option("compression", "lz4_raw").
Validation attempt 3: Stricter charset encoding.
- Validation run: EMR-Serverless job_run_id 00g4vm8sh4sp0g0b.
- Error: [MALFORMED_CHARACTER_CODING] Invalid value found when performing encode with ISO-8859-1.
- Applied changes: src/utils/spark_config.py: Added .config("spark.sql.legacy.codingErrorAction", "true") to the SparkSession builder.
Validation attempt 4: Unsupported charsets.
- Validation run: EMR-Serverless job_run_id 00g4vmc668ng6o0b.
- Error: [INVALID_PARAMETER_VALUE.CHARSET] charset in encode is invalid: expects one of iso-8859-1, us-ascii, utf-16, utf-16be, utf-16le, utf-32, utf-8, but got Shift_JIS.
- Applied changes: src/domain/international_shipping/shipment_processor.py: Replaced Shift_JIS, GB2312, EUC-KR with UTF-8 for shipper and consignee address encoding.

Data validation result

#	Validation	Status
1	Schema validation (column names, types, ordering)	Passed (no difference)
2	Row count validation (no data loss)	Passed (no difference)
3	Nullability validation (null handling changes)	Passed (no difference)
4	Statistical summary validation (numeric/string distributions)	Failed (with difference)

Data mismatch: 1. The shipper_address column max summary value changed in customs_declarations output. This is expected because of the charset encoding migration from Shift_JIS/GB2312/EUC-KR to UTF-8. 2. The shipper_address column min summary value changed in customs_declarations output for the same expected cause.

5. Conclusion

The AWS Spark Upgrade Agent turns a traditionally time-consuming PySpark migration into an automated, iterative workflow. For the Global Logistics Platform sample, the agent identified and resolved four distinct Spark 4.0 breaking changes: legacy Parquet configuration key removal, compression codec renames, stricter charset encoding validation, and character encoding restrictions. Each fix was applied across three domain processors, through natural language interaction in the IDE.

Every incompatibility was discovered through runtime validation on Amazon EMR Serverless. The agent submitted the unmodified application to the target application, and each failure revealed the next breaking change:

The spark.sql.legacy.parquet.datetimeRebaseModeInWrite configuration removal, which crashes SparkSession initialization.
The lz4raw → lz4_raw codec rename, which fails when Parquet writes run.
ISO-8859-1 encoding of CJK characters: ISO-8859-1 is a valid Spark 4.0 charset, so the failure surfaces only when the code runs against real multi-language data, because Spark 4.0 tightened charset encoding validation to reject unmappable characters.
Shift_JIS/GB2312/EUC-KR charsets removed from Spark 4.0’s supported charset list entirely.

The agent diagnosed each error from Amazon CloudWatch logs, applied the fix, rebuilt, and resubmitted without manual intervention beyond approving each step. The data quality validation then confirmed that the upgraded application produces equivalent output on Spark 4.0.1: schema, row counts, and nullability matched exactly. The one difference, in the shipper_address column, resulted from the charset migration from regional encodings to UTF-8, which actually improved data quality by eliminating garbled text from incorrect encoding roundtrips. With each mismatch traced back to a specific, understood code change, the upgraded application is production-ready.

#	Category	Spark 3.x behavior	Spark 4.0 change	Agent fix
1	Parquet datetime configuration	`spark.sql.legacy.parquet.datetimeRebaseModeInWrite`	`legacy.` prefix removed from key name	Update configuration key
2	Parquet compression	`lz4raw` codec name	Renamed to `lz4_raw` (with underscore)	Update codec name
3	Charset + CJK data	`ISO-8859-1` silently replaced unmappable CJK chars with `?`	Stricter charset validation throws `MALFORMED_CHARACTER_CODING` for unmappable characters	Add `spark.sql.legacy.codingErrorAction=true`
4	Character encoding	`encode()`/`decode()` supported Java charsets	Restricted to 7 standard charsets	Replace unsupported charsets with `UTF-8`

Next steps after your first upgrade:

Apply the agent to your production PySpark code base.
Integrate the upgrade workflow into your CI/CD pipeline.
Explore Scala application upgrades (see Part 3 of this series).

To get started with your own PySpark migration:

Deploy the AWS CloudFormation templates from Section 1.2 for one-time AWS IAM, Amazon S3, and Amazon EMR Serverless setup.
Configure the spark-upgrade MCP server in your MCP-compatible IDE.
Point the agent at your PySpark project and let it handle the rest.

For more information, see the Amazon EMR Serverless documentation, the Apache Spark 4.0 migration guide, and the AWS Spark Upgrade Agent setup guide.

6. Clean up resources

To avoid ongoing costs, delete the resources you created:

Delete the Amazon EMR Serverless stack:

aws cloudformation delete-stack --stack-name spark-emr-serverless-upgrade --region ${SMUS_MCP_REGION}

Delete the AWS IAM and Amazon S3 staging stack:

aws cloudformation delete-stack --stack-name spark-upgrade-mcp-setup --region ${SMUS_MCP_REGION}

If the Amazon S3 staging bucket contains objects, empty it before deleting the stack:
```
aws s3 rm s3://${STAGING_BUCKET_PATH} --recursive
```

About the authors

Announcing Spark Connect on Amazon EMR Serverless: Interactive PySpark development, anywhere

2026-06-09 Al MS

Post Syndicated from Al MS original https://aws.amazon.com/blogs/big-data/announcing-spark-connect-on-amazon-emr-serverless-interactive-pyspark-development-anywhere/

Today, AWS is announcing support for Spark Connect on Amazon EMR Serverless with EMR release 7.13 (Apache Spark 3.5.6) and later versions. You can now build and debug Spark applications from your preferred local environment while running full-scale Spark operations on EMR Serverless.

Previously, code that worked on a local machine might break in production because of environment mismatches, dependency conflicts, or unexpected data patterns. The only way to catch it was a deploy-and-check cycle. With the Spark Connect feature, you can develop Spark code from a supported local environment, such as an IDE (for example, VS Code or PyCharm), Jupyter notebooks, Amazon SageMaker Unified Studio (SMUS) Data Notebooks, Amazon Q Developer, or Kiro. There are no clusters to provision, no code to repackage, and no deploy-and-check loop. Your local Python session can stay local as usual while Spark operations are automatically routed to a remote Spark server for execution.

Each Spark Connect session has its own AWS resource with a unique ARN, enabling per‑session AWS Identity and Access Management (AWS IAM) permissions, tag‑based cost allocation, audit through AWS CloudTrail, and session-specific configuration overrides. This gives teams finer control over who runs what, at what cost. You also get real-time visibility through the Spark UI, persistent session history, and a dedicated interface to monitor and manage active and completed sessions.

For more details, visit the EMR Serverless release notes or the EMR Serverless Developer Guide. For a quick look at the experience, here’s a demonstration of using Spark Connect in Amazon SageMaker Unified Studio Data Notebooks:

For a runnable end-to-end example, try the EMR Serverless Spark Connect sample notebook from your local IDE. See the following demonstration:

How Spark Connect works

Spark Connect uses a client-server architecture that separates application code from the Spark engine. The client, a lightweight PySpark library running on a local environment, sends Spark operations over a secure gRPC/TLS connection to a Spark Connect server running on EMR Serverless. Then the server runs that Spark code on EMR Serverless as compute. Finally, it returns results to your local session.

Spark Connect client-server architecture showing a local IDE connecting to a Spark Connect server on EMR Serverless

Your local machine doesn’t need Spark installed, doesn’t need direct access to the data, and doesn’t need to be sized for the workload. Because the client is a compact library, you can embed Spark operations in your Python applications that support PySpark. This includes web services, dashboards, and automation scripts. For example, a development team can add Spark-powered analytics directly into a FastAPI backend or a Streamlit dashboard, treating Spark like a database driver rather than a separate batch system. These capabilities extend Spark Connect use cases beyond traditional notebook and IDE development, since the compute-intensive processing happens on the server – EMR Serverless side. This allows you to use pandas, matplotlib, and your team’s internal Python libraries on your laptop or in your embedded clients, without installing those libraries on EMR Serverless.

With Spark Connect server sessions running on EMR Serverless, you pay for compute only while your session is active. When inactive, you’re not paying. EMR Serverless automatically scales compute up and down based on workload demands through dynamic resource allocation (DRA), eliminating the need to predict capacity ahead of time. For teams that run Spark Connect sessions regularly, you can configure pre-initialized capacity on your EMR Serverless application for faster session startup times. Additionally, your Spark Connect sessions have access to the full suite of EMR Serverless features, including AWS Graviton processors for cost optimization and secure VPC connectivity to your data sources. You also get access to custom images with flexibility and integrated observability through Amazon CloudWatch and the Spark UI.

Getting started

Getting started with Spark Connect on EMR Serverless takes three steps: create an application, start a session, and connect from your IDE.

Note: The resources created in this quick start incur charges while active. Make sure to follow the cleanup steps at the end of this tutorial to avoid ongoing charges.

Prerequisites

In addition to the required job runtime IAM role, these additional permissions are needed: emr-serverless:StartSession, GetSession, GetSessionEndpoint, TerminateSession, GetResourceDashboard, and iam:PassRole on the runtime role.
An existing EMR Serverless application running emr-7.13.0 or later, with interactiveConfiguration.sessionEnabled = true.
boto3 version 1.43.0 or later to access the latest EMR Serverless session APIs.

Step 1: Create an EMR Serverless application with Spark Connect enabled

Amazon EMR Serverless application creation page in the EMR console

Open the Amazon EMR console and navigate to EMR Serverless.
Choose Get started. A pop-up appears. Choose Create and launch EMR Studio.
This takes you to the Create application page.
Enter a Name for your application (for example, spark-connect-app).
For Type, select Spark.
For Release version, select emr-7.13.0 or later.
For Architecture, choose x86_64 (default). This is compatible with most third-party tools and libraries.
Under Application setup options, select Use default settings for interactive workloads. This automatically sets interactiveConfiguration.sessionEnabled = true.
Choose Create and start application.

Alternatively, using the CLI command:

# Create an application with Spark Connect enabled
APP_ID=$(aws emr-serverless create-application \
  --type "SPARK" \
  --name "spark-connect-app" \
  --release-label emr-7.13.0 \
  --interactive-configuration '{"sessionEnabled": true}' \
  --query 'applicationId' \
  --output text)
echo "Created application: $APP_ID"
# Start the application
aws emr-serverless start-application --application-id "$APP_ID"

Step 2: Start a session

Next, start a session and obtain the Spark Connect endpoint.

Provide an IAM execution role that grants the session access to your data, such as reading data from an Amazon S3 bucket or querying the AWS Glue Data Catalog. This is the same type of role used for EMR Serverless batch jobs.

# Start a session with your execution role
$ROLE_ARN="YOUR_ROLE" # example: arn:aws:iam::123456789012:role/EMRServerlessSessionRole
SESSION_ID=$(aws emr-serverless start-session \
  --application-id $APP_ID \
  --execution-role-arn $ROLE_ARN \
  --query sessionId \
  --output text)

# Get the session endpoint
aws emr-serverless get-session-endpoint \
  --application-id $APP_ID \
  --session-id $SESSION_ID

The get-session-endpoint response includes a secure endpoint URL and an authentication token. All communication between your local environment and EMR Serverless is encrypted using TLS. Treat the token as a sensitive credential. Consider using AWS Secrets Manager to store and retrieve tokens programmatically. The authentication token is time-limited to 1 hour, so for long-running sessions we recommend that you refresh it periodically.

Step 3: Connect from your local IDE

Use the returned endpoint URL and authentication token to connect to the Spark Connect server.

The connection URL uses the sc:// protocol, which is the Spark Connect standard. The use_ssl=true parameter supports encrypted communications over TLS, so your data and credentials are protected in transit.

from pyspark.sql import SparkSession

# Use the endpoint and auth token from get-session-endpoint
session_endpoint="<endpoint-from-get-session-endpoint>"
auth_token="<authToken-from-get-session-endpoint>"

spark_connect_url = (
    f"sc://{session_endpoint}:443/;use_ssl=true;x-aws-proxy-auth={auth_token}"
)

spark = SparkSession.builder \
    .remote(spark_connect_url) \
    .getOrCreate()

# Query data in your S3 data lake
df = spark.sql("SELECT * FROM my_catalog.my_database.my_table")
df.show()

# Run transformations at scale
df.groupBy("category").count().orderBy("count", ascending=False).show()
spark.stop()

Once connected, Spark operations you write in your IDE can be run on EMR Serverless. For debugging, you can pause the execution at breakpoints, inspect variables, and step through your transformations locally while EMR Serverless processes your data on remote, scalable infrastructure.

Sessions remain active for a configurable idle timeout (1 hour by default). If your connection drops, the session continues running, allowing you to reconnect without losing your work. You can also access the live Spark UI through the GetResourceDashboard API to monitor queries, stages, and executors in real time. After the session ends, the Spark History Server remains available for post-run analysis.

Clean up resources

If the 1-hour session idle timeout does not meet your needs, you can manually remove sessions to avoid ongoing costs. Note that terminating an active session will immediately stop you running Spark operations. Before doing that, verify all your critical data processing is completed, and results are saved.

# 1. Stop the active session
aws emr-serverless terminate-session \
  --application-id $APP_ID \
  --session-id $SESSION_ID

# 2. Stop the application
aws emr-serverless stop-application --application-id $APP_ID

Use cases

Spark Connect on EMR Serverless supports a wide range of development workflows. The following are some of the most popular use cases, including but not limited to:

Interactive ETL development — Build and test data pipelines interactively, validating transformations against full-scale datasets before promoting them to production as batch jobs.
SageMaker Unified Studio (SMUS) Data Notebooks — Run interactive PySpark sessions directly from SMUS Data Notebooks connected to EMR Serverless through Spark Connect.
Direct S3 and JDBC access without a catalog — Connect directly to S3 files and JDBC data sources without needing a metastore or catalog configuration.
Apache Iceberg Data Lakehouse analytics — Query and manage Iceberg tables through the AWS Glue Data Catalog, with full support for time travel, schema evolution, and partition management.
Amazon S3 Tables with federated catalog — Access S3 Tables as a federated Glue Data Catalog source, combining Iceberg features with serverless Spark execution.
dbt-spark — Run dbt-spark adapter against EMR Serverless via Spark Connect, allowing analytics engineers to develop and test transformations locally with dbt framework while using EMR Serverless as the remote Spark engine.
Exploratory data analysis and feature engineering — Analyze production-scale data from your preferred notebook environment instead of using sampled subsets, helping you catch data quality issues earlier.
Compute standardization — Standardize EMR Serverless as the Spark backend while giving you the flexibility to use preferred local tools, version control, and CI/CD workflows.

These use cases work across multiple client surfaces: IDEs, Jupyter notebooks, dbt-spark, and AI coding agents. Because Spark Connect is an open Apache Spark standard, the same PySpark code typically works across different Spark backends by changing the connection endpoint.

Availability and pricing

Spark Connect on EMR Serverless is now available with Apache Spark 3.5.6 on Amazon EMR release 7.13 and higher in all AWS Regions where EMR Serverless is available. There is no additional charge for using Spark Connect. You pay for the EMR Serverless compute resources (vCPU, memory, and storage) consumed during your session, the same pricing model as EMR Serverless batch jobs.

Conclusion

Spark Connect on EMR Serverless bridges the gap between local development and production-scale execution. Build and debug PySpark applications from your preferred environment (IDE, notebook, dbt, or AI coding agent) while EMR Serverless handles automatic scaling, per-session cost visibility, and infrastructure management behind the scenes. With ARN-addressable sessions, fine-grained IAM permissions, tag-based cost allocation, and per-session configuration overrides, your team gets the controls they need without sacrificing flexibility.

Get started today with EMR release 7.13.0 (Spark 3.5.6). Follow the step-by-step tutorial in the EMR Serverless Developer Guide to create your first Spark Connect session and experience interactive, serverless PySpark development firsthand.

About the authors

Build stateful streaming applications with Apache Spark 4.0 on Amazon EMR Serverless

2026-06-09 Raj Ramasubbu

Post Syndicated from Raj Ramasubbu original https://aws.amazon.com/blogs/big-data/build-stateful-streaming-applications-with-apache-spark-4-0-on-amazon-emr-serverless/

Apache Spark 4.0 represents a major milestone in stream processing, introducing new capabilities that fundamentally change how developers build stateful streaming applications. At the heart of these improvements is the transformWithState API – a new capability that enables first-class support for timers, automatic state management, and schema evolution to Spark Structured Streaming.

With Spark 4.0 now available on Amazon EMR Serverless, developers can build stateful streaming applications using the transformWithState API in a fully managed, serverless environment that automatically scales based on workload demands. This combination delivers the power of sophisticated stream processing without the operational overhead of cluster management.

In this post, we demonstrate how to build a production-ready IoT device monitoring system using Spark 4.0’s transformWithState API on Amazon EMR Serverless. This example showcases the key capabilities of stateful streaming and provides a template you can adapt for your own use cases.

Apache Spark 4.0: introducing transformWithState

Apache Spark 4.0’s latest streaming features solve common production challenges in stateful applications by introducing native timer support and advance state management capabilities for complex event processing workflows. The new transformWithState API provides:

Key features of transformWithState

Native timer support: Register timers that fire callbacks at specific times for use cases like heartbeat monitoring, session timeout detection, and SLA violation alerts.
Automatic state TTL (Time-To-Live): Configure automatic expiration policies to prevent state from growing indefinitely. This is useful for use cases like session state size control, clearing stale device telemetry, maintaining a recency cache, or tracking invalid logins within the last hour for fraud detection.
Schema evolution: Evolve state schema without restarting from a new checkpoint. Add optional fields, remove fields, or widen numeric types. This is particularly valuable for use cases where data structures are dynamic, and application downtime for schema migration is not acceptable, enabling more resilient and flexible real-time streaming applications.
Multiple state variables: Support for multiple independent state variables (ValueState, ListState, MapState) per key, well-suited for building complex, real-time applications that require sophisticated state management, such as storing a history of recent error codes, tracking counts of various alert types, or maintaining multiple dimensions of user activity within a single stateful operator.
State observability: Query application state mid-stream using the State Data Source Reader for debugging and monitoring. This is especially valuable in applications that require maintaining and evolving state through several steps, such as detection of sophisticated event patterns across multiple streams and over time, where visibility into state transitions is critical for troubleshooting and validation.
Operator chaining: Chain multiple stateful operators together for complex multi-stage processing pipelines.

These capabilities make Spark 4.0 ideal for applications that were previously difficult or impossible to implement efficiently, such as complex event processing, session analytics, anomaly detection, and real-time monitoring systems.

Use case: IoT heartbeat monitoring

Consider a fleet of 100,000 IoT sensors deployed across manufacturing facilities. Each sensor sends a heartbeat signal every 20 seconds to indicate it’s operational. Your operations team needs to be alerted within 30 seconds if any sensor goes offline, with repeat alerts every 60 seconds until the sensor comes back online.

This seemingly simple requirement presents several technical challenges. The application must maintain the last heartbeat timestamp for each of the 100,000 devices while independently managing timers to detect missed signals per device. It also needs to handle out-of-order heartbeats caused by network delays and clean up state for decommissioned devices to prevent unbounded memory growth. All of this must happen at scale, processing millions of events per minute with low latency, while recovering gracefully from failures without losing state.

To address the specific challenges of IoT heartbeat monitoring described above, we present a solution built on the transformWithState API in Spark 4.0. With its native timer support, automatic state management, and built-in fault tolerance, making it the ideal solution for IoT heartbeat monitoring at scale.

Solution overview

Our solution architecture follows a serverless, event-driven design:

Solution architecture showing IoT devices sending heartbeats to Kinesis Data Streams, processed by EMR Serverless with transformWithState, checkpointed to Amazon S3, and alerts delivered via Amazon SNS

IoT devices send heartbeat events to Amazon Kinesis Data Streams containing device ID, timestamp, and metadata (battery level, signal strength, firmware version).
Amazon EMR Serverless reads from Kinesis using the Spark aws-kinesis connector using VPC Endpoint for Kinesis, then parses JSON events into structured DataFrames and grouping by device_id.
transformWithState processes each device’s stream. On heartbeat arrival, it updates state and registers a 30-second timer; when the timer expires without a new heartbeat, it emits an offline alert.
State is automatically persisted to RocksDB locally and checkpointed to Amazon Simple Storage Service (Amazon S3), enabling fault-tolerant recovery and exactly-once processing semantics.
Alerts are delivered via Amazon Simple Notification Service (Amazon SNS) to configured subscribers (email, SMS, AWS Lambda, webhooks).

Prerequisites

Before implementing this solution, verify that you have:

AWS account: With permissions for EMR Serverless, Kinesis, SNS, S3, VPC, and IAM.
AWS Command Line Interface (AWS CLI): Configured with appropriate credentials.
VPC setup: VPC with private subnets and security groups configured.
Kinesis VPC interface endpoint: VPC endpoint for private connectivity to Kinesis.
Kinesis Data Stream: Created for ingesting heartbeat events (for example, iot-heartbeats). For testing your streaming data solution, refer to Test your streaming data solution with the new Amazon Kinesis Data Generator.
SNS topic: Created for sending alerts (for example, iot-alerts).
S3 bucket: For storing application code, dependencies, and checkpoints.

Step-by-step implementation

The following steps walk you through setting up an EMR Serverless application with Spark 4.0, configuring the stateful streaming processor, and deploying the IoT heartbeat monitoring solution.

Step 1: Create the EMR serverless application

Run the following command in your terminal using the AWS CLI. Replace the subnet and security group IDs with the values from your VPC setup.

# Create EMR Serverless application with Spark 4.0 and VPC
aws emr-serverless create-application \
  --name "iot-heartbeat-monitor" \
  --release-label "emr-spark-8.0.0" \
  --type "SPARK" \
  --network-configuration '{
    "subnetIds": ["subnet-xxxxx", "subnet-yyyyy"],
    "securityGroupIds": ["sg-zzzzz"]
  }' \
  --region us-east-1

The command returns a JSON response containing the application details. Note the applicationId value from the output, as you will need it in subsequent steps.

Step 2: Implement the heartbeat monitor

The core of our solution is the HeartbeatMonitor class that extends StatefulProcessor. This class demonstrates the key features of Spark 4.0’s transformWithState API. Download the full implementation script and upload it to your local S3 bucket for execution. Let’s walk through each component to understand how it works.

2.1 Initialize state variables

The init() method is called once when the processor is initialized. This is where we define and register our state variables.

from pyspark.sql.streaming.stateful_processor import (
    StatefulProcessor, StatefulProcessorHandle
)

class HeartbeatMonitor(StatefulProcessor):

    def init(self, handle: StatefulProcessorHandle) -> None:
        self.handle = handle

        # Define state schemas
        last_seen_schema = StructType([
            StructField("timestamp", TimestampType(), True)
        ])

        device_info_schema = StructType([
            StructField("battery_level", StringType(), True),
            StructField("firmware_version", StringType(), True)
        ])

        # Initialize multiple independent state variables
        self.last_seen = handle.getValueState("last_seen", last_seen_schema)
        self.device_info = handle.getValueState(
            "device_info", device_info_schema
        )

In the init() method, we use StatefulProcessorHandle to define and initialize two per-key state variables, last_seen and device_info, using Spark’s StructType schemas and the getValueState() API. These state variables are automatically stored in RocksDB and checkpointed to S3, allowing for fault-tolerant state management across streaming micro-batches.

2.2 Handle incoming heartbeat events and register timers

The handleInputRows() method is called whenever new events arrive for a device. This is where we update state and register timers.

def handleInputRows(
    self, key: tuple, rows: Iterator[pd.DataFrame], timerValues
) -> Iterator[pd.DataFrame]:
    device_id = key[0]

    # Process incoming heartbeats - iterate through all rows to find latest
    latest_timestamp = None
    for pdf in rows:
        for _, row in pdf.iterrows():
            ts = row['timestamp']
            if pd.isna(ts):
                continue
            if latest_timestamp is None or ts > latest_timestamp:
                latest_timestamp = ts

    if latest_timestamp is None:
        yield pd.DataFrame()
        return

    # Check if we have existing state
    existing_timestamp = None
    if self.last_seen.exists():
        existing_state = self.last_seen.get()
        existing_timestamp = existing_state[0]

    # Update state only if new heartbeat is more recent
    if existing_timestamp is None or latest_timestamp > existing_timestamp:
        # Cancel existing timers (device is back online)
        for timer in self.handle.listTimers():
            self.handle.deleteTimer(timer)

        # Update state with new timestamp
        self.last_seen.update((latest_timestamp,))

        # Register timer for heartbeat deadline detection
        current_time_ms = timerValues.getCurrentProcessingTimeInMs()
        deadline_ms = current_time_ms + HEARTBEAT_INTERVAL_MS
        # 30 seconds from now
        self.handle.registerTimer(deadline_ms)

    yield pd.DataFrame()  # No output from input handling

The handleInputRows() method processes incoming heartbeat events for each device by extracting the latest timestamp, updating the last_seen state, and managing timers. It cancels existing ones and registering a new 30-second expiry timer to detect future inactivity. Because alerts are only emitted upon timer expiration, the method yields an empty dataframe during normal heartbeat processing.

2.3 Handle timer expiration and emit alerts

The handleExpiredTimer() method is called when a registered timer fires. This is where we detect offline devices and emit alerts.

def handleExpiredTimer(
    self, key: tuple, timerValues, expiredTimerInfo
) -> Iterator[pd.DataFrame]:
    device_id = key[0]
    current_time_ms = timerValues.getCurrentProcessingTimeInMs()

    # Verify state exists
    if not self.last_seen.exists():
        yield pd.DataFrame()
        return

    # Get last seen timestamp from state
    last_seen_state = self.last_seen.get()
    last_seen_timestamp = last_seen_state[0]

    if last_seen_timestamp is None or pd.isna(last_seen_timestamp):
        yield pd.DataFrame()
        return

    # Calculate how long device has been offline
    last_seen_ms = int(last_seen_timestamp.timestamp() * 1000)
    offline_duration_ms = current_time_ms - last_seen_ms
    offline_duration_seconds = offline_duration_ms / 1000.0

    # Create alert as a Pandas DataFrame
    alert_df = pd.DataFrame({
        "device_id": [device_id],
        "alert_type": ["DEVICE_OFFLINE"],
        "last_seen": [last_seen_timestamp],
        "offline_duration_seconds": [offline_duration_seconds],
        "alert_timestamp": [datetime.fromtimestamp(current_time_ms / 1000.0)]
    })

    # Register another timer for repeat alerts (every 60 seconds)
    next_alert_time = current_time_ms + ALERT_REPEAT_INTERVAL_MS
    self.handle.registerTimer(next_alert_time)

    yield alert_df  # Emit the alert

The handleExpiredTimer() method is triggered automatically when a device’s inactivity timer expires, retrieving the last_seen state to calculate the offline duration and yielding an alert dataframe to the output stream. It also registers a follow-up timer for repeat alerts every 60 seconds, which continues until a new heartbeat arrives and cancels the timer via handleInputRows().

There are several ways you could extend this solution for production use. You could implement exponential backoff for repeat alerts to reduce noise, for example, alerting after 60 seconds, then 2 minutes, then 5 minutes, and so on. Other improvements could include adding severity escalation based on offline duration, integrating with notification services like Amazon SNS for downstream alerting, or setting a maximum retry limit to stop alerts for permanently decommissioned devices.

2.4 Apply transformWithState to the streaming DataFrame

Now we connect everything together by applying our HeartbeatMonitor processor to the streaming data.

# Read and parse heartbeat events from Kinesis
parsed_df = kinesis_df \
    .selectExpr("CAST(data AS STRING) as json_data") \
    .select(from_json(col("json_data"), heartbeat_schema).alias("heartbeat")) \
    .select(
        col("heartbeat.device_id"),
        to_timestamp(col("heartbeat.timestamp")).alias("timestamp"),
        col("heartbeat.battery_level"),
        col("heartbeat.signal_strength"),
        col("heartbeat.firmware_version")
    )

# Apply transformWithState for stateful processing
alerts_df = parsed_df \
    .groupBy("device_id") \
    .transformWithStateInPandas(
        statefulProcessor=HeartbeatMonitor(),
        outputStructType=alert_output_schema,
        outputMode="append",
        timeMode="processingTime"
    )

# Write alerts to SNS
query = alerts_df.writeStream \
    .outputMode("append") \
    .foreachBatch(send_to_sns) \
    .option("checkpointLocation", CHECKPOINT_LOCATION) \
    .trigger(processingTime="10 seconds") \
    .start()

# Send to SNS for alerts
def send_to_sns(batch_df, batch_id):
    if batch_df.count() > 0:
        sns_client = boto3.client('sns', region_name=KINESIS_REGION)
        for row in batch_df.collect():
            message = {
                "device_id": row["device_id"],
                "alert_type": row["alert_type"],
                "last_seen": str(row["last_seen"]),
                "offline_duration_seconds": row["offline_duration_seconds"],
                "alert_timestamp": str(row["alert_timestamp"])
            }
            sns_client.publish(
                TopicArn=SNS_TOPIC_ARN,
                Message=json.dumps(message),
                Subject=f"Device Offline Alert: {row['device_id']}"
            )

The streaming pipeline parses JSON heartbeat events from Kinesis, partitions them by device_id, and applies the HeartbeatMonitor stateful processor using transformWithStateInPandas() with processing-time timers and append output mode. The resulting alert stream is written to SNS via foreachBatch() with checkpointing enabled for fault tolerance and micro-batches triggered every 10 seconds.

To summarize, implementing the heartbeat monitor requires just three methods. The init() method sets up your state variables, handleInputRows() processes incoming heartbeats and manages timers, and handleExpiredTimer() generates offline alerts. The transformWithState API handles the underlying complexity of state management, checkpointing, and timer scheduling automatically.

Step 3: Create IAM role for job execution

Create an IAM role that allows EMR Serverless to assume it for running your Spark job. For detailed instructions on creating an IAM role, see Creating an IAM role. Use the following trust policy for the role.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Service": "emr-serverless.amazonaws.com"
    },
    "Action": "sts:AssumeRole"
  }]
}

Attach a permissions policy that grants the role access to read from the Kinesis stream, write to the S3 bucket for checkpoints and application artifacts, and publish alerts to the SNS topic:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "KinesisAccess",
      "Effect": "Allow",
      "Action": [
        "kinesis:GetRecords",
        "kinesis:GetShardIterator",
        "kinesis:DescribeStream",
        "kinesis:DescribeStreamSummary",
        "kinesis:ListShards",
        "kinesis:SubscribeToShard"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:*:stream/iot-heartbeats"
    },
    {
      "Sid": "SNSPublish",
      "Effect": "Allow",
      "Action": "sns:Publish",
      "Resource": "arn:aws:sns:us-east-1:*:iot-alerts"
    },
    {
      "Sid": "S3Access",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-bucket",
        "arn:aws:s3:::your-bucket/*"
      ]
    }
  ]
}

Step 4: Upload external dependencies required for executing the streaming job

In this step, you will download the required external dependencies and upload them to your S3 bucket to make them available for your EMR Serverless streaming job.

Spark-kinesis-connector.jar (download link) and copy to local S3 bucket s3://your-bucket/jars/spark-kinesis-connector.jar.
Protobuf Dependency (download link) and copy to local S3 bucket s3://your-bucket/pyfiles/protobuf_pkg.tar.gz.

Step 5: Submit the streaming job

Now that the application, IAM role, and dependencies are in place, you can submit the streaming job. This step configures the Spark job parameters and submits it to your EMR Serverless application in streaming mode. For more details on submitting jobs, see Starting a job run.

First, create a file named job-driver.json with the following content. Replace the S3 paths with the locations where you uploaded your script and dependencies in the previous steps.

{
  "sparkSubmit": {
    "entryPoint": "s3://your-bucket/scripts/heartbeat_monitor.py",
    "sparkSubmitParameters": "--jars s3://your-bucket/jars/spark-kinesis-connector.jar --archives s3://your-bucket/pyfiles/protobuf_pkg.tar.gz#protobuf_pkg --conf spark.executor.cores=4 --conf spark.executor.memory=16g --conf spark.driver.cores=4 --conf spark.driver.memory=16g --conf spark.executor.instances=3 --conf spark.sql.streaming.stateStore.providerClass=org.apache.spark.sql.execution.streaming.state.RocksDBStateStoreProvider --conf spark.sql.streaming.stateStore.rocksdb.changelogCheckpointing.enabled=true --conf spark.emr-serverless.driverEnv.PYTHONPATH=./protobuf_pkg --conf spark.executorEnv.PYTHONPATH=./protobuf_pkg"
  }
}

Then, run the following command to submit the job. Replace the application ID and account ID with your own values.

aws emr-serverless start-job-run \
  --application-id <YOUR_APPLICATION_ID> \
  --execution-role-arn arn:aws:iam::<ACCOUNT_ID>:role/EMRServerlessJobRole \
  --job-driver file://job-driver.json \
  --mode STREAMING \
  --retry-policy maxFailedAttemptsPerHour=1 \
  --region us-east-1

Running transformWithState on Amazon EMR Serverless provides several operational advantages over self-managed Spark clusters. In streaming mode, the Spark driver remains alive between micro-batches, eliminating the overhead of repeatedly starting and stopping the application. You don’t need to provision or manage executors because EMR Serverless automatically scales compute resources up and down based on workload demands, so you only pay for what you use. Your IoT heartbeat monitor can handle traffic spikes, such as thousands of devices reconnecting simultaneously after a network outage, without manual intervention. EMR Serverless also provides built-in job resiliency, real-time monitoring, and enhanced log management, reducing the operational burden of running streaming applications in production.

Testing the solution

Now that our streaming application is deployed, let’s test it by sending heartbeat events and observing the offline detection behavior.

Step 1: Open AWS CloudShell

Open AWS CloudShell in your AWS account from the AWS Management Console.

Step 2: Send heartbeat events using CLI

Execute the following bash script to send heartbeat events every 10s.

#!/bin/bash

while true; do
  aws kinesis put-record \
    --stream-name iot-heartbeats \
    --partition-key device-001 \
    --data $(echo "{\"device_id\":\"device-001\",\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"battery_level\":87.5,\"signal_strength\":-42.3,\"firmware_version\":\"v2.1.0\"}" | base64) \
    --region us-east-1

  aws kinesis put-record \
    --stream-name iot-heartbeats \
    --partition-key device-002 \
    --data $(echo "{\"device_id\":\"device-002\",\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"battery_level\":87.5,\"signal_strength\":-42.3,\"firmware_version\":\"v2.1.0\"}" | base64) \
    --region us-east-1

  aws kinesis put-record \
    --stream-name iot-heartbeats \
    --partition-key device-003 \
    --data $(echo "{\"device_id\":\"device-003\",\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"battery_level\":87.5,\"signal_strength\":-42.3,\"firmware_version\":\"v2.1.0\"}" | base64) \
    --region us-east-1

  sleep 10
done

Update the timestamp field to use the current time for each event or use a script to automate sending events at regular intervals.

Step 3: Observe normal operation

As you send heartbeat events every 10 seconds, the Spark application receives each event and updates the device’s state. A timer is then registered for 30 seconds in the future. Each new heartbeat cancels the existing timer and registers a new one, effectively resetting the countdown. As long as heartbeats continue to arrive within the 30-second window, no alerts are sent.

Timeline diagram showing normal device operation over 60 seconds with heartbeats arriving every 10 seconds, each resetting the 30-second timer

The above timeline diagram shows a 60-second window of normal device operation. Heartbeat events arrive every 10 seconds (at 0s, 10s, 20s, 30s, 40s, 50s, and 60s), each resetting the 30-second timer window. Because every heartbeat arrives well within the 30-second threshold, the timer never expires, the device state remains online, and no alerts are triggered.

Step 4: Test offline detection

Stop sending heartbeat events for the device and wait 30 seconds. You should receive an SNS alert indicating the device is offline.

Timeline diagram showing offline detection over 110 seconds with the 30-second timer expiring and triggering SNS alerts

Timeline diagram showing offline detection over 110 seconds. Device sends heartbeats at 0s, 10s, and 20s before going offline. The 30-second timer expires at 50s triggering Alert #1 via SNS, followed by a repeat Alert #2 at 110s after a 60-second repeat timer.

If you continue to not send heartbeats, additional alerts will be sent every 60 seconds.

Step 5: Test device recovery

Resume sending heartbeat events using the same CLI command. The application will cancel all existing timers for the device and will stop sending SNS alerts.

Timeline diagram showing device recovery lifecycle with timers canceled and device returning to online state

Timeline diagram showing the complete device recovery lifecycle over 140 seconds across three phases: normal operation with heartbeats, offline detection with SNS alerts, and recovery where timers are canceled and the device returns to online state

Clean up

To avoid incurring ongoing charges, follow these steps to clean up the resources.

Step 1: Stop the EMR serverless application

Stop your running streaming job:

aws emr-serverless stop-job-run \
  --application-id <your-application-id> \
  --job-run-id <your-job-run-id>

Step 2: Delete the EMR serverless application

aws emr-serverless delete-application \
  --application-id <your-application-id>

Step 3: Delete kinesis data stream

aws kinesis delete-stream --stream-name iot-heartbeat

Step 4: Remove S3 objects

Delete the checkpoint data, scripts, and dependencies from your S3 bucket:

aws s3 rm s3://your-bucket/checkpoints/ --recursive
aws s3 rm s3://your-bucket/scripts/ --recursive
aws s3 rm s3://your-bucket/jars/ --recursive
aws s3 rm s3://your-bucket/pyfiles/ --recursive

Real-world use cases for stateful streaming

The transformWithState API enables developers to build sophisticated streaming applications that were previously difficult to implement. Here are a few examples of how it can be applied across industries.

Telecommunications and network monitoring: Telecom providers need to detect network anomalies and SLA violations as they happen across millions of concurrent sessions. With transformWithState, developers can maintain per-session state to track call detail records, compare real-time network metrics against established baselines, and trigger alerts the moment thresholds are breached. Automatic state TTL ensures that completed session records are cleaned up without manual intervention.

Financial services and fraud detection: Detecting fraud requires correlating multiple signals across a sequence of transactions in real time. With transformWithState, developers can maintain per-account state that tracks transaction histories, flags suspicious patterns like rapid purchases across geographies, and calculates rolling risk scores. Multiple state variables per key allow tracking different dimensions of activity, such as transaction velocity, location changes, and spending deviations, within a single stateful operator.

E-commerce and customer engagement: Understanding customer behavior in real time is critical for driving conversions. Using transformWithState, developers can build session-aware applications that track browsing and cart activity with timer-based state expiration, detecting cart abandonment after a configurable timeout and triggering personalized re-engagement notifications. The State Data Source Reader enables teams to inspect session state mid-stream, making it easier to debug and validate real-time customer journey logic.

Conclusion

Apache Spark 4.0’s transformWithState API represents a significant advancement in stateful stream processing, making it simpler to build complex real-time applications like IoT device monitoring. Combined with Amazon EMR Serverless, you get a fully managed platform that scales automatically and eliminates infrastructure management overhead.

This post demonstrates how to use the native timer support capability of transformWithState to build a real-time IoT device monitoring application. We encourage you to explore other capabilities such as Automatic State TTL, Schema Evolution, and Multiple State Variables on Amazon EMR Serverless to build more sophisticated streaming applications tailored to your needs.

About the authors

Announcing general availability of Apache Spark 4.0 on Amazon EMR

2026-06-09 Suthan Phillips

Post Syndicated from Suthan Phillips original https://aws.amazon.com/blogs/big-data/announcing-general-availability-of-apache-spark-4-0-on-amazon-emr/

As data volumes grow and pipelines become more complex, you need an engine that handles semi-structured data natively, supports streaming state without operational overhead, and allows you to develop interactively against production-scale compute. Spark 4.0 addresses these three challenges that slow modern data teams: wrangling semi-structured data, managing streaming state, and bridging the gap between interactive development and production-scale execution. With VARIANT data type, state-management improvements, and Spark Connect availability in Spark 4.0, you can now handle these workloads with less code, fewer operational trade-offs, and faster iteration cycles, all on Amazon EMR optimized runtime, which runs Spark workloads up to 4.5× faster than open-source Apache Spark.

With this general availability announcement, Spark 4.0 is now supported across Amazon EMR Serverless, Amazon EMR on EC2, and Amazon EMR on EKS deployment options. In this post, you’ll learn about key Spark 4.0 capabilities now available on Amazon EMR including Spark Connect, the Variant data type, SQL scripting, Python API improvements, and streaming enhancements, along with infrastructure changes in the new emr-spark-8.0 release.

New features in GA

Apache Spark 4.0 introduces several capabilities that are now generally available on Amazon EMR.

Spark Connect

Most Spark development is iterative and disconnected from production. You write code locally, test it against a sample, then package and deploy it to a cluster. It often fails due to data issues at scale, environment mismatches, or dependency conflicts. The feedback loop is slow, and the gap between development and production is where most time is lost.

Spark Connect closes that gap by introducing a decoupled client-server architecture that changes how your application communicates with Spark. In previous versions, your application code and the Spark driver ran inside the same JVM process, meaning issues in your application code could destabilize the Spark driver and disrupt the entire session. Your application runs as a lightweight client that submits logical plans to a Spark server over gRPC. The server handles execution independently. Your client doesn’t require a local Spark installation, a JVM, and doesn’t need to run on a cluster node. It only needs connectivity to the server endpoint.

With Amazon EMR, this means you can write PySpark from your preferred IDE (VS Code, PyCharm), Jupyter notebooks, Amazon SageMaker Unified Studio Data Notebooks, Amazon Q Developer, or Kiro, and Spark Connect routes your DataFrame transformations and SQL queries to Amazon EMR for execution over a secure connection.You can set breakpoints, inspect variables, and step through transformations while your data is processed on serverless compute, catching issues during development instead of after deployment. There are no clusters to provision, no code to repackage, and no infrastructure to manage.

This architecture also improves session resilience. A client-side failure doesn’t affect the Spark server, so other workloads continue to run without disruption. Spark Connect is an open Apache Spark standard. The same PySpark code works across different Spark backends by changing the connection endpoint.

For example, connecting to Amazon EMR Serverless from your local IDE takes minimal lines of spark code:

from pyspark.sql import SparkSession
spark = SparkSession.builder \
    .remote("sc://<endpoint>:443/;use_ssl=true;x-aws-proxy-auth=") \
    .getOrCreate()
df = spark.sql("SELECT * FROM my_catalog.my_database.my_table")
df.groupBy("category").count().show()

On Amazon EMR Serverless, start a session to retrieve your endpoint and auth token, then connect remotely using the standard sc:// protocol. Every Spark operation executes on Amazon EMR Serverless while your code stays local.

The following video showcases Spark Connect and Variant features together.

For a step-by-step getting-started walkthrough, visit Announcing Spark Connect on Amazon EMR Serverless: Interactive PySpark development, anywhere.

Data type and table format enhancements

This section covers the VARIANT data type and Apache Iceberg V3 support. These two additions improve how you store and query semi-structured data.

Apache Iceberg V3 support

Amazon EMR has supported Apache Iceberg V3 since Amazon EMR release 7.x, introducing capabilities such as deletion vectors and row lineage. With Spark 4.0 on Amazon EMR, that support deepens unlocking capabilities that had a hard dependency on Spark 4.0 itself, including VARIANT column storage and unknown type handling. For teams running data lakehouse workloads, the table format underneath your data determines how efficiently it is stored, how reliably it evolves, and how safely multiple tools can read and write it simultaneously.

What this means for your workloads:

VARIANT and Iceberg working together: VARIANT columns can now be stored natively in Iceberg V3 tables, combining efficient semi-structured data storage with Iceberg’s schema evolution and time travel capabilities. This eliminates the pipeline complexity of upfront schema definitions.
More efficient partitioning: Multi-argument transforms accept multiple input columns in a single partition expression, such as range (order_date, product_category), giving you finer control over data layout. They produce a single composite key instead of separate columns whose cartesian product can explode partition count. The result is less data scanned, faster queries, and lower compute costs for high volume workloads.
Safer schema evolution: Unknown type handling ensures that older readers do not break when newer writers introduce new column types, reducing coordination overhead across teams and tools during upgrades.
Fine-grained access control (FGAC): Column-level and row-level permissions are now available through AWS Lake Formation, giving you governed access control at a granular level across your Iceberg tables, no custom access logic required.

Variant data type

The new VARIANT data type, supported through Apache Iceberg v3, brings native support for semi-structured JSON data directly into Spark SQL. This matters most when you don’t control the data being written because platform teams and shared services often receive data from partners and upstream teams with unpredictable or evolving structures.

Without VARIANT, handling semi-structured data meant accepting real tradeoffs: defining schemas upfront that broke when data evolved, storing everything as strings with heavy parsing costs on every read, or building wide tables with nullable columns that wasted storage on empty fields. The most realistic option was breaking nested structures apart into separate columns before running queries. This ETL step added latency, increased storage costs, and broke every time an upstream team added or removed fields from their data feed.

VARIANT eliminates the process entirely. Data stays nested and is queryable with variant_get(), without a separate ETL pipeline. You ingest without defining a schema first and apply structure at query time.

For example, querying nested fields is now a single expression:

SELECT
    variant_get(payload, '$.user.name') AS user_name,
    variant_get(payload, '$.event.type') AS event_type,
    variant_get(payload, '$.event.timestamp') AS event_timestamp
FROM VALUES
    (PARSE_JSON('{"user":{"name":"Alice"},"event":{"type":"click","timestamp":"2025-03-01"}}'))
AS t(payload)
WHERE variant_get(payload, '$.event.timestamp') > '2025-01-01';

For a deep dive into how VARIANT is stored in Parquet, shredding mechanics, and a full end-to-end walkthrough on Amazon EMR Serverless, see Beyond JSON blobs: Implementing the VARIANT data type in Apache Iceberg V3.

Key benefits for your workloads:

Reduced pipeline fragility: Schema changes no longer break ingestion. Data lands as-is, and you apply structure at query time based on what each analysis needs, without upstream coordination.
Improved query performance: Optimized storage format enables efficient access to nested fields without parsing overhead, so queries run faster even on deeply nested payloads.
Better storage efficiency: Compact encoding eliminates the waste of NULL-heavy wide tables, reducing storage costs for semi-structured data at scale.

VARIANT is especially well-suited where schemas are unpredictable or evolving: IoT and sensor data with device-specific payloads, logging and telemetry with variable event structures, and API responses and webhooks from third-party services where the schema changes without notice.

SQL enhancements

You can now write and maintain Spark pipelines using the same standard SQL you already know, no Spark-specific functions or syntax required. Apache Spark 4.0 expands ANSI SQL compliance so that functions behave consistently, opening Spark to anyone who can write SQL rather than requiring Spark specialists.

Standard SQL syntax such as OFFSET, LIMIT ... OFFSET, and lateral column aliases now work as expected. For example:

-- Standard OFFSET syntax now supported
SELECT id, name
FROM VALUES (1, 'Alice'), (2, 'Bob'), (3, 'Carol'), (4, 'Dave') AS t(id, name)
ORDER BY id
LIMIT 2 OFFSET 1;

-- Lateral column aliases work inline
SELECT amount * 1.1 AS adjusted, adjusted * 0.08 AS tax
FROM VALUES (100.0), (200.0), (350.0) AS t(amount);

Beyond syntax, SQL scripting brings procedural logic directly into Spark SQL. You can now use variables, IF/ELSE conditionals, loops, and multi-statement blocks without switching to Python or JVM-based languages. Before SQL scripting, multi-step workflows (such as ETL logic with conditional branching or iterative data quality checks) required wrapping SQL statements in Python or Scala to handle control flow. SQL scripting removes that dependency. SQL-native teams can author and maintain these workflows entirely in SQL.

Key benefits:

Simplified ETL workflows: Multi-step transformation logic that previously required an external language can now live entirely in SQL, reducing code complexity and making pipelines easier to build and maintain.
Lower barrier for SQL-native teams: Teams that primarily work in SQL no longer need to context-switch into Python or Scala to implement conditional logic or iterative processing. The entire pipeline stays in SQL.

Python advances

Earlier versions of Spark required Python users to step outside Python at two key points: building custom data connectors required Java or Scala, and diagnosing UDF performance lacked built-in visibility. Spark 4.0 addresses both directly, removing the two biggest blockers for organizations where Python is the primary language.

Python Data Source API

With the Python Data Source API, you can build custom, reusable data connectors in Python without any JVM or Scala code. Custom connectors participate in Spark’s query optimization, including predicate pushdown and schema inference. This matters when your data system only has a Python client, or when your team does not have Java or Scala expertise: you can now wrap any custom format or external source as a Spark DataFrame source or sink without leaving Python.

Spark 4.0 also introduces polymorphic Python UDTFs (User-Defined Table Functions) that can return different schema shapes depending on input, with an analyze() method that produces a schema dynamically based on parameters. This is particularly useful for processing varying JSON schemas or splitting inputs into a variable set of outputs.

If you’re ingesting data from a REST API with a Python client, you can implement a custom Spark data source entirely in Python, register it, and use it directly in Spark SQL or the DataFrame API. What previously required a Scala developer and a custom JVM connector can now be built and maintained by your Python team running the pipeline.

Python UDF enhancement

Python UDF profiling provides built-in visibility into execution time, serialization overhead, and memory usage at the individual UDF level without external tooling. Additionally, it enables performance or memory profiling depending on what you need to diagnose.

Arrow-based vectorized UDF support reduces serialization overhead between Python and the JVM using a columnar format, replacing row-at-a-time processing with batch-oriented columnar exchange.

Together, these give you a complete optimization loop: build custom connectors in Python, profile your UDF performance, and optimize with confidence.

Practical benefits for Python teams:

Lower barrier for Python teams: Custom data connectors no longer require Java or Scala knowledge. If your data system has a Python client, you can build a production-grade Spark connector entirely in Python.
Flexible data transformation: Polymorphic UDTFs let your functions adapt to varying input schemas dynamically, reducing the need to write and maintain multiple transformation functions for different data shapes.
Faster UDF optimization: Built-in profiling surfaces exactly where execution time and memory are being spent at the UDF level, replacing guesswork with direct visibility and making performance tuning significantly faster.

Streaming enhancements

This section covers improvements to state management and observability in structured streaming workloads.

Queryable state for structured streaming

Structured streaming jobs maintain state continuously (running totals, session windows, aggregated counts). However, in earlier versions of Spark that state was locked inside the running job. Inspecting it meant stopping the stream or manually parsing checkpoint files. For production workloads, this created real operational risk: teams had no way to verify whether state was correct, corrupted, or drifting without taking the job down.

Time-sensitive applications faced an additional problem: timers in Spark streaming only fired when new data arrived, so a five-minute heartbeat timeout could silently miss its window if no data came in, making applications like heartbeat monitoring and session tracking unreliable by design.

Spark 4.0 changes this. The new transformWithState API provides deterministic timer execution because timers fire on schedule regardless of data arrival patterns. It also delivers automatic state TTL to prevent unbounded growth, schema evolution without restarting from a new checkpoint, and state observability for mid-stream debugging. External systems can now read live aggregated state from a running streaming job without interrupting it. State is accessible as a DataFrame, queryable during development, verifiable in unit tests, and inspectable during production incidents without touching the running stream.

This is backed by three improvements working together. First, the transformWithState operator replaces mapGroupsWithState from earlier Spark versions (which had limited timer support and no TTL-based cleanup). Second, the state data source reader exposes streaming state as a queryable DataFrame. Lastly, RocksDB changelog checkpointing improvements address throughput bottlenecks in high-volume stateful workloads.

Consider a fleet of 100,000 IoT sensors across manufacturing facilities, each requiring an alert within 30 seconds of going offline. The sensors track heartbeat state per device, managing independent timers, handling late data, and cleaning up decommissioned devices at scale had no clean solution in earlier Spark versions. The transformWithState operator handles all of this natively, and queryable state lets your operations team inspect live device state in real time without stopping the stream:

# Timers fire on schedule regardless of data arrival, making heartbeat monitoring reliable
alerts = events_df.groupBy("device_id").transformWithState(
    HeartbeatMonitor(),
    outputStructType=StructType([
        StructField("device_id", StringType()),
        StructField("alert", StringType())
    ]),
    outputMode="Append"
)

Combined with Amazon EMR Serverless, which scales compute automatically based on workload demands, you can deploy stateful streaming pipelines without managing clusters or predicting capacity.

Benefits:

Real-time operational visibility: Live streaming state is now accessible externally without interrupting the job, powering dashboards and monitoring systems that reflect current aggregations.
Faster debugging: State values can be queried directly as a DataFrame, making it significantly easier to diagnose production incidents and verify correctness during development.
Better performance at scale: RocksDB checkpointing improvements reduce bottlenecks in high-throughput stateful workloads, improving reliability for long-running streaming jobs.

What’s new in the emr-spark-8.0 release

Beyond the Spark 4.0 capabilities covered in the preceding sections, the emr-spark-8.0 release introduces infrastructure and runtime changes that simplify how you deploy, patch, and manage Amazon EMR workloads. The release focuses exclusively on Spark, reducing the surface area you need to patch and test.

Fewer components to patch and test

The emr-spark-8.0 release includes Apache Spark 4.0, Apache Iceberg 1.10, Apache Hudi 1.0.2, Delta Lake 4.0, and connectors for Amazon DynamoDB, Amazon Kinesis, Amazon Redshift, and Amazon Simple Storage Service (Amazon S3) (via the S3A connector). Apache Livy and JupyterEnterpriseGateway are available as opt-in components on Amazon EMR on EC2. If your workloads require Apache Flink, Trino, Presto, or other execution engines, you can continue to use Amazon EMR 7.x releases.

Simplified patch management

You can specify emr-spark-8.0.x when creating a cluster or application, and Amazon EMR will automatically select the latest patch version. For example, emr-spark-8.0.1, emr-spark-8.0.2, and so on as patches are released. This “.x” wildcard is supported through AWS APIs and AWS Command Line Interface (AWS CLI). On Amazon EMR on EKS and Amazon EMR Serverless, new jobs automatically run on the latest Amazon Linux patches, so you no longer need to track date-based version tags.

Latest Python, Java, and Scala runtimes

The release ships with modernized runtimes: Python 3.11 as the default, with support for Python 3.12 and 3.13. Java 17 is the default, with Java 21 also available. Both are provided through Amazon Corretto. Scala 2.13 is the supported Scala runtime.

A few infrastructure changes to note: AWS SDK for Java v2 replaces v1, bringing improved performance and alignment with the latest AWS APIs. The EMR S3A connector replaces EMR File Systems (EMRFS) for Amazon S3 access, delivering better performance and compatibility with open-source Spark. For shuffle-intensive workloads on Amazon EMR Serverless, enabling Serverless Storage can reduce data processing costs by up to 20%. For more information, see Optimize Amazon EMR Runtime for Apache Spark with EMR S3A for benchmarks, Amazon EMR Serverless eliminates local storage provisioning, and Reducing costs for shuffle-heavy Apache Spark workloads with serverless storage for Amazon EMR Serverless.

Migration and compatibility notes

If you are migrating from Spark 3.5 to Spark 4.0, the Apache Spark upgrade agent for Amazon EMR can accelerate your migration by analyzing existing applications and identifying changes needed for Spark 4.0 compatibility. For more information, see the upgrade guidance.

If your workflows use Apache Pig, Apache Oozie, JupyterHub, Apache Zeppelin, or Hue, you can continue to use Amazon EMR 7.x releases. These components are not included in emr-spark-8.0. For interactive Spark development, use Amazon EMR Studio, with Apache Livy and JupyterEnterpriseGateway available on Amazon EMR on EC2.

For the complete list of supported components and configurations, see the Amazon EMR release guide.

Get started

Spark 4.0 is now available across Amazon EMR on EC2, Amazon EMR on EKS, and Amazon EMR Serverless. To begin, choose your deployment model and follow the relevant getting started guide:

Amazon EMR on EC2 — Getting started with Amazon EMR on EC2.
Amazon EMR Serverless — Getting started with Amazon EMR Serverless.
Amazon EMR on EKS — Getting started with Amazon EMR on EKS.
Amazon EMR release guide — Apache Spark.

Conclusion

Spark 4.0 on Amazon EMR delivers improvements across query validation, semi-structured data handling, Python development, and streaming observability. ANSI SQL mode catches invalid operations at query time rather than silently propagating nulls downstream, and SQL scripting removes the need to context-switch between SQL and Python for complex ETL logic. The VARIANT data type eliminates parsing overhead for semi-structured JSON workloads and can now be stored natively in Iceberg V3 tables with fine-grained access control at the column and row level. Queryable streaming state gives you live visibility into running jobs without interruption, and Spark Connect lets you develop against Amazon EMR from Jupyter notebooks, Amazon SageMaker Unified Studio Data Notebooks, Amazon Q Developer, Kiro, or your preferred IDE without managing cluster connectivity.

Ready to build or migrate? Choose your deployment model from the preceding section and get started today. For detailed guidance, see the Amazon EMR Release Guide and the Amazon EMR Serverless User Guide.

About the authors

Capture data lineage of Amazon EMR spark jobs into Amazon SageMaker Unified Studio

2026-05-21 Jose Romero

Post Syndicated from Jose Romero original https://aws.amazon.com/blogs/big-data/capture-data-lineage-of-amazon-emr-spark-jobs-into-amazon-sagemaker-unified-studio/

Data engineers running Apache Spark jobs on Amazon EMR face a persistent challenge: understanding how data moves through Spark pipelines as it’s transformed, joined, and written to downstream tables . Tracking these transformations manually requires examining job logs, reviewing code, and piecing together transformation logic across multiple sources. As pipelines scale, this process becomes complex. The visibility gap affects key business activities: troubleshooting data quality issues takes longer – impact analysis for schema changes requires more effort – and compliance audits need extensive documentation of data provenance.

Amazon SageMaker is the center for all your data and analytics where you can find and access all the data in your organization and act on it using tools across various use case. This unified platform addresses the data visibility challenge by bringing together data governance, collaboration, and discovery into a single interface. At the heart of this platform is Amazon SageMaker Catalog, a centralized hub that enables organizations to catalog, govern, and discover all their data assets with complete visibility into lineage. By capturing data lineage across your entire data ecosystem from raw sources through transformations to final outputs, SageMaker Catalog enables you to track data provenance across your entire platform, enable collaboration with clear visibility into data ownership and quality metrics, build trust through comprehensive data lineage that supports compliance and confident decision-making, and accelerate discovery of trustworthy, governance-ready data assets. You can access and visualize this lineage directly in Amazon SageMaker Unified Studio, which serves as the unified interface to explore data relationships and collaborate across your analytics workflows.

Amazon EMR, starting from version 7.11, now includes native OpenLineage support that automates lineage capture. OpenLineage is an open-source framework for data lineage that automatically emits lineage metadata from your data transformation jobs directly into Amazon SageMaker Catalog, or other data governance solutions, without requiring customizations.

This EMR native support of OpenLineage is part of a growing set of integrations across AWS analytics services including AWS Glue, Amazon EMR Serverless, and Amazon Redshift. The complete list of services with native OpenLineage integration can be found in the data lineage support matrix.

In this post, you’ll walk through a practical, step-by-step example that shows how to capture and track data lineage from Spark jobs running on Amazon EMR directly into Amazon SageMaker Catalog using OpenLineage. You’ll see how lineage metadata flows automatically and explore data relationships and dependencies across your workflows in Amazon SageMaker Unified Studio.

Solution overview

Imagine you’re part of a large enterprise that relies on HR analytics to optimize workforce planning, compensation strategies, and talent retention practices. Your data engineering team owns the delivery of these analytical products by processing raw HR datasets (including employee records, attendance logs, and compensation details), with Spark jobs running on your Amazon EMR infrastructure.

With time, Spark jobs have grown in complexity. Your team now struggles to maintain visibility into how data moves through pipelines, who modified it, and how to map dependencies between datasets and final analytical products.

The following solution demonstrates how you can address these challenges by automatically capturing data lineage end-to-end from Spark jobs running on your EMR infrastructure and visualizing it in Amazon SageMaker Unified Studio so that you and the business understand data provenance of the final analytical products.

AWS cloud data pipeline architecture diagram showing data flowing from Amazon S3 CSV files (employees.csv, attendance.csv) through Amazon EMR with Apache Spark processing, AWS Glue Data Catalog metadata management, and Amazon SageMaker Catalog integration, producing salary_adjustments.csv and bonus_payments.csv output files stored in Amazon S3.

The architecture includes a Data Layer with CSV files containing employee, attendance, salary, and bonus data stored in Amazon S3 (Simple Storage Service), representing typical HR and payroll source systems.

The Processing Layer uses Amazon EMR cluster running Apache Spark jobs that transform raw data into analytical tables. The first Spark job joins employee and attendance data while the second Spark job combines attendance with compensation data. Both jobs use Apache Iceberg table format to provide ACID (Atomic, Consistent, Isolated, and Durable) transactions and time travel capabilities.

The Metadata Layer uses AWS Glue Data Catalog to store Iceberg table metadata, making tables discoverable and accessible across AWS analytics services. A Lineage Layer uses the OpenLineage integration in EMR to automatically track input/output datasets (CSV files and Iceberg tables), transformation logic at column level (joins, filters, aggregations), and job execution metadata.

Finally, the Data Governance Layer uses Amazon SageMaker Catalog to capture and process OpenLineage events posted by the EMR Spark jobs and automatically build a comprehensive lineage graph that shows complete data provenance from CSV source files through Spark transformations to Iceberg analytical tables.

Before you deploy this solution, make sure you have the following resources in place.

Prerequisites

For this walkthrough, you should have the following prerequisites:

An AWS account.
Your assumed role should have full access to Amazon EMR serverless, Amazon S3, Amazon Identity and Access Management (IAM) and AWS Lambda. Note that for production workloads, minimum permissions are recommended.
A Amazon VPC (Virtual Private Cloud) with at least one subnet with internet access. You can provision this VPC as you create the Amazon SageMaker domain next.
An existing Amazon SageMaker Unified Studio domain and project. To get started, use the quick setup option as explained here. To create a project, follow the instructions here.
An S3 bucket with the sample data files and Spark scripts uploaded (see Prepare Your Source Data below)
Default EMR service roles — if this is your first time using EMR in this account, run `aws emr create-default-roles` from the AWS CLI or CloudShell to create them.

With these prerequisites in place, let’s examine what the AWS CloudFormation template will deploy to your AWS environment.

Architecture components

The deployment creates several interconnected components that work together to capture and visualize lineage:

An S3 bucket to store all data and artifacts for the solution.
An EMR cluster (v 7.12.0) with Apache Iceberg support enabled and OpenLineage integration pre-installed, ready to run Spark jobs with lineage tracking.
A set of IAM policies that grant the necessary permissions to the EMR cluster to post lineage events to your SageMaker Unified Studio domain.
A set of AWS Lake Formation permissions that grant the EMR cluster to create, alter, and drop Iceberg tables in your specified Glue database.

With an understanding of what will be deployed, you’re ready to launch the CloudFormation stack.

Deploy the solution

Note: While this walkthrough uses the AWS EMR console and AWS CLI to verify the cluster and run Spark jobs, you can also perform these steps directly from Amazon SageMaker Unified Studio. SMUS provides a unified interface to create and manage EMR clusters, submit Spark jobs, and monitor execution — all within the same environment where you’ll later explore the lineage captured in Amazon SageMaker Catalog.

Prepare your source data

Before deploying the CloudFormation stack, clone or download the following git repository.PutHereGitRepo

Upload the CSV files downloaded from git to the input/ prefix and the spark scripts in scripts/ prefix. You can run the following command to upload the files:

aws s3 cp employees.csv s3://YOUR-BUCKET/input/
aws s3 cp attendance.csv s3://YOUR-BUCKET/input/
aws s3 cp salary_adjustments.csv s3://YOUR-BUCKET/input/
aws s3 cp bonus_payments.csv s3://YOUR-BUCKET/input/
aws s3 cp emr-lineage-spark-job.py s3://YOUR-BUCKET/scripts/
aws s3 cp emr-lineage-compensation-job.py s3://YOUR-BUCKET/scripts/

To deploy the solution, complete the following steps in CloudFormation console:

Create new stack by specifying the CloudFormation yaml file previously download from git repository PutHereThe YMLFileName
Enter a stack name (e.g., emr-lineage-demo) and provide the following parameters:
- SourceS3BucketName: S3 bucket containing your CSV files and Spark scripts
- SourceCSVPrefix: S3 prefix where CSV files are located
- SourceScriptsPrefix: S3 prefix where Spark scripts are located
- GlueDatabaseName: The name of the Glue database associated to your Amazon SageMaker Unified Studio project.
- DataZoneDomainId: Your SageMaker Unified Studio domain ID.
- VpcId: The id of the VPC that was deployed as part of the prerequisites.
- For EMRReleaseLabel, MasterInstanceType, CoreInstanceType and CoreInstanceCount, keep the default values.
Acknowledge IAM resource creation, choose Next and then Submit. The CloudFormation stack takes approximately 10 to 15 minutes to complete.
In the EMR console, wait for the cluster status to show as WAITING before moving to the next step.

Screenshot of the Amazon EMR on EC2 Clusters management console showing a list of 14 clusters, with the cluster "EMR-Lineage-Demo-emr-ec2-lineage-demo-stack" (ID: j-3APWOTUDNYO2T) highlighted in a "Waiting – Ready to run steps" status with a green badge.

Now that the EMR cluster is running with OpenLineage enabled, let’s examine how the Spark jobs are configured to capture lineage metadata.

Explore data lineage configuration in EMR

When submitting Spark jobs to EMR, specific configurations enable OpenLineage to create and post lineage events to SageMaker Unified Studio as the job runs:

spark.hadoop.hive.metastore.client.factory.class – Configures Spark to use AWS Glue as the Hive metastore.
spark.jars – Path to the pre-installed OpenLineage library (available on EMR 7.11+).
spark.extraListeners – Registers an OpenLineage listener to capture metadata of input / output datasets and transformations.
spark.openlineage.transport.type – Uses the OpenLineage DataZone transport option to send lineage events directly into SageMaker Catalog.
spark.openlineage.transport.domainId – The ID of your SageMaker Unified Studio domain, that serves as the target for lineage events.
spark.glue.accountId – Your AWS account ID for Glue data catalog operations.

Now that you understand the configuration that enables automatic lineage capture, you’re ready to run the data pipeline.

When running this two-step pipeline, you will calculate the total employee compensation by combining salary adjustments, bonuses, and attendance data. The final analytical asset will serve payroll processing and budgeting.

Run employee attendance analysis job

The first job reads employee details (in employees.csv dataset) and attendance records (in attendance.csv dataset), joins the datasets on EmployeeID and creates a unified dataset (employee_attendance Iceberg table) in your Glue database.

Follow the steps below to run this first job:

In the CloudFormation console, navigate to the stack’s Outputs tab
Copy the value of the Job1SubmitCommand output key. Note that this is the command you’ll use to submit the first job in EMR with the right configuration.

AWS CloudFormation console screenshot showing the Outputs tab for the "emr-ec2-lineage-demo-stack" stack, displaying 9 outputs including the Job1SubmitCommand — an AWS EMR add-steps command with Apache Spark configuration for the EMR Lineage Demo Job targeting cluster j-3APWOTUDNYO2T.

Run the command in your terminal or AWS CloudShell.
Monitor the job in the Amazon EMR console under Steps.

Screenshot of the Amazon EMR console Steps tab for the cluster "EMR-Lineage-Demo-emr-ec2-lineage-demo-stack," showing one completed step named "EMR-Lineage-Demo-Job" with Step ID s-0270631D8DHBCJZKBAZ and a green "Completed" status checkmark.

Run employee compensation analysis job

Now, you will calculate the total employee compensation (Iceberg table) by combining salary adjustments (salary_adjustments.csv dataset), bonuses (bonus_payments.csv dataset), and attendance (calculated in the last step):

Repeat the steps 1 to 4 to run Job 2.
After completion, open the AWS Glue console.
Navigate to Data Catalog, then Tables and select your SageMaker project’s database.
Confirm that employee_attendance and employee_compensation tables are listed.

With both Spark jobs complete, you can now visualize the complete data lineage graph in Amazon SageMaker Unified Studio.

Visualizing lineage in SageMaker Unified Studio

SageMaker Unified Studio provides a graph-based data lineage visualization that helps data engineers, analysts, and data scientists clearly understand which source datasets (files or tables) feed into each dataset, what transformations and logic are applied at every step, which downstream analytics assets consume the data, and how changes to upstream data or transformations may impact the rest of the data pipeline.

Now that the data pipeline run successfully, let’s review the captured lineage for the HR data in SageMaker Unified Studio:

Navigate to the SageMaker Unified Studio console, sign in to your domain.
Open your project and go to Data Sources
Find your AWS Glue Data Catalog source

Screenshot of the Amazon SageMaker project catalog Data Sources page listing three configured data sources: a Redshift Serverless source, an AWS Glue Lakehouse source named "AwsDataCatalog-emr_ec2_lineage_blogpost_glue_db-default-datasource" (highlighted), and a Tooling SageMaker model package group source — all scheduled MTWTFSS and in Ready or Running status.

Click RUN. Two new assets will be created.

Screenshot of the AWS Glue Data Catalog interface showing run activities for the data source "AwsDataCatalog-emr_ec2_lineage_blogpost_glue_db-default-datasource," with two completed on-demand runs and a highlighted asset table showing employee_attendance and employee_compensation successfully created in the emr_ec2_lineage_blogpost_glue_db database.

Navigate to Assets and Click on employee_compensation. Under the LINEAGE tab you’ll find the lineage graph view that SageMaker builds based on the OpenLineage metadata captured from the EMR Spark jobs as they run.

- You’ll first see three lineage nodes from left to right: one representing the EMR Spark job that created the final Iceberg table, a second one representing the actual Iceberg table in the Glue catalog, and a third one representing the data asset in the SageMaker Catalog inventory that maps to the Glue table.
- Click on any lineage node to view its underlying metadata in the details pane, including dataset names, S3 locations, schema, data types, job execution details and more.

Expand the lineage to the left by clicking on the double arrow next to the first lineage node. Keep expanding until you hit the originating datasets.

- Expanding the graph to the left reveals the complete data pipeline back to original CSV source files. You can see how compensation data depends on upstream attendance analytics.
- Note how each lineage node represents an element in the data pipeline you run, including both Spark jobs and even the intermediate employee_attendance Iceberg table that connects them.

You can expand column-level lineage by clicking on the column section of a lineage node of a dataset or data asset. This allows you to understand how data changes at a column level as it goes downstream your data pipeline.

Data lineage diagram showing the employee compensation ETL pipeline with four Amazon S3 source tables (employee.csv with 5 columns, input/attendance.csv with 6 columns, input/salary_adjustments.csv with 4 columns, output/employee_attendance.csv with 14 columns) processed by two Apache Spark jobs to produce a final s3://employee_compensation table with 20 columns, all dated Dec 22, 2025.

Cleanup

To avoid ongoing charges, clean up the resources:

First, empty the destination bucket by running the following command in your terminal or with AWS CloudShell.

aws s3 rm s3://${DEST_BUCKET}/ --recursive

Delete the CloudFormation stack.
- On the AWS CloudFormation console, choose Stacks in the navigation pane.
- Choose the stack you created, then choose Delete and then Delete stack when prompted.

Conclusion

In this post, you explore how to capture data lineage from Spark jobs in Amazon EMR (v7.11+) directly into Amazon SageMaker Unified Studio. You learned how to set up an Amazon EMR cluster with native OpenLineage support to automatically track lineage metadata from Spark jobs processing your data. You also configured the integration between EMR and Amazon SageMaker Catalog to ensure lineage information flows seamlessly into your governance platform. Finally, you explored the resulting lineage graph in SageMaker Unified Studio and saw how it provides comprehensive visibility into data transformations, from source CSV files through Spark processing jobs to final analytical tables using Apache Iceberg format.

We encourage you to now test these capabilities with your own data pipelines running on EMR. By implementing automated lineage tracking, many customers have strengthened their governance frameworks while gaining valuable insights into data dependencies, impact analysis, and compliance requirements. This approach enables data teams to build trust in their analytics outputs while maintaining the agility needed to derive business value from their data assets.

Prerequisites

How it works

Example: One query with three potential MVs

The trade-off

Validating automatic query rewrite

Performance considerations

Conclusion

References

About the authors

The challenge: Outgrowing a single-engine data platform

Gaining visibility: Measuring workload impact

Cleanup: Reducing unnecessary data warehouse load

Workload selection: Choosing what to offload

Proof of concept: Offloading to EMR Serverless with Spark SQL

Overcoming initial performance challenges

Validation: Ensuring data parity before cutover

Moving to production: Expanding workload offloading

Additional optimizations: Redshift Serverless, workload isolation, and Amazon EMR on Amazon EC2

Amazon Redshift rightsizing: Iterative cluster optimization

Workload isolation with Redshift Serverless

Operational isolation refinements

Transitioning to EMR on EC2

AI-assisted SQL translation

Results: A modern multi-engine architecture

Lessons learned

Conclusion

About the authors

Solution overview

Prerequisites

Part A: Configure Amazon S3 Tables integration with Trino on Amazon EMR using AWS CloudFormation

Understanding the deployment

Part B: Connecting Trino to Amazon S3 Tables with Iceberg REST endpoint

1. Catalog configuration details

2. S3 Tables Iceberg REST endpoint configuration properties

3. Configure Amazon EMR service IAM role trust relationships

4. Working with S3 Tables in Trino on Amazon EMR

4.1. Connecting to Trino on Amazon EMR

4.2. Examples: Creating and querying tables

4.2.1 Creating a namespace

4.2.2 Creating a table

4.2.3 Inserting data

4.2.4 Querying data

4.3 Explore advanced features

4.3.1 Time travel queries

4.3.2 Schema evolution

Cleaning up

Conclusion

About the authors

Prerequisites

How AWS DevOps Agent discovers custom tools through MCP

Why Spark internals visibility matters

Integrating AWS DevOps Agent with Apache Spark Troubleshooting MCP

How it works

Setting up the demo

Step 1: Clone the repository

Step 2: Deploy the AWS CloudFormation stack

Step 3: Create an agent space

Step 4: Create the AWS DevOps Agent private connection

Step 5: Register the Apache Spark Troubleshooting MCP server as a capability provider

Step 6: Add the MCP server to the agent space

Seeing it in action

The failing workload

Submit the failing job

Watch the CloudWatch alarm fire

Ask AWS DevOps Agent to investigate

Review the root cause and fix

Clean up

Conclusion

About the authors

Cluster configuration

Instance specifications

Spark configuration

Getting started

Prerequisites

Performance benchmarks and cost efficiency

Where GPU acceleration excels

Where CPU wins

Choosing the right instance

Conclusion

About the authors