Post Syndicated from Neil Mukerje original https://aws.amazon.com/blogs/big-data/introducing-the-apache-spark-optimization-agent-for-amazon-emr-serverless-and-aws-glue-etl/
With Amazon EMR Serverless and AWS Glue ETL, you can run Apache Spark applications without provisioning or managing clusters. You submit jobs, the service allocates the compute, scales it during execution, and releases it when the job finishes. Both services deliver a balanced performance profile by default. However, because of different datasets and transformations, customers who want to push further on cost or performance still need to tune Spark settings. Investigating why a specific job is slow or expensive often requires deep Spark expertise: analyzing Spark History Server timelines, shuffle metrics, executor sizes, and task distributions. That work falls to the few Spark experts on the team, and their time is often the bottleneck.
Today we are announcing the general availability of the Apache Spark optimization agent for Amazon EMR Serverless and AWS Glue ETL. The agent analyzes completed Spark job runs and delivers optimization recommendations in minutes, replacing hours of manual analysis by Spark experts. Any engineer can select a cost or performance goal and receive specific configuration changes grounded in evidence from the job’s own Spark telemetry. The agent supports single job-run analysis as well as comparison against a known-good baseline job run.
The Apache Spark optimization agent is available at no additional charge.
In this post, we explain what the agent does, how it works, and how to use it.
What the Apache Spark optimization agent does
The optimization agent analyzes Spark jobs and recommends changes that reduce cost, improve performance, or both. You can access the agent from the EMR Studio UI, AWS Glue Studio UI, or through the Amazon SageMaker Unified Studio MCP Server using any MCP-compatible client such as Kiro. You point it at a completed job run, choose either cost or performance as the intent, and receive recommendations that cite the specific metrics from that run. Often tuning for faster performance also lowers resource usage and cost, but in some cases a faster job requires more resources, which increases cost. The agent optimizes for the goal you choose, cost or performance, and makes sure the one you did not pick does not get noticeably worse. For example, a cost recommendation can make a job run slightly longer, but no more than the normal variation between runs. For larger and more expensive jobs, the agent tightens the threshold further.
There are two ways to analyze a Spark job:
- Single job-run analysis. The agent examines one completed run and identifies common sources of inefficiency: data skew, memory pressure, excessive shuffle and disk spill, undersized or oversized workers, suboptimal join strategies, and input layout issues such as small files. Each recommendation shows the Spark properties to change, the current and proposed values, and the telemetry evidence behind the suggestion.
- Baseline comparison. When a job that used to meet its targets starts running longer or costing more, compare it against a run that performed well. Provide the current run and the earlier baseline, and the agent compares their configuration, metrics, and stage behavior to pinpoint what changed.
The agent also evaluates a best practices checklist from Top 10 best practices for EMR Serverless, Best practices for performance tuning AWS Glue, and Improving AWS Glue performance. It recommends known platform-level improvements such as upgrading to a newer Spark version, using AWS Graviton processors, and applying best practices for data formats like Apache Iceberg.
The agent is read-only. It does not change your code or job settings. You decide which recommendations to apply.
How the Apache Spark optimization agent works
The following diagram shows how a request flows from the AWS console or your MCP-compatible client to the remote MCP server, through feature extraction and the recommendation engine, and back to you as recommendations.
Figure 1: How a request flows from the console or MCP client through the recommendation engine and back as recommendations
The workflow consists of the following steps:
- State your intent in plain language, such as performance or cost for a completed job run.
- Your MCP client or the AWS console sends the job run to the remote MCP server.
- The agent reads the run’s Spark History Server telemetry, configuration, logs, and cost through existing APIs, over TLS-encrypted connections, and uses them for feature extraction and context building.
- A recommendation engine combines rule-based investigations, intent-based deterministic models, and a knowledge base to find the bottleneck and determine the fix.
- You get back labeled recommendations, each with the evidence behind it.
The agent approaches each request with the objective to optimize either performance or cost. It then uses read-only access through AWS APIs and the Persistent Spark History Server or AWS Glue Apache Spark Web UI to gather the Spark configuration and telemetry. Additionally, the configuration inspector collects the job’s settings and compares them against a baseline run when one is provided.
The investigation and recommendation engine looks at these artifacts together to find what is driving runtime and cost. A slow stage may stem from too little parallelism, a skewed join key, memory pressure that forces spill, or a slow upstream read. High cost might come from idle executors or capacity well beyond what the job uses.
Rule-based investigations and an intent-based deterministic model calculate the recommended values. An AI model combines the findings with a knowledge base of the settings most often tuned across millions of Spark jobs on AWS. The agent only recommends changing a setting when the evidence supports it. When cost and runtime trade off against each other, the agent optimizes for the metric you choose and keeps any impact on the other within defined limits.
Note: The agent does not use customer data to train or build the knowledge base.
The Spark optimization agent includes built-in guardrails for personally identifiable information (PII), sensitive data, security, and cost thresholds to make sure recommendations are safe.
Each recommendation is labeled with what it changes:
- [infra]: the compute resources your job runs on, such as the EMR Serverless application’s pre-initialized and maximum capacity, the worker vCPU and memory size, or the AWS Glue worker type and number of workers.
- [config]: Spark configuration, such as
spark.sql.shuffle.partitions,spark.executor.memory, or an Adaptive Query Execution (AQE) setting. - [data]: a change to the underlying data’s physical layout, such as repartitioning a table on a different key or compacting small files.
Each category needs a different level of review. A [config] change can typically be validated in the next run without any code change. A [data] change modifies the job’s actual behavior, or the data it depends on, and should go through the same review and testing discipline as any other production change before it ships.
How to use the Apache Spark optimization agent
You can run the optimization agent from the AWS console or from any MCP-compatible client, such as Kiro, Claude Code, or Codex, through the analyze_spark_workload tool on the Amazon SageMaker Unified Studio remote MCP server. From an MCP client, ask in plain language, and words like “faster” or “cheaper” set the goal. For setup steps, see Use Spark Optimization Agent.
The following steps show how to use the agent in the AWS console.
EMR Serverless
- In EMR Studio, select your Application, and choose the Batch job that you want to optimize.
- Choose Analyze with AI.
- You can select to optimize for Performance or Cost. If you have a previous job run that was faster, you can also choose to Compare with a baseline job. Then, choose View Recommendations.
- Review the recommendations in the AI Analysis tab and apply them to the next run.
AWS Glue
- Go to the AWS Glue console.
- Under ETL jobs, choose the AWS Glue ETL job that you want to optimize.
- Choose the Runs tab, select the succeeded job run and choose Analyze with AI.
- You can select to optimize for Performance or Cost. If you have a previous job run that was faster, you can also choose to Compare with a baseline job.
- Once the AI Analysis has completed, you can view the recommendations, apply the changes, and perform the next run.
Use cases
The agent works across many situations. The following table shows sample prompts for each goal. In an MCP-compatible client, include the job run ID, and for a comparison, the baseline run ID.
| Goal | Sample prompts |
| Performance | “My job is slow. How do I make this job faster?” |
| Cost | “My bill is too high.” / “Reduce spend without impacting reliability.” |
| Baseline comparison | “Why is my job running slowly today?” / “My job used to finish in 10 minutes but now it takes an hour.” |
The following three use cases are common situations that data teams run into. Each one describes a scenario, the Spark issues that the agent looks for, and what you get back. They apply to both Amazon EMR Serverless and AWS Glue ETL.
My job is missing its SLA. How do I make it finish sooner?
A nightly orders pipeline that feeds an 8 AM dashboard takes longer as time goes on, and adding more compute power is not always a good practice. You ask the agent to optimize the run for performance. It identifies the bottleneck and then recommends the exact settings to change for that stage so the job finishes sooner.
How can I make this job cheaper?
The bill keeps rising, but every job finishes on time, so the jobs rarely get reviewed for optimization. You ask the agent to optimize the run for cost without significantly affecting the runtime. It compares the executors a job held with what its stages kept busy, exposing over-provisioning, such as a minimum executor count set too high or dynamic allocation left uncapped. It then sizes capacity to actual use. Before you apply these changes, it shows the runtime trade-off between cost and performance. For example, reducing the number of executors can improve resource utilization and lower cost but can also increase the job’s runtime.
Why is this run slower than my baseline run?
A job now takes several times as long on the same datasets compared to previous runs. This is a baseline comparison scenario. You give the agent the slow run and an earlier run that finished on time. Instead of tuning the slow run in isolation, the agent compares the two runs side by side, covering their configuration, metrics, and stage behavior. It isolates the differences that explain the regression, and tells you which settings to restore to get back to baseline performance.
Optimization results
The agent tunes every recommendation to the specified goal and checks to confirm it does not regress the other dimension beyond tolerance.
We tested the Apache Spark optimization agent across hundreds of workloads on Amazon EMR Serverless and AWS Glue ETL. The tests included industry benchmarks (derived from TPC-DS and TPC-H) at various scale factors, along with customer-representative workloads shared by enterprise AWS Glue and Amazon EMR customers.
The evaluation set is made up of Spark workloads that follow common patterns from multiple different industries such as ecommerce, industrial telemetry, manufacturing, and more. Each workload copies the structure of a typical production job, including its join strategies, shuffle sizes, and skew, so the agent’s recommendations can be checked against realistic performance. Together, the workloads cover most production ETL patterns: ingestion, multi-way joins, upserts, slowly changing dimensions, rollups, and full query benchmarks. They run on several open table formats and on standard columnar storage.
The following table shows the ranges of cost improvement observed when invoking the agent with the cost intent and the performance improvement observed with the performance intent. Across hundreds of workloads, the agent reduced cost by 24-31% with the cost intent and improved performance by 23-49% with the performance intent.
| Workload | Cost improvement | Performance improvement |
| Aggregate (across 100+ workloads) | 24–31% | 23–49% |
| TPC-DS 3 TB | 30–78% | 32–44% |
| TPC-H 3 TB | 10–64% | 19–33% |
The ranges in the table reflect testing across different versions of Amazon EMR Serverless and AWS Glue ETL.
Note: The cost or performance improvement you see depends on your workload, its current configuration, and the version of Amazon EMR Serverless or AWS Glue ETL you are running.
Conclusion
In this post, we introduced the Apache Spark optimization agent for Amazon EMR Serverless and AWS Glue ETL. We showed how the agent analyzes successful Spark runs and recommends changes that reduce cost and improve runtime, how it explains why a run regressed against earlier faster runs, and how each recommendation points back to the measurement behind it. We also walked through how the agent works and different ways to use it.
If a job run fails, the Apache Spark troubleshooting agent under the same MCP endpoint analyzes the run’s logs and Spark event data, identifies the root cause, and explains it in plain language.
To get started, pick a job that you run often or that you want to optimize, and ask the agent how to optimize it for cost or performance. Validate the recommendations in a test run before you apply them in production. If you already use the Apache Spark troubleshooting agent, you can start right away because both agents use the same MCP server.








