Customers running multiple Amazon Kinesis Data Streams often struggle to estimate the cost impact of switching between capacity modes. As accounts grow to tens or hundreds of streams, manually reviewing Amazon CloudWatch metrics for each stream and comparing pricing across Provisioned, On-demand Standard, and On-demand Advantage becomes impractical. Teams often stay on their current mode, unsure whether switching would save money or cost more, leaving potential savings unquantified. On-demand Advantage is an account-level setting that unlocks additional capabilities and a different pricing structure for on-demand streams in an AWS Region. However, without a clear, data-driven comparison, the decision to enable it remains difficult to justify.
In this post, we show you how to deploy an AI-powered agent built on Amazon Bedrock. The agent automatically analyzes every Kinesis Data Stream in your account and compares costs across all three capacity modes. It tells you exactly which streams to move to On-demand and whether your account qualifies for On-demand Advantage pricing, all on a daily or weekly schedule with zero manual intervention. As we showed in Kinesis On-demand Advantage saves 60%+ on streaming costs, choosing the right mode can save over 60 percent on streaming costs. This agent automates that analysis for you.
What is the Kinesis Mode Optimizer Agent?
The Kinesis Mode Optimizer Agent is an open source, serverless solution that uses Amazon Bedrock AgentCore, a platform to build, connect, and optimize agents at scale, with any framework or model. The agent autonomously analyzes your Kinesis Data Streams usage. It collects 7 days of Amazon CloudWatch metrics for every stream in the Region and discovers Enhanced Fan-Out (EFO) consumers. It then computes a three-way cost comparison (On-demand Standard, On-demand Advantage, Provisioned) and generates per-stream recommendations along with an account-level On-demand Advantage assessment.
The agent strongly prefers on-demand modes for their operational simplicity (automatic scaling, no capacity planning, and no throttling risk).
Results are stored as both a visual HTML report and machine-readable JSON in Amazon Simple Storage Service (Amazon S3).
Architecture
The solution uses the following architecture:
Figure 1: Architecture of the Kinesis Mode Optimizer Agent
The architecture flow includes the following steps:
Amazon EventBridge Schedule triggers the Scheduler Lambda, an AWS Lambda function, on your configured cadence (daily, weekly, or custom cron).
Scheduler Lambda invokes the Amazon Bedrock AgentCore harness with the instruction to analyze streams and generate a report.
Amazon Bedrock AgentCore Gateway, a capability of Amazon Bedrock AgentCore powered by Claude Sonnet, interprets the request and routes it to the appropriate Model Context Protocol (MCP) tools exposed by the Tool Lambda.
Tool Lambda performs the heavy lifting, fanning out to three downstream services:
Kinesis Data Streams – Lists streams in the Region, describes each stream (shard count, mode, retention), and discovers Enhanced Fan-Out consumers per stream.
CloudWatch – Pulls 7 days of metrics per stream (IncomingBytes, OutgoingBytes, throttle events) and computes three-way cost comparison.
Amazon S3 – Generates per-stream recommendations and an account-level Advantage assessment, then stores the final HTML and JSON reports.
Amazon Bedrock AgentCore harness summarizes the findings and returns them to the caller.
The entire stack is deployed using AWS Cloud Development Kit (AWS CDK) with a single cdk deploy command.
Prerequisites
Before you begin, verify that you have the following:
AWS CDK – npm install -g aws-cdk.
Python 3.12+.
aws-cdk-lib >= 2.251.0 – for AgentCore L2 constructs.
AWS Command Line Interface (AWS CLI) configured with credentials that have permissions to deploy the required resources.
Amazon Bedrock model access – verify you have access to Claude Sonnet 4.5 (or your chosen model) in the target Region. Check in the Amazon Bedrock console under Model access.
Walkthrough
In the following sections, you deploy the Kinesis Data Streams Mode Optimizer Agent and test it against your Kinesis streams.
Step 1: Clone the repository
git clone https://github.com/aws-samples/sample-kinesis-optimizer-agent.git
cd sample-kinesis-optimizer-agent
Step 2: Install CDK dependencies
cd infra
pip install -r requirements.txt
Step 3: Set your target Region
The stack deploys to whatever Region is set in AWS_DEFAULT_REGION. Set it before running any CDK commands:
# Linux/macOS
export AWS_DEFAULT_REGION=us-east-1
# Windows PowerShell
$env:AWS_DEFAULT_REGION="us-east-1"
Step 4: Bootstrap CDK (first time per account/Region)
cdk bootstrap aws://<ACCOUNT_ID>/<REGION>
Step 5: Deploy
cdk deploy
Optionally customize the schedule and bucket name:
# Weekly instead of daily
cdk deploy --parameters ReportSchedule="rate(7 days)"
# Custom bucket name
cdk deploy --parameters ReportBucketName=amzn-s3-demo-bucket
The HTML report includes a per-stream action table with priority indicators, detailed cost breakdowns, and the account-level Advantage recommendation.
Figure 2: Sample HTML report with the per-stream action table
Figure 3: Account-level On-demand Advantage recommendation in the sample report output
Multi-Region deployment
The agent is Region-specific. When deployed to a Region, it analyzes only the streams in that Region. To cover multiple Regions, change the environment variable and repeat the deployment:
Each deployment is independent, with its own agent, Amazon S3 bucket, and schedule.
Clean up
To remove the stack from a Region:
cd infra
cdk destroy
Note: The Amazon S3 bucket has a RemovalPolicy.RETAIN setting and isn’t deleted with the stack. Delete it manually if you no longer need the reports.
Conclusion
In this post, you deployed an AI-powered agent that autonomously analyzes your Amazon Kinesis Data Streams and recommends the optimal capacity mode for each stream. The agent alleviates the manual effort of reviewing CloudWatch metrics across dozens or hundreds of streams and produces actionable, cost-aware recommendations on a recurring schedule.
By shifting from manual capacity reviews to autonomous, scheduled optimization, you gain three key benefits. First, you can reduce streaming costs by identifying streams that should switch modes. Second, you alleviate throttling risk by catching under-provisioned streams before they impact performance. Third, you free your team from repetitive operational work. All of this is achievable with a single cdk deploy.
Imagine preparing for your biggest sales event of the year, and you want to ensure your customer identity management service can handle the elevated traffic for carrying out application activities. For security teams, business leaders, and technologists managing identity infrastructure at scale, this scenario has been all too familiar. Whether you’re a CISO evaluating security controls, a CFO managing cloud costs, traditional support ticket processes for adjusting Amazon Cognito rate limits meant waiting 10–14 days for capacity increases, requiring teams to plan weeks in advance or rush to escalate.
Today, we’re announcing provisioned limits for Amazon Cognito, a capability that transforms how you manage authentication rate limits. This introduces a feature in the AWS Management Console for Amazon Cognito for on-demand capacity adjustments, working alongside the existing account-level maximum limits in AWS Service Quotas. Together, they give you self-service control over your authentication infrastructure so you can scale up for Black Friday (or similar sales events), scale down after tax season, and optimize costs with unprecedented precision. What once took up to 2 weeks now happens in minutes.
In this post, you’ll learn how provisioned limits work, the relationship between account-level maximums and provisioned capacity, the cost implications and optimization strategies, and step-by-step implementation guidance. This capability fundamentally changes how you approach authentication capacity planning.
Manual processes that can’t keep pace
Identity management services open the front door to your application. When users can’t sign in, everything else stops. Amazon Cognito offers extremely flexible limit management: customers can request adjustments as granular as 1 request per second (RPS) for as short as 1 day. As customer demand for faster, self-service adjustments grew, we identified opportunities to address the following challenges:
Support tickets required for each rate limit change
10–14 day approval timelines for standard review and processing
Advance planning needed weeks ahead of anticipated traffic spikes
For businesses with seasonal traffic, like tax preparation services that see 90% of annual authentication volume in March and April, or ecommerce platforms preparing for Black Friday, these factors meant teams had to plan capacity decisions well in advance with limited ability to adjust in the moment.
Provisioned limits and account-level max limits
Starting July 6, 2026, Amazon Cognito introduced provisioned limits in the Amazon Cognito console. At the account level (per AWS Region, per account), you’ll find a Provisioned limits tab next to the User Pools tab. This gives you direct control over your authentication rate limits through two complementary mechanisms:
Provisioned limits (Amazon Cognito console) – Adjust your provisioned capacity up or down on-demand. Changes take effect immediately. You’re billed for the capacity you provision above the default limit, regardless of how much you use.
Account-level max limit (Service Quotas console) – Set your account’s ceiling, the maximum RPS your account is allowed to provision. Raising this ceiling doesn’t incur additional charges. Approximately 90% of requests are automatically approved within minutes. For larger limit increases (depending on the API category and Region), manual approval through AWS Support might still be required.
The following experience shows the end-to-end workflow of adjusting your provisioned limits and requesting a higher account-level max.
Figure 1: Provisioned limit experience workflow
How they work together
Let’s use the UserCreation API as an example. The default limit is 50 RPS, and initially the provisioned limit is also 50 RPS—meaning billed capacity is 0 (no additional charge). The applied account-level max limit is also 50 RPS. So you have three values: default (50), provisioned limit (50), and account-level max (50).
Start by going to the Amazon Cognito console and choosing User pools from the navigation pane.
Figure 2: UserCreation with default values
Choose Edit provisioned limit, to go to the Edit provisioned limit page with an input field for New provisioned limit. However, because the account-level max limit is 50 RPS, you can’t set the provisioned limit above 50. For example, if you want to provision 55 RPS, the console won’t allow it because 55 exceeds the current account max of 50.
Figure 3: Editing the provisioned limit constraint constraint to more than the account-level max
To set a higher limit, choose Request an increase. This takes you to the Service Quotas console, where you can choose Request increase at account level to request a higher account-level max, for example, 55 RPS. Most requests are automatically approved within minutes. At any time, you can check the status of the request using the Request history tab.
Figure 4: Service Quotas page where the account-level max increase is requested and auto-approved
After receiving approval, return to the Amazon Cognito console to edit your provisioned limit up to 55 RPS. Your billed capacity becomes 5 RPS (55 minus the 50 default).
Figure 5: Provisioned after increasing the provisioned limit to 55 RPS, with billed capacity of 5 RPS.
This two-part model gives you precise control over both cost and capacity. Raising the account-level max in Service Quotas doesn’t incur additional charges—it only sets your ceiling. You are billed for what you provision in the Amazon Cognito console above the default, so you benefit from right-sizing your provisioned capacity to match expected demand. Raise your account max ahead of time to prepare for future scaling needs without incurring any cost. When the time comes, increase your provisioned limit to what you need, and scale back down after the event. You stop being charged for the extra capacity as soon as you reduce the provisioned limit. This applies equally to seasonal spikes, planned load tests, or unexpected viral growth—all self-service, all within minutes.
How the provisioned limits experience works
The provisioned limits experience introduces three key concepts that work together:
Default limit – The baseline rate included at no additional cost (for example, 50 RPS for UserCreation).
Provisioned limit – The capacity you actively request and reserve in the Amazon Cognito console. Because this capacity is reserved specifically for your account, it’s the chargeable dimension. You’re billed for any provisioned capacity above the default, regardless of how much you consume. For example, if the default is 50 RPS and you provision 80 RPS, you’re billed for 30 RPS even if your actual usage is only 60 RPS. If your provisioned capacity is 50 RPS (the default), your billed capacity is 0.
Applied account-level max limit – The ceiling managed through Service Quotas. This determines how high you can set your provisioned limit. Raising this ceiling doesn’t incur charges, it only unlocks the ability to provision higher capacity. Importantly, the Provisioned limits page displays each API category with its adjustability status. For example, UserCreation is marked Adjustable (shown earlier in Figure 5) and can be modified. However, UserList is marked Not adjustable, meaning the account-level max limit can’t be adjusted for that category. You can still see its default and provisioned limit on the page, but you can’t modify them. For adjustable categories, you will see the default limit, current provisioned limit, and billed capacity at a glance.
Figure 6: Provisioned limits overview with non-adjustable API categories.
Multi-tenant SaaS considerations
For software as a service (SaaS) providers managing multiple tenants with varying throughput requirements, the UpdateProvisionedLimit API enables programmatic management of provisioned capacity. Teams using dedicated user pools per tenant, for example, can integrate this into their infrastructure-as-code pipelines to adjust provisioned limits per tenant tier. With provisioned limits, SaaS vendors can tier their capacity management per tenant, for example, provisioning higher capacity for enterprise-tier tenants and lower capacity for free-tier tenants, and adjust each tenant’s provisioned capacity independently through the API based on their service tier and demand patterns.
Conclusion
With provisioned limits, whether you’re preparing for peak shopping season, tax filing deadlines, or any other scaling event, you can now adjust provisioned limits to respond to your organization’s needs on demand. The separation between the account-level max (in Service Quotas) and the provisioned limit (in the Amazon Cognito console) gives you full control to plan ahead, respond to demand changes in minutes, and optimize costs on your own terms.
Get started
The provisioned limits experience for Amazon Cognito user pools launched on July 6, 2026, and is available across all AWS Regions where Amazon Cognito is supported. To get started:
Review your current authentication traffic patterns using Amazon CloudWatch metrics to understand your baseline
Set up CloudWatch alarms at 70% and 85% of your current rate limits
Ensure your team has appropriate AWS Identity and Access Management ( IAM) permissions for both Service Quotas and the Provisioned limits tab in the Amazon Cognito console
Raise your account-level max in Service Quotas based on your demand expectations
Use the Amazon Cognito console Provisioned limits tab to adjust capacity up or down as needed
Organizations adopt Infrastructure as Code (IaC) to manage cloud environments reliably, repeatably, and at scale. As teams grow and infrastructure complexity increases, IaC becomes the backbone of consistent deployments, compliance enforcement, and operational agility. The developer’s experience around IaC, however, remains fragmented — engineers routinely context-switch between documentation portals, linting tools, deployment consoles, and logging systems just to complete a single deploy cycle. This friction compounds across teams: slower iteration means delayed feature releases, longer incident recovery times, and increased operational risk. When a deployment fails, diagnosing the root cause across disconnected interfaces can take longer than writing the template itself — turning a feedback loop that could take hours of manual investigation into a more streamlined process.
The AWS Infrastructure as Code (IaC) MCP Server brings AWS CloudFormation documentation search, template validation, and deployment troubleshooting into your AI assistant, so you can move through a full AWS CloudFormation development cycle without leaving the chat interface. Developing AWS CloudFormation templates often means switching between documentation pages, linters, the deployment console, and AWS CloudTrail Logs. Each context switch adds friction to the inner development loop — the tight cycle of writing, validating, deploying, and fixing infrastructure code. This fragmented workflow increases time-to-deployment, delays feedback, and reduces developer productivity, particularly for teams managing complex, multi-resource stacks at scale.
The AWS Infrastructure as Code (IaC) Model Context Protocol (MCP) Server unifies these capabilities in one place. This post demonstrates how the IaC MCP Server tools work together in a real workflow — from authoring and validation through deployment and runtime troubleshooting — all within a single AI assistant conversation.
The walkthrough follows four steps that map to IaC MCP Server tools:
Author: Search CloudFormation documentation and generate a template
Validate: Check syntax with cfn-lint and compliance with cfn-guard
Deploy: Deploy the stack using a CloudFormation service role
Troubleshoot: Diagnose a deployment failure using CloudTrail correlation
Figure 1 shows the four-step workflow. Steps 1, 2, and 4 run inside the IaC MCP Server, while Step 3 uses the AWS CLI directly.
Figure 1. End-to-end CloudFormation workflow with the IaC MCP Server
In the prerequisites, you deploy a CloudFormation service role stack that deliberately omits the iam:PassRole permission. During the walkthrough, you use the AI assistant to generate and deploy an application stack. When CloudFormation tries to assign the Lambda execution role, the deployment fails with AccessDenied. The troubleshoot tool then correlates stack events with CloudTrail to pinpoint the root cause.
This role grants CloudFormation permission to create S3 buckets, Lambda functions, and CloudWatch Logs log groups, but deliberately omits iam:PassRole — you’ll diagnose this gap in Step 4.
You use the --capabilities CAPABILITY_NAMED_IAM flag to acknowledge that the stack creates IAM resources with custom names.
We provide this role template for demonstration purposes only and do not intend it for production use.
Note the role ARN from the stack outputs. You must use this ARN in Step 3:
The four steps that follow map to IaC MCP Server tools: authoring with documentation search, validating with cfn-lint and cfn-guard, deploying with a CloudFormation service role, and troubleshooting with CloudTrail correlation.
Step 1: Generate a CloudFormation template
Start by asking your AI assistant to search CloudFormation documentation and generate a template. The IaC MCP Server calls the search_cloudformation_documentation tool behind the scenes to retrieve up-to-date resource property references.
Prompt:
Create a CloudFormation template with an S3 bucket, a Lambda function (Python 3.13 runtime, inline hello-world code), an IAM execution role for the function, and a CloudWatch Logs log group. Include common security configurations. Save it as iac-mcp-blog-app-stack.yaml in the current directory.
The AI assistant calls the search_cloudformation_documentation tool to look up resource properties for AWS::S3::Bucket, AWS::Lambda::Function, AWS::IAM::Role, and AWS::Logs::LogGroup. You can see the tool invocations in Kiro’s chat interface. The search results include up-to-date property references and example configurations, which the AI assistant uses to generate a template.
The generated template should include resources similar to the following (your output may vary):
An S3 bucket with versioning, encryption, and public access block
A Lambda function with inline Python code
An IAM role with a least-privilege policy for CloudWatch Logs
A log group with a retention policy
The following snippet shows the key resources. Your AI assistant’s output may differ in naming or structure, but the core configuration should be similar:
Before deploying, ask the AI assistant to validate the template. The IaC MCP Server provides two validation tools that wrap open source checkers: cfn-lint for syntax validation and cfn-guard for policy-as-code compliance checks.
Prompt:
Validate iac-mcp-blog-app-stack.yaml for syntax errors and compliance violations.
The AI assistant runs two checks:
Syntax validation (validate_cloudformation_template): Uses cfn-lint to catch structural errors, invalid property names, and schema violations.
Compliance check (check_cloudformation_template_compliance): Uses cfn-guard to evaluate the template against security rules such as S3 bucket encryption, public access block settings, and log group retention.
If either check reports issues, ask the AI assistant to fix them. Continue iterating until both checks pass.
Note that the compliance check might flag violations related to S3 object lock, access logging, replication, and inline IAM policies. For a production workload, you would address each of these issues. In this walkthrough, the AI assistant resolves them to demonstrate the iterative validate-and-fix workflow. Your results might vary depending on the template the AI assistant generated in Step 1.
After the AI assistant resolves the violations, the S3 bucket resource gains access logging and object lock properties. The following snippet shows the typical shape of these additions (see iac-mcp-blog-app-stack-fixed.yaml in the companion repository for the complete hardened template):
Your template now passes both cfn-lint and cfn-guard checks. These security improvements improve your template’s security posture but are unrelated to the deployment failure you’ll encounter next. The failure in Step 3 is caused by missing permission on the service role, not by anything in the template itself.
Step 3: Deploy the stack
With validation complete, deploy the template. This deployment will fail — not because of a template error, but because the CloudFormation service role deployed in the prerequisites is missing iam:PassRole. This is the scenario you’ll diagnose in Step 4.
Now deploy the validated template using the service role you created in the prerequisites:
Prompt:
Deploy iac-mcp-blog-app-stack.yaml as a stack named “iac-mcp-blog-app-stack” in us-east-1 using the service role ARN from iac-mcp-blog-role-stack.
The AI assistant runs the AWS CLI deployment command for you. If your AI assistant doesn’t support running shell commands directly, you can deploy manually with the AWS CLI:
The deployment fails. The stack event shows an AccessDenied error on the IAM role resource, but doesn’t identify which permission on the CloudFormation service role is missing or why. At this point, we move from static analysis to runtime troubleshooting.
Step 4: Troubleshoot the failure
Ask the AI assistant to diagnose the failure:
Note: CloudTrail events typically take 5–15 minutes to appear. Wait at least 5 minutes after the deployment failure before running the troubleshoot tool for the most complete analysis.
Prompt:
Troubleshoot the failed deployment of iac-mcp-blog-app-stack in us-east-1.
The AI assistant calls troubleshoot_cloudformation_deployment, which:
Retrieves the stack events and identifies the failed resources
Correlates the failure timestamps with CloudTrail API calls
Identifies AccessDenied errors and the missing permissions that caused them
The troubleshoot tool identifies that the CloudFormation service role is missing iam:PassRole — the permission required to assign the Lambda execution role to the function. If your template includes the cfn-guard hardening from Step 2 (access logging, object lock), the tool may also surface additional missing S3 permissions such as s3:PutBucketObjectLockConfiguration for the logging bucket.
Prompt:
Fix iac-mcp-blog-role-stack.yaml to add the missing permissions identified by the troubleshoot tool. Save it as iac-mcp-blog-role-stack-fixed.yaml.
The AI assistant adds the missing permissions to the service role template. Now ask the AI assistant to deploy the fix, delete the failed stack, and redeploy:
Prompt:
Deploy iac-mcp-blog-role-stack-fixed.yaml to update iac-mcp-blog-role-stack, then delete the failed iac-mcp-blog-app-stack and redeploy it with the same service role.
The AI assistant runs the necessary CLI commands: updating the role stack, deleting the failed application stack, and redeploying the application stack. The failed stack is in ROLLBACK_COMPLETE state, a terminal state that CloudFormation cannot update in place, so you must delete it before redeploying.
The stack deployment succeeded.
Cost considerations
For information about costs associated with the resources in this walkthrough, including S3 storage, Lambda invocations, CloudWatch Logs, and CloudFormation operations, see AWS Pricing. Confirm that your account usage falls within any applicable free tier limits. If you enabled S3 access logging or object lock through the validation-and-fix workflow in Step 2, the logging bucket stores a small amount of access log data that falls under S3 standard pricing. See AWS Pricing for current rates and confirm that your account is within the Free Tier limits before you deploy.
Cleaning up
To avoid ongoing charges, delete both stacks.
Option A: Clean up with your AI assistant
Ask your AI assistant to run the cleanup for you. The IaC MCP Server lets the AI assistant inspect stack outputs, empty buckets, and delete both stacks in the correct order:
Clean up the iac-mcp-blog-app-stack and iac-mcp-blog-role-stack stacks in us-east-1. Empty any S3 buckets they created (including access log buckets) before deleting the application stack, then delete the role stack.
Option B: Clean up manually
Delete the application stack first because it was deployed with the service role:
Warning: If your template included access logging, the logging bucket may contain objects. CloudFormation cannot delete a non-empty bucket. Empty it first:
If any S3 bucket was created with DeletionPolicy: Retain or still contains objects (for example, server access logs), CloudFormation leaves it in place. Empty and delete those buckets from the S3 console or with aws s3 rb s3://<bucket-name> --force.
Next steps
If you manage CloudFormation infrastructure and find yourself losing time to context-switching between docs, linters, consoles, and logs, here’s how to streamline your workflow starting today:
Set up the IaC MCP Server — Install and configure the IaC MCP Server with an MCP-compatible AI assistant such as Kiro to bring documentation search, validation, and troubleshooting into a single conversational interface.
Run the walkthrough end-to-end — Clone the companion repository and follow this post step by step to experience the full author-validate-deploy-troubleshoot loop in your own AWS account.
Integrate into your team’s workflow — Replace manual context-switching by embedding the IaC MCP Server’s tools into your day-to-day CloudFormation development process, reducing iteration time from hours to minutes.
Extend to AWS CDK — Apply the same conversational workflow to CDK-based infrastructure using the IaC MCP Server’s CDK capabilities described in the introductory blog post.
Contribute and share feedback — Report issues or suggest enhancements on the AWS MCP GitHub repository to help shape future capabilities.
Conclusion
In this walkthrough, you used the IaC MCP Server to move through a complete CloudFormation development cycle without leaving your AI assistant. The documentation search tool retrieved up-to-date resource property references that the AI assistant used to generate a template. The validation tools caught syntax errors and compliance gaps before deployment. When the deployment failed due to missing permissions on the service role (an issue that static analysis cannot detect), you used the troubleshoot tool to correlate stack events with CloudTrail and pinpoint the root cause in seconds.
By combining static validation with runtime diagnostics, you shorten your develop-validate-fix cycle for CloudFormation. Instead of switching between browser tabs, CLI sessions, and the CloudTrail console, you stay in one interface — turning a multi-step troubleshooting session that previously meant switching between consoles, CLI sessions, and CloudTrail into a few prompts in a single conversation.
Shuto Yukawa is an Associate Delivery Consultant at AWS Professional Services. He helps customers modernize their applications and adopt cloud-native practices on AWS.
G SS Harsha Vardhan is an Associate Delivery Consultant at AWS Professional Services. He guides customers to migrate and transform their workloads to AWS, driving modernization across people, process, and technology.
Amazon Managed Streaming for Apache Kafka (Amazon MSK) manages core operational tasks for running Apache Kafka, including cluster provisioning, patching, high availability, and more. But operating Kafka clusters at scale still involves decisions that benefit from deep domain knowledge. For example, where do I start investigating application latency? How do I right-size a cluster to balance performance and cost? How do I analyze my applications, cluster configurations, and other requirements to support a smooth migration from self-managed Kafka to Amazon MSK?
With the new Agent Skills for Amazon MSK, you can access AI-assisted guidance for operations and migration planning directly in your development environment. Two complementary skills, managing-amazon-msk and migrate-to-msk, encode domain expertise based on AWS best practices, structured troubleshooting workflows, and programmatic sizing and compatibility analysis.
In this post, we walk through installing both skills and demonstrate their key capabilities. These include diagnosing a performance issue, sizing a cluster with cost breakdowns, and migration planning from self-managed Kafka to Amazon MSK including discovery, compatibility assessment, and target sizing.
How Agent Skills enhance documentation
Baseline large language models encode knowledge from their training data. That data can go stale as services evolve, and it often lacks the specific, contextual detail a task needs. As a result, a general-purpose assistant can produce answers that sound convincing but are factually wrong (hallucinations). For example, Amazon MSK Provisioned clusters come in two broker types, Standard and Express. Both broker types include their own considerations to achieve your performance, latency, availability, and durability requirements. Because training data mixes the two together, general-purpose assistants routinely conflate them and apply advice to the incorrect broker type.
These skills solve this problem by encoding the correct context for Amazon MSK broker operations, performance management, client configuration, and migrations, aligned with AWS best practices. This helps agents give more accurate, contextual guidance.
Overview of solution
The two Amazon MSK Agent Skills cover the full lifecycle of Amazon MSK cluster ownership:
Skill 1: managing-amazon-msk
Operations expertise for Amazon MSK Provisioned clusters with both Standard and Express broker types:
Workflow
What it does
Performance troubleshooting
Structured decision tree: CPU saturation, batch size analysis, Amazon Elastic Block Store (Amazon EBS) throughput entitlements (Standard), Express brokers entitlements
Consumer lag diagnosis
Determines if lag is broker-side, partition-level (hot keys), or client-side. Provides targeted fixes
Programmatic right-sizing and cost estimate tool comparing all Standard and Express instance types with cost breakdowns
Monitoring and alarms
Set up actionable Amazon CloudWatch alarms with broker-type-aware thresholds that follow best practices for monitoring
Maintenance operations
Rolling restart impact analysis, patching and broker upgrades, version upgrade planning, and transient failure analysis (distinguishing expected maintenance disruptions from real issues).
Skill 2: migrate-to-msk
Migration planning from self-managed Apache Kafka to Amazon MSK in three phases:
Phase
What it does
Discovery
Inventories your source cluster from infrastructure as code (IaC) files, Kafka CLI output, or manual input. Produces a standardized cluster-config.json
Assessment
Five-pillar compatibility check (topology, version, configs, auth, quotas) plus target cluster sizing using the AWS-published Amazon MSK Sizing and Pricing workbook
Simulation (Optional)
Deploys temporary Amazon MSK cluster and Amazon EC2 load-generation fleet in your account to test performance under synthetic load before you migrate. Produces an Amazon CloudWatch dashboard with throughput, broker health, latency, and consumer lag metrics.
After assessment, the skill provides guidance on using Amazon MSK Replicator for the actual data migration to your new Amazon MSK cluster.
Permissions to describe and manage Amazon MSK clusters, retrieve Amazon CloudWatch metrics for performance diagnostics, and create and delete CloudWatch alarms.
For migrate-to-msk:
Optional read-only access (CloudWatch metrics, describe clusters) to gather runtime metrics from an existing AWS estate for a more accurate assessment.
The optional Simulation phase requires permissions to create AWS CloudFormation stacks.
Verify MCP installation by checking the MCP server status in your IDE’s MCP panel.
Verify skill installation with:
aws agent-toolkit list-installed-skills
You should see both skills listed for your detected agents. To confirm they’re active, ask your AI assistant an Amazon MSK question, and it should load the skill to engage with broker-type-aware guidance.
Scenario 1: Diagnosing high latency
During your evaluation of Amazon MSK your team notices elevated produce latency. You ask the AI assistant for help,
“Our Amazon MSK Express broker cluster is experiencing high produce latency that we think is related to our client application. The producer code is in this working directory. Can you help diagnose?”
The agent immediately identifies that this question would be well suited for the managing-amazon-msk-skill and activates it. In the same step, the agent opens your producer code to diagnose the real client configuration. The skill ships with reference guides, and the agent selects the two that matter for this specific problem. It then maps your application code directly onto the skill’s diagnostic workflow, landing on a diagnosis:
The skill identifies three compounding anti-patterns in the configuration, specifically linger.ms=0, an undersized batch.size, and compression.type=none. It then explains why they negatively impact Kafka cluster performance: every tiny message becomes its own produce request, saturating broker request-handler threads. Based on these observations, the skill delivers a targeted solution:
The skill uses best practice client-configuration references to provide specific recommendations to improve your application. It then goes on to provide additional context, considerations, and the Amazon CloudWatch metrics to observe to verify that the configurations have improved your end-to-end performance.
You can try this yourself by bringing your own producer code and letting the skill diagnose it. If you give it access to the AWS CLI the agent can pull live Amazon CloudWatch metrics from your actual cluster. This lets it correlate broker-side signals with what it sees in your client configuration for a more complete diagnosis.
Scenario 2: Migrating to Amazon MSK Express brokers
The migrate-to-msk skill guides you through a structured migration from self-managed Apache Kafka to Amazon MSK in three phases: discovery, assessment, and optional simulation. When you prompt the skill, it launches the discovery phase.
Phase 1: Discovery — analyze your source cluster
In this scenario, you point the skill at your infrastructure as code (IaC) files describing a self-managed Kafka deployment:
“Here’s our Kafka infrastructure, can you help us plan a migration to Amazon MSK Express brokers?”
The skill pulls static details: broker topology, versions, security configuration, and topic definitions directly from your IaC files.
For runtime values the skill can’t derive from IaC, such as actual peak throughput or consumer-group count, the skill identifies these as flagged gaps. For each gap, the skill provides the specific Kafka CLI commands you can run against your live cluster to capture those values.
The skill supports discovery from multiple source types: Terraform, CDK, CloudFormation, Docker Compose, Kubernetes manifests, or manual input in conversation.
Phase 2: Assessment — validate compatibility and size the target
With discovery complete, the assessment phase runs two independent analyses against your current cluster infrastructure.
Compatibility assessment evaluates your source cluster across five pillars:
Pillar
What it checks
Topology
AZ count, broker count, KRaft or ZooKeeper
Kafka version
Source version against Amazon MSK supported set (3.6, 3.8, 3.9)
Configs
Broker and topic configs against Amazon MSK’s editable/enforced/range-restricted sets
Auth
Authentication mechanism compatibility
Quotas
Peak workload against Amazon MSK per-broker ceilings
Each pillar produces one of the following finding types:
Verdict
Meaning
INFO
Already aligns with Amazon MSK. No action needed.
ADVISORY
Amazon MSK handles this differently, but migration can proceed. Review so the behavior change is expected.
ACTION_REQUIRED
Amazon MSK will not accept this in its current form. Remediation recommended.
Target sizing uses your current cluster’s usage metrics to perform right-sizing for Amazon MSK, including instance type, broker count, and projected monthly cost for your workload. This gives you a shareable artifact to use for sizing against different inputs and assumptions.
Next, you ask the skill to run the assessment:
“Assess my cluster for Amazon MSK Express broker compatibility and size the target”:
The skill runs both analyses against your cluster configuration. It outputs a compatibility report, sizing inputs, and sizing outputs, giving you a complete picture of what needs attention before migration and what your target cluster should look like.
Once you’ve validated compatibility and provisioned your Amazon MSK Express brokers, Amazon MSK Replicator handles the actual data migration. Amazon MSK Replicator is the native AWS solution for replicating data between Amazon MSK Provisioned clusters. For migrations, it supports replication of data from self-managed Apache Kafka clusters (including on-premises, self-hosted on AWS, or other cloud providers) to Amazon MSK Provisioned clusters.
Phase 3: Simulation (optional) — validate performance before cutover
With assessment complete, you can optionally ask the skill to guide you through setting up a live test environment:
“Can we run a simulation to see how Amazon MSK Express brokers handle our workload before we commit to migrating?”
The skill walks you through deploying temporary Amazon MSK Express brokers and EC2 client fleet in your own AWS account. These are sized from your Phase 2 workbook or numbers you provide, so that you can see real performance on your actual workload rather than relying on estimates. It confirms the target account and permission before deploying any billable resources.
Once the cluster is up, you choose a provided test (end-to-end latency or broker restart under load), and the skill runs it. It then surfaces metrics related to throughput, broker health, latency, and consumer lag on a CloudWatch dashboard. When you’re done, the skill helps you tear the stack down so you stop incurring cost.
Scenario 3: Sizing a cluster with cost breakdowns
You’re planning a new streaming workload and need to determine the right configuration:
“Size an Amazon MSK cluster for 200 MiB/s peak ingress, 600 MiB/s peak egress (3 consumer groups), 1,500 partition replicas, 168 hours retention. Compare Standard and Express.”
The skill’s programmatic sizing calculator evaluates your workload against every available instance type simultaneously, sizing across four constraints: ingress capacity, egress capacity, partition limits, and storage volume. Each is rounded up to a multiple of 3 Availability Zones (AZs).
When you ask the skill to size a cluster, it uses its sizing script to identify and recommend the least expensive viable option per broker class, and to break down the cluster cost across various sizing dimensions.
The calculator accounts for factors that manual sizing often misses, such as replication overhead on EBS, network bandwidth, and cross-AZ data transfer costs. The skill flags exactly which constraint is the bottleneck for each instance type, so you understand why a particular broker count was chosen.
Security considerations
Both skills recommend Transport Layer Security (TLS) encryption and IAM authentication. Discovery and assessment outputs contain broker addresses and configuration details. Treat them as sensitive and avoid sharing them in public channels without redaction. The migration artifacts do not store passwords or secrets.
Cleaning up
If you ran the optional Simulation phase with the migrate-to-msk skill, it deployed real resources in your AWS account, including an Amazon MSK Express cluster and an EC2 load-generation fleet, that continue to incur charges until you delete them. Ask the skill to tear down the simulation, or delete its CloudFormation stack yourself, to stop incurring cost. Only one simulation can exist per account at a time.
Migration artifacts (migrate-to-msk-skill-artifacts/) are local files that you can delete at your discretion.
Conclusion
Traditionally, Kafka administrators have relied on web-based UIs and dashboards for cluster health management and troubleshooting. With these skills, you can accelerate agent workflows that integrate directly into development environments and DevOps processes. Amazon MSK aims to expand this Agent Skills portfolio with additional tools and capabilities, so customers can build more sophisticated agentic DevOps workflows for their streaming infrastructure.
The Amazon MSK Agent Skills bring structured, broker-type-aware expertise to operating and migrating Amazon MSK clusters. Instead of searching through documentation to determine whether a metric applies to Standard or Express, or manually cross-referencing compatibility matrices for a migration, you get targeted guidance that routes to the correct path based on your cluster’s actual configuration.
Get started by installing both skills from the Agent Toolkit for AWS on the GitHub website into your development environment. Then try a prompt like:
“Size Amazon MSK Express brokers for 100 MiB/s ingress with 3 consumer groups and 72-hour retention”
or
“My Amazon MSK Express brokers have high produce latency. Help me diagnose”
The skills support you at any stage in the cluster lifecycle.
Debugging a failed Apache Spark application on Amazon EMR on EKS often means correlating signals from several places at once. These signals include Spark driver and executor pod logs, Spark event logs, and container termination signals that surface as pod exit codes rather than clear Spark errors. For example, a single out-of-memory failure can appear as a Kubernetes exit code 137 with no obvious link back to the line of code or configuration that caused it. This cross-system investigation can extend a single incident’s mean-time-to-resolution (MTTR) to days and requires deep Spark and Kubernetes expertise.
We recently announced Amazon EMR on EKS now supports Apache Spark troubleshooting agent extending the Apache Spark troubleshooting agent to support Amazon EMR on EKS. The agent already helps data engineers diagnose Spark failures on Amazon EMR on EC2, Amazon EMR Serverless, and AWS Glue using natural language prompts. With this launch, you can now point the same workflow at a failed Amazon EMR on EKS job run. From a single natural language prompt, the agent automatically retrieves your Spark logs from Amazon Simple Storage Service (Amazon S3) or Amazon CloudWatch (depending on your job’s logging configuration) along with Spark History Server Event log data, identifies the root cause, and recommends a fix when the failure is code-related. This can help reduce incident MTTR from days to minutes. Amazon EMR on EKS customers can use the agent at no additional cost. You only pay for your existing Amazon EMR on EKS resources.
In this post, we show you how to set up the agent for Amazon EMR on EKS and walk through troubleshooting a failed job run. We demonstrate the workflow from both the Amazon EMR console and an AI assistant that supports the Model Context Protocol (MCP), an open standard for connecting AI assistants to external tools and data.
How the troubleshooting agent works on Amazon EMR on EKS
The troubleshooting agent exposes a single interface to diagnose failed Spark applications across Amazon EMR on EKS, Amazon EMR on EC2, Amazon EMR Serverless, AWS Glue, and Amazon SageMaker notebooks. Instead of navigating different consoles, APIs, and log locations for each service, you describe your failed job in natural language, and the agent handles the rest. You can reach the agent from the Amazon EMR console or from MCP-compatible AI assistants, such as Kiro CLI, Kiro IDE, or Claude Code. We walk through both later in this post.
The troubleshooting agent runs as a fully managed MCP server, so you do not need to deploy or maintain a local MCP server. It uses a single-tenant design to keep your application data and code isolated. Operations are read-only and governed by AWS Identity and Access Management (IAM) permissions. The agent can only access the resources and actions your IAM role grants. Tool calls are automatically logged to AWS CloudTrail for complete auditability.
What’s specific to Amazon EMR on EKS is how the agent gathers its inputs. On Amazon EMR on EKS, your Spark driver and executor logs can be delivered to Amazon S3, Amazon CloudWatch Logs, or both, depending on your job’s monitoring configuration. The agent handles both sources automatically:
Driver and executor pod logs in Amazon S3 – When your job is configured with S3 monitoring, the agent reads the Spark event logs and the per-container stderr/stdout logs from your S3 log location, including discovering executor pod logs.
Driver and executor container logs in Amazon CloudWatch – When your job is configured with CloudWatch monitoring, the agent reads the driver and executor container log streams directly from your CloudWatch log group.
Spark History Server (SHS) data through the Amazon EMR Persistent UI – For the richer SHS signals (query plans, executor timelines, stage metrics, and configurations), the agent connects to the Amazon EMR Persistent UI for your job run, the same mechanism used for Amazon EMR on EC2.
Drawing on years of AWS experience running millions of Spark applications at scale, the agent extracts the relevant features and signals from these sources, work that would otherwise require manual correlation across Amazon S3, Amazon CloudWatch, and the Spark UI. It then uses a large language model on Amazon Bedrock, grounded in a managed knowledge base of Spark and AWS troubleshooting expertise through Retrieval Augmented Generation (RAG), to produce a root cause analysis and, when the failure is code-related, a code recommendation.
The large language model (LLM), the knowledge base, and the retrieval that connects them are fully managed as part of the agent. There’s nothing for you to provision, host, or tune. This managed inference is provided at no additional cost for Amazon EMR on EKS. You pay only for the AWS resources you already use to run your Spark applications and to validate recommended changes.
Getting started
You can use the agent from either the Amazon EMR console or an MCP client. Both rely on setting up a single IAM role. The following sections walk through creating that role and then troubleshooting a failed job run with each method.
Set up IAM permissions
The IAM role grants the agent read access to the diagnostic sources it analyzes, such as your Amazon EMR on EKS job runs, the Amazon EMR Persistent UI, and your Spark logs in Amazon S3 and Amazon CloudWatch. Creating this role is the only setup required for the console experience. The MCP client path has a few additional prerequisites, covered later in the section on troubleshooting from an MCP client.
The agent uses your IAM role to authorize operations at the AWS service level. It can only access what your role allows. Create a role your account can assume, then attach a policy granting the permissions the agent needs for Amazon EMR on EKS.
First, set some variables for the commands that follow. ACCOUNT_ID is derived from your configured credentials. Set REGION to the AWS Region where you run your Amazon EMR on EKS workloads:
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
REGION=us-east-2 # replace with your Region
Create a trust policy that allows your account to assume the role, and create the role:
Create and attach a policy granting the agent read access to your Amazon EMR on EKS job runs, the Amazon EMR Persistent UI, and your S3 and CloudWatch logs. Replace amzn-s3-demo-logging-bucket with the name of your logging bucket and replace my_log_group_name and my_log_stream_prefix with your CloudWatch log group name and log stream prefix, respectively.
Note: If you prefer an automated setup, an AWS CloudFormation template that creates this role with the required permissions is available in the setup documentation. The previous CLI steps give you the same result with finer control over each permission.
Troubleshooting a failed Amazon EMR on EKS job run
You can reach the troubleshooting agent two ways: directly from the Amazon EMR console, or from an MCP-compatible AI assistant such as Claude Code. We walk through both, using two different failures to show the range of what the agent diagnoses.
Option 1: Troubleshoot from the Amazon EMR console
The console offers the fastest path. Once you’ve created the IAM role in the Set up IAM permissions section, no additional setup is required. Here we troubleshoot a job that failed with a driver out-of-memory error. The application generates a large dataset and calls collect() to pull it back to the driver, exceeding the configured spark.driver.maxResultSize of 512 MiB.
Open the Amazon EMR console, choose Virtual clusters (under Amazon EMR on EKS), and select the virtual cluster that ran your job.
In the Jobs list, find your failed job run and choose its Failed status. This opens a popover with a Troubleshoot with AI button.
Choose Troubleshoot with AI. The agent analyzes the job and returns its findings directly on the console, namely the analysis insights, a root cause, and a recommendation. For this job, it identifies that the collect() operation on line 24 attempts to materialize the full result set on the driver, exceeding the spark.driver.maxResultSize safety limit. This fails the job before an actual driver out-of-memory crash. Because the failure stems from the application code, the agent also returns a code recommendation: a before-and-after diff that replaces the collect() call with a distributed write to the destination path. Executors then persist their partitions in parallel instead of funneling the data through the driver.
Option 2: Troubleshoot from an MCP client (Claude Code)
You can also use the agent from MCP-compatible AI assistants. This option requires a one-time setup to connect the assistant to the agent’s MCP servers, and it unlocks a conversational workflow where the agent chains from analysis into a concrete code fix. In this walkthrough, we use Claude Code.
Prerequisites
In addition to the IAM role from the Set up IAM permissions section, the MCP client path requires:
Python 3.10 or higher.
The uv package manager. For instructions, see Installing uv.
Claude Code installed. For instructions, see Install Claude Code. You can also use another MCP-compatible AI assistant such as Kiro CLI or Kiro IDE.
Configure an AWS CLI profile
Configure a profile that assumes the IAM role you created, so the MCP servers call AWS with the agent’s permissions:
export IAM_ROLE=arn:aws:iam::${ACCOUNT_ID}:role/SparkTroubleshootingMCPRole
export SMUS_MCP_REGION=${REGION}
aws configure set profile.smus-mcp-profile.role_arn ${IAM_ROLE}
aws configure set profile.smus-mcp-profile.source_profile default
aws configure set profile.smus-mcp-profile.region ${SMUS_MCP_REGION}
Add the MCP servers
The troubleshooting agent provides two tools through two MCP servers: analyze_spark_workload (workload analysis and root cause) and spark_code_recommendation (code fixes). Add both to your assistant.
Verify your setup by running the /mcp command in Claude Code to confirm the sagemaker-unified-studio-mcp-troubleshooting and sagemaker-unified-studio-mcp-code-rec servers are connected and their tools are available.
Verify with the /tools command in Kiro CLI to confirm the analyze_spark_workload and spark_code_recommendation tools are available.
Run the agent
For this walkthrough, we troubleshoot a different failure to show how the agent chains from analysis into a concrete code fix. The job is a small PySpark application that reads a CSV file into a DataFrame and registers it as a temporary view named people. It runs a Spark SQL query to uppercase the Name column before displaying the results. The job run failed because the query calls UPPERX, a function that doesn’t exist in Spark SQL (it’s a typo for the built-in UPPER).
From the Claude Code terminal (or MCP-compatible assistants), describe your failed job run in natural language, providing the virtual cluster ID and job run ID:
Debug my EMR on EKS job with job run id <jr-id> and virtual cluster id <vc-id> in <region>
The agent invokes the analyze_spark_workload tool, which automatically:
Calls the Amazon EMR on EKS API to retrieve your job run’s configuration and determine where its logs are stored.
Retrieves your Spark logs from Amazon S3 or Amazon CloudWatch, depending on your job’s logging configuration.
Connects to the Amazon EMR Persistent UI to extract Spark UI features such as the execution plan, stage metrics, and executor timelines.
Analyzes the correlated signals and returns a root cause explanation.
For this job, the agent returns:
Root cause: SQL function error. Your Spark SQL query references a function UPPERX that doesn’t exist in an available function catalog (system.builtin, system.session, or spark_catalog.default). Category: SQL_ERROR. The job failed because the function name can’t be resolved. UPPERX is almost certainly a typo for the built-in UPPER function.
Because the failure is code-related, the agent then chains into the spark_code_recommendation tool, which produces a concrete before-and-after fix:
df.createOrReplaceTempView("people")
- result = spark.sql("SELECT UPPERX(Name) FROM people")
+ result = spark.sql("SELECT UPPER(Name) FROM people")
result.show()
spark.stop()
The two tools work together. analyze_spark_workload identifies the root cause, and when the failure stems from the application code, spark_code_recommendation returns the exact edit to make. You review the recommendation and apply it with full control over the change. The agent only provides the analysis and recommendations.
Supported failure categories
The troubleshooting agent diagnoses a wide range of Apache Spark failures on Amazon EMR on EKS, including:
Out-of-memory and resource exhaustion – Driver and executor out-of-memory errors, including driver-side failures from operations like collect() and executor terminations that surface as Kubernetes pod exit codes (such as exit code 137).
Data skew and shuffle issues – Uneven partitioning and shuffle failures that concentrate work on a few executors.
Configuration errors – Misconfigured Spark settings that lead to failures or inefficiency.
Code-level issues – Problems such as incorrect API usage, unbounded collect() calls, and user-defined function (UDF) errors, for which the agent can recommend code fixes.
Code recommendations are supported for PySpark workloads on Amazon EMR on EKS, Amazon EMR on EC2, Amazon EMR Serverless, and AWS Glue.
Conclusion
With support for Amazon EMR on EKS, the Apache Spark troubleshooting agent gives platform and data engineering teams a shared workflow for investigating failed Spark applications. By bringing together Spark and Kubernetes diagnostic signals, the agent can reduce manual investigation and repeated handoffs between teams, helping engineers identify likely causes and corrective actions faster.
There’s no additional charge for using the troubleshooting agent, including the large language model used through Amazon Bedrock. You pay only for the AWS resources used to run your Spark applications and validate recommended changes.
The HIPAA Security Rule’s Technical Safeguards (§164.312) define five standards and nine implementation specifications covering access control, audit controls, integrity, authentication, and transmission security.
The guidance also covers the 2025 NPRM proposed changes,including encryption at rest and in transit becoming required, multi-factor authentication (MFA) becoming mandatory for all electronic Personal Health Information (ePHI) access, and new specifications for network segmentation, configuration management, anti-malware protection, patch management, software removal, incident response and breach notification.
Key topics included
Shared responsibility for HIPAA on AWS – A responsibility matrix mapping each §164.312 specification to what AWS manages nd what the customer must configure and operate.
ePHI boundary architecture – Guidance on establishing a defined ePHI boundary
ePHI data flow and encryption – A reference architecture tracing ePHI with the applicable §164.312 specification
Foundation checklist – Prerequisite recommendation before configuring individual Technical Safeguard controls.
This guidance is written for cloud architects, security engineers, CISOs, and compliance teams at covered entities and business associates building or operating AWS healthcare workloads. It assumes familiarity with AWS services and is intended as a practical implementation reference, not a legal or regulatory interpretation. This guidance focuses exclusively on Technical Safeguards.
HHS published a Notice of Proposed Rulemakingin January 2025, proposing significant updates to the HIPAA Security Rule—including eliminating the Addressable designation, making encryption, MFA, and asset inventory mandatory, and introducing new technical requirements not present in the current rule. As of June 2026, the final rule has not been published. This guidance covers both the current rule and the proposed changes and recommends treating all specifications as Required for new workloads.
For questions about HIPAA readiness on AWS, including Administrative Safeguards, Physical Safeguards, risk analysis, and assessment preparation, contact the AWS Security Assurance Services teamor your AWS account representative.
This guidance is provided by AWS Security Assurance Services, LLC, a HITRUST External Assessor Firm and PCI-QSAC along with contribution from AWS HCLS, AWS Compliance teams. It is for informational and guidance purposes only and does not constitute legal, regulatory, or compliance advice. Recipients are solely responsible for determining applicability to their specific environments and legal obligations.
If you have feedback about this post, submit comments in the Comments section below.
Amazon Inspector is an automated vulnerability management service that continually scans Amazon Web Services (AWS) workloads for software vulnerabilities. The vulnerability management capabilities of Amazon Inspector are powered by an asset inventory engine known as the Amazon Inspector SBOM Generator (inspector-sbomgen), a standalone command-line tool that produces a software bill of materials (SBOM) from container images, directories, archives, local systems, compiled binaries, and more. Over the past two years, we’ve expanded inspector-sbomgen’s coverage across dozens of programming language ecosystems, operating systems, and widely deployed applications.
We’re pleased to announce a new capability for builders using inspector-sbomgen: a plugin system for writing your own custom package collectors that you can use right away, without requiring source code compilation nor waiting for an official release.
In this post, we walk you through what the inspector-sbomgen plugin system does, why we built it, and how you can write your first plugin in a few minutes. Along the way, we also cover how plugin-generated package components integrate with Amazon Inspector for vulnerability scanning, and we explore the plugin safety model, which helps ensure security-hardened and predictable plugin behavior.
Why we built a plugin system
Software ecosystems are dynamic. New language package managers, lockfile formats, and end user applications ship constantly, and many are adopted quickly, in some cases with little security scrutiny. That leaves security teams with a visibility gap: production workloads running software that their SBOM tooling doesn’t yet recognize. Customers have asked us to inventory many of these ecosystems directly, and until recently, the only path to support was to open a feature request and wait for the inspector-sbomgen team to onboard the ecosystem and deploy a new release.
The inspector-sbomgen plugin system changes that. With plugins, you can:
Onboard ecosystems that inspector-sbomgen doesn’t support out of the box. New open source ecosystems, niche or fast-moving package formats, and internal or proprietary tooling can all be inventoried without modifying inspector-sbomgen.
Prototype detection for an ecosystem quickly. We designed a plugin system that is friendly to developers and AI coding assistants alike. Plugins are written in Lua, loaded at runtime, and require no Go toolchain nor compilation. You can use the built in test harness to iterate on a plugin and see results immediately.
Build on a stable foundation. The plugin API abstracts away artifact-type differences, so you write your detection logic once and it works seamlessly across container images, archives, local systems, and more. And because plugins stay decoupled from the internals of sbomgen, the core tool’s regression surface stays small.
Internally, we’ve used the plugin system to ship new ecosystem coverage faster than before. In our 1.13 release, more than 20 ecosystems that were previously implemented in Go, including Apache Tomcat, NGINX, MySQL, Redis, WordPress, and the OpenSSH toolchain, are now embedded as plugins inside the sbomgen binary. The same release also added more than ten brand-new ecosystems as plugins, including Apache Cassandra, Apache Struts, Conda, Swift packages, and AI-agent collectors (Amazon Q Developer, Kiro CLI, Claude Code, GitHub Copilot, and Ollama).
How inspector-sbomgen plugins work
Sbomgen plugins follow a two-step pipeline:
Discovery – Scan the artifact’s file system to identify files that contain installed package metadata.
Collection –Open each discovered file, parse file contents, and publish findings into the SBOM.
Under the hood, an event bus connects discovery and collection plugins. Discovery plugins publish events listing discovered files, and one or more collection plugins subscribe to these events, triggering package collection. Developers might recognize this behavior as the observer pattern.
This decoupling lets a single discovery plugin feed multiple collectors, for example, one extracting package metadata, another scanning for secrets, and another checking policy. Each collection plugin works from the same file list without re-walking the artifact filesystem, a computationally expensive operation.
Write your first plugin in 5 minutes
Inspector-sbomgen makes it straightforward to bootstrap a plugin environment. The plugin new command tells sbomgen to create a new plugin workspace, and the —-with-example flag populates the workspace with a discovery-collection plugin pair, that you can run immediately.
inspector-sbomgen plugin new --with-example
After invoking the preceding command, you will be prompted to provide a plugin name and a directory that will contain your plugin workspace. You can provide custom values or use the default values:
Plugin name (identifies the software ecosystem your plugin will inventory, e.g. debian-dpkg, rhel-rpm, python-pip, cmake) [my-custom-ecosystem]: <enter>
Project directory [my-sbomgen-plugins]: <enter>
Created plugin "my-custom-ecosystem" in my-sbomgen-plugins/
Note that you can skip interactive prompts by specifying the plugin name and directory using the corresponding command line interface (CLI) arguments:
After creating your plugin workspace, inspector-sbomgen will display a next steps screen, which guides developers and AI code assistants to the source files they need to change and to supporting documentation:
Next steps:
Get started:
1. Open plugin folder in a code editor (VS Code recommended)
2. Add test files that your plugin will discover and parse
(e.g., config files, lockfiles, binaries, etc.):
my-sbomgen-plugins/discovery/cross-platform/extra-ecosystems/my-custom-ecosystem/_testdata/
Develop:
3. Edit discovery: my-sbomgen-plugins/discovery/cross-platform/extra-ecosystems/my-custom-ecosystem/init.lua
4. Edit collection: my-sbomgen-plugins/collection/cross-platform/extra-ecosystems/my-custom-ecosystem/init.lua
Test:
5. Write unit tests: my-sbomgen-plugins/discovery/cross-platform/extra-ecosystems/my-custom-ecosystem/init_test.lua
6. Run unit tests: inspector-sbomgen plugin test --path my-sbomgen-plugins
Deploy:
7. Distribute your plugin directory wherever you run inspector-sbomgen:
inspector-sbomgen <arguments> --plugin-dir /path/to/my-sbomgen-plugins
Example:
inspector-sbomgen container --image alpine:latest -o /tmp/sbom.json --plugin-dir /path/to/my-sbomgen-plugins
For code completion, install the VS Code Lua language server extension:
https://luals.github.io/#vscode-install
For more information:
- Plugin guide: my-sbomgen-plugins/docs/sbomgen-plugin-developer-guide.md
- Testing guide: my-sbomgen-plugins/docs/sbomgen-plugin-testing-guide.md
- API reference: my-sbomgen-plugins/docs/sbomgen-plugin-api-reference.md
- Documentation: https://docs.aws.amazon.com/inspector/latest/user/sbom-generator.html
Now that you have a plugin workspace, let’s explore its contents in greater detail:
The scaffolded project includes a working discovery and collection plugin pair, passing unit tests with test fixtures under _testdata/, a .vscode/settings.json for integrated development environment (IDE) integration, and a local copy of the developer documentation.
The scaffolding is deliberately succinct and complete, so it reads well for both humans and AI coding assistants. Every file has clear comments that explain what each function does and what the plugin author needs to fill in.
To test a plugin, you first need something to scan, such as a package lock file or a compiled binary. The example plugin inventories a fictional example.lock with the following contents:
The provided discovery plugin knows how to look for instances of example.lock within the artifact file system:
-- my-custom-ecosystem discovery plugin
-- Discovers example.lock files in the artifact file list.
function discover()
return sbomgen.find_files_by_name({"example.lock"})
end
And the provided collection plugin knows how to parse the contents of example.lock and publish package findings to the output SBOM.
-- my-custom-ecosystem collection plugin
-- Parses example.lock files and extracts package name and version.
function collect(file_path)
local content = sbomgen.read_file(file_path)
if content == nil then
return
end
for line in content:gmatch("[^\n]+") do
local name, ver = line:match("^(.+)==(.+)$")
if name and ver then
sbomgen.push_package({
name = name,
version = ver,
purl_type = "generic",
namespace = "my-custom-ecosystem",
component_type = sbomgen.component_types.APPLICATION,
})
end
end
end
Run the tests
Plugins ship with a built-in test framework so you can validate your logic before scanning a real artifact. Tests are written in Lua, live next to the plugin in init_test.lua, and reference fixture data in _testdata/:
function test_discovers_packages()
local result = testing.scan_directory("_testdata")
testing.assert_equals(3, #result.findings)
testing.assert_equals("my-package-alpha", result.findings[1].name)
testing.assert_equals("1.0.0", result.findings[1].version)
end
function test_no_findings_for_empty_directory()
local result = testing.scan_directory("_testdata/empty")
testing.assert_equals(0, #result.findings)
end
Run the tests with the following command:
inspector-sbomgen plugin test --path my-sbomgen-plugins -v
=== RUN my-custom-ecosystem/discovery/init_test/test_discovers_packages
--- PASS: my-custom-ecosystem/discovery/init_test/test_discovers_packages (0.04s)
=== RUN my-custom-ecosystem/discovery/init_test/test_no_findings_for_empty_directory
--- PASS: my-custom-ecosystem/discovery/init_test/test_no_findings_for_empty_directory (0.04s)
ok 2 tests passed
This is the tightest development loop we could design: no Go toolchain, no rebuild, no container spin-up. Write a test, run it, iterate.
Scan a real artifact
For plugins to produce findings, inspector-sbomgen needs an artifact that contains the files your plugin looks for. For the example plugin, any directory with an example.lock file works. The fixture we generated earlier is a good stand-in:
The --plugin-dir flag tells inspector-sbomgen where to load your Lua plugins from. The resulting SBOM contains a CycloneDX component for each of the three packages in example.lock, for example:
Every plugin-generated component carries an amazon:inspector:sbom_generator:source_path property that records the file the component was collected from, so you can always trace a component back to the artifact that produced it.
Vulnerability scanning with Amazon Inspector
Plugin-generated findings are first-class SBOM components. They work with every downstream consumer that reads CycloneDX SBOMs, including Amazon Inspector. To send an SBOM to Amazon Inspector for vulnerability analysis, add the --scan-sbom flag (this requires an active AWS account):
An important caveat when you onboard a brand-new ecosystem: Plugin authors can inventory arbitrary ecosystems, but Amazon Inspector can only report vulnerabilities for components it has advisories for. When you point Amazon Inspector at a component whose ecosystem isn’t in its advisory feeds yet, Inspector will return the component with a property, Component skipped: no supported rules found. For example:
This is expected behavior, not an error. The SBOM is still generated correctly, the component is still tracked, and the source_path tells you exactly which file produced it. If and when Amazon Inspector adds advisory coverage for the ecosystem, the same SBOM will start producing vulnerability findings without any change to your plugin. For ecosystems Inspector already supports, plugin-generated components are indistinguishable from components produced by built-in scanners.
First class IDE support
We care about productivity and efficiency when writing plugins. Writing Lua without modern conveniences such as autocomplete isn’t fun, so every plugin project scaffolded with the plugin new command ships with a library/sbomgen.lua definition file and a .vscode/settings.json that automatically wires it up to the Lua Language Server extension for VS Code.
For code completion and IDE support, first install the sumneko.lua extension, open your plugin project in VS Code, and every sbomgen.* function will get:
Parameter hints with types.
Hover documentation.
Autocomplete for constants (sbomgen.component_types.*, sbomgen.groups.*, sbomgen.platform.*).
Type checking on function calls.
Inline warnings when required fields are missing from push_package().
The same definition file makes plugin development work well with AI coding assistants. The types and documentation are embedded in a form that tools can read, so assistants can generate correct plugin code with far less monitoring than writing against a raw language would require.
A safe foundation
Plugins run real code inside the same process as inspector-sbomgen, so we designed the execution environment to keep that code stable and security-hardened. Every Lua plugin runs in an isolated sandbox. Every Lua virtual machine (VM) has access to a restricted subset of the Lua standard library to ensure only safe operations are permitted:
No direct filesystem access. The Lua io library isn’t loaded. All file operations go through sbomgen.* functions, which route through sbomgen’s internals so your plugin behaves identically whether it’s scanning a directory on disk, a container image, a compressed archive, or a mounted volume.
No subprocess execution or environment mutation. The Lua os library is blocked, so plugins can’t spawn processes, modify environment variables, or touch files outside the artifact.
No VM introspection. The Lua debug library is blocked.
No unbounded code loading.dofile, loadfile, and loadstring are removed. require() is available but restricted to the plugin’s own directory tree, so plugins can share helper modules with themselves but cannot load code from other plugins or system paths.
If a plugin raises an unhandled Lua error, inspector-sbomgen logs a warning and continues with the next file or plugin; one faulty plugin does not prevent other plugins from running. Plugins never override inspector-sbomgen’s built-in package collectors. Every plugin must declare a unique name. If a custom plugin uses a name that’s already claimed by an official built-in plugin, the custom plugin is skipped with a warning. Built-in plugins always take precedence, so a custom plugin can never silently replace or shadow the tool’s own detection behavior.
Whether you’re adding support for an internal lockfile format, prototyping detection for a new open source ecosystem, or replacing a home-grown scanner with something your whole organization can run at scale, the plugin system is designed to make the path from idea to working SBOM as short as possible. We can’t wait to see what you build with it. If you have feedback about this post, submit comments in the Comments section below. If you have questions about this post, contact AWS Support.
With Amazon EMR Serverless, you can run open source big data frameworks such as Apache Spark and Apache Hive without managing clusters or infrastructure. Customers are increasingly choosing EMR Serverless for their analytics workloads because of the simplicity of a fully managed, serverless experience. As adoption grows, teams want to bring their most demanding jobs to Serverless too. These jobs include large-scale joins, shuffle-heavy ETL, and memory-intensive analytics that previously required carefully sized clusters. Customers migrating these heavyweight workloads from their Spark clusters often need the same compute shapes on EMR Serverless to achieve the same price-performance and make migration easier.
Today, we’re excited to announce a new 32 vCPU / 244 GB worker configuration on Amazon EMR Serverless, giving you the headroom to run your most intensive workloads without leaving the serverless experience.
Overview of larger workers
An EMR Serverless application uses workers to run your Spark tasks, and you can choose a worker size that matches your workload. The new 32 vCPU worker offers a large compute and memory footprint (32 vCPUs and 244 GB of memory) that supports attaching up to 2,000 GB of shuffle-optimized disk. This combination benefits three common workload patterns:
Shuffle-intensive workloads – Wide transformations such as join, groupBy, sortBy, and repartition redistribute large amounts of data across the cluster. Larger workers keep more shuffle data local to each executor and read and write shuffle blocks on higher-throughput disk, reducing remote fetches and shuffle wait time.
I/O-heavy workloads – Queries that scan large datasets or spill intermediate data to disk are limited by disk throughput and IOPS. The shuffle-optimized disk raises the ceiling on both sides. Large workers also improve network bandwidth.
Memory-intensive workloads – Higher per-executor memory (244 GB versus 30 GB) lets more data be cached and processed in memory without spilling, which helps with data skew and caching.
Benchmark setup
We compared the recommended large-worker configuration against common small-worker defaults at identical total compute (192 vCPUs).
Small workers – 48 executors × (4 vCPU / 30 GB / 200 GB standard disk).
The two configurations differ in both worker shape and disk class. We compare them as paired configurations, since shuffle-optimized disk is the intended disk type for the 32 vCPU worker. The reported gains reflect this combined effect.
Both configurations set spark.dynamicAllocation.enabled=false and spark.scheduler.minRegisteredResourcesRatio=1 to reduce variance from worker launch times. Requiring full registration ensures each query starts only after the cluster is ready. We ran every query with 3 iterations and reported the median. Both benchmarks ran on EMR release emr-7.13.0 in a virtual private cloud (VPC) with an Amazon Simple Storage Service (Amazon S3) gateway endpoint attached to the private subnets.
Benchmark results
This post presents benchmark results comparing the new 32 vCPU workers against the existing 4 vCPU standard workers using the industry-standard TPC-DS and TPC-H benchmarks. Across 126 queries (104 TPC-DS and 22 TPC-H), large workers delivered an average of 29% faster query execution and 29% lower query-attributed cost, with zero regressions and improvements peaking at 45–55% on shuffle-heavy, multi-table join queries. Both the larger executor shape and the shuffle-optimized disk contribute to these gains. The disk advantage is most pronounced on the shuffle and I/O-heavy queries where the largest improvements appear.
The following table summarizes the results across both benchmarks. Large workers won every query on both performance and cost.
Note: The benchmark results in this post are derived from the TPC-DS and TPC-H benchmark specifications. TPC-DS and TPC-H are trademarks of the Transaction Processing Performance Council.
Benchmark
Queries tested
Avg performance improvement
Avg cost improvement
TPC-DS (3 TB)
104
26.7%
27.4%
TPC-H (1 TB)
22
38.5%
37.2%
Combined
126
28.8%
29.1%
TPC-DS 3 TB benchmark
TPC-DS is an industry-standard decision support benchmark that models complex analytical workloads with multi-table joins, subqueries, and aggregations. We ran 104 queries from the TPC-DS v2.4 suite against a 3 TB partitioned Parquet dataset, with 3 iterations per query for statistical confidence.
Queries: 104 (full suite minus 4 incompatible with the dataset schema).
Iterations: 3 per query (one query per start-job-run, a fresh Spark application each time).
Networking: VPC with an Amazon S3 gateway endpoint.
EMR release: emr-7.13.0.
The following chart shows the top 10 and bottom 5 queries by performance improvement. All 104 queries show a positive improvement, with the largest gains on shuffle-heavy queries such as q58 (45%), q21 (43%), and q12 (42%).
Performance improvement for the top 10 and bottom 5 TPC-DS queries, 32 vCPU compared to 4 vCPU standard workers
Why large workers are faster for TPC-DS
TPC-DS queries are characterized by complex multi-table joins that generate large shuffle operations. With 6 large executors instead of 48 small ones, the shuffle-optimized disk provides significantly higher random I/O throughput for reading and writing shuffle blocks. In addition, fewer executors mean less network coordination during shuffle. Each executor fetches shuffle data from only 5 remote sources instead of 47. This increases the share of shuffle data read locally, which improves performance.
TPC-H 1 TB benchmark
TPC-H is a decision support benchmark that focuses on ad hoc analytical queries. We ran all 22 TPC-H queries against a 1 TB dataset, with 3 iterations per query. Each query was submitted as a separate start-job-run (a fresh Spark application) to simulate the realistic pattern of independent ad hoc queries arriving without session warmup.
The test environment was as follows:
Dataset: 1 TB partitioned Parquet.
Queries: 22.
Iterations: 3 per query (one query per start-job-run, a fresh Spark application each time).
Networking: VPC with an Amazon S3 gateway endpoint.
EMR release: emr-7.13.0.
Performance improvement for TPC-H queries, 32 vCPU compared to 4 vCPU standard workers
Why large workers are faster for TPC-H
The shuffle-optimized disk accelerates this initial table-scan phase. The subsequent query execution benefits from higher per-executor memory (244 GB versus 30 GB), which lets more data be processed in memory without spilling to disk.
Cost calculation and improvement results
The following table compares the cost of running TPC-DS and TPC-H benchmarks on larger workers (32 vCPU / 244 GB / 2,000 GB shuffle-optimized disk) versus smaller workers (4 vCPU / 30 GB / 200 GB standard disk) on EMR Serverless. Both configurations use identical total compute (192 vCPUs).
TPC-DS 3 TB
Metric
Larger Workers (6 × 32 vCPU)
Smaller Workers (48 × 4 vCPU)
Runtime
2,780.8s
3,670.2s
Resource Billed
vCPU = 148.31 | Memory = 1,130.87 |
Disk = 9,269.47
vCPU = 195.74| Memory = 1,468.07 |
Disk = 9787.15
Cost
$15.37
$19.87
Total vCPU
192
192
Total Disk
12 TB (shuffle-optimized)
9.6 TB (standard)
Improvement
27.4% lower cost
Baseline
TPC-H 1 TB
Metric
Larger Workers (6 × 32 vCPU)
Smaller Workers (48 × 4 vCPU)
Runtime
940.7s
1,529.5s
Resource Billed
vCPU = 50.17 | Memory = 382.54 |
Disk = 3,135.60
vCPU = 81.57 | Memory = 611.81 |
Disk = 4,078.72
Cost
$5.20
$8.28
Total vCPU
192
192
Total Disk
12 TB (shuffle-optimized)
9.6 TB (standard)
Improvement
37.2% lower cost
Baseline
Notes:
Runtime represents the sum of median query execution times across all queries in the benchmark (3 iterations per query, median reported).
Calculated cost is computed using EMR Serverless on-demand pricing: vCPU-hr ($0.052624), Memory GB-hr ($0.0057785), Storage GB-hr ($0.000111).
Cost is proportionally attributed to query execution time, excluding Spark initialization and shutdown overhead.
Both configurations use identical total vCPU (192) with Dynamic Resource Allocation disabled.
The improvement percentage represents the cost reduction achieved by larger workers relative to smaller workers.
Disk Sizing: The larger workers provisioned 25% more disk (12 TB versus 9.6 TB), yet the total cost came out 22.6–37.2% lower. Disk is by far the lowest-priced billing dimension on EMR Serverless ($0.000111 per GB-hour, versus 52x that for memory and 474x for vCPU) and made up only $1.03 of the $15.37 TPC-DS total. Many customers under-provision disk to trim this smallest line item, and it backfires. Shuffles slow down, jobs run longer, and every extra second is billed on the costlier vCPU and memory dimensions. On large workers (8+ vCPUs), shuffle-optimized disks scale IOPS and throughput with capacity, which helps move shuffle data faster. Size disk as a performance lever, not a cost lever.
When to use large workers
To determine whether the 32 vCPU worker with shuffle-optimized disk will benefit your Spark applications, consider the following:
Check the Stages tab of the Spark History Server for your EMR Serverless application and review the Shuffle Read and Shuffle Write columns. The larger the shuffle volume relative to the number of executors, the more a job benefits from keeping shuffle data local on high-throughput disk. Jobs that shuffle tens of gigabytes or more per executor are strong candidates.
Check the Stages tab for the Spill (memory) and Spill (disk) columns and the Executors tab for peak JVM memory. If data is spilling to disk or peak memory is close to the configured executor memory, the higher memory of the large worker can remove the spill and improve performance.
When not to use large workers
Large workers are not the best fit for every workload:
I/O-bound jobs – For workloads whose runtime is dominated by reading and writing data (rather than shuffle or memory pressure), a larger number of smaller workers (for example, 8 or 16 vCPU) with the same disk sizes can deliver better aggregate throughput. Spreading the work across more executors increases read/write parallelism, although a few large workers can leave disk and network bandwidth underused.
Conclusion
In this post, we demonstrated that EMR Serverless 32 vCPU workers deliver performance and cost benefits for shuffle-intensive Spark workloads. Across 126 TPC-DS and TPC-H queries, larger workers achieved an average 29% faster execution and 29% lower cost.
We recommend evaluating the 32 vCPU worker with shuffle-optimized disk for your shuffle-intensive and I/O-heavy EMR Serverless Spark workloads. To get started, use the following configuration:
For more information about worker configurations, refer to Worker configurations in the Amazon EMR Serverless User Guide. We also recommend staying up to date with the latest EMR releases to take advantage of ongoing performance improvements.
If you use and install packages from npm or PyPI, the first hours after a package is published are the riskiest because scanners can’t analyze packages before publication. Recent supply chain events affecting NodeJS and Python packages have been detected and removed within hours. However, while those packages were available to the general public, it’s possible that they were installed by users, creating the potential for a security incident. As you will see from the data that follows, if users had waited 1 day before accessing those packages, none of the recent supply chain security events would have had an impact.
In this post, I show you a one-line configuration that you can use to eliminate this exposure in your environment: a dependency cooldown for npm and pip. This change tells your package manager to skip versions published in the last 24 hours, giving the security community time to detect and remove unexpected packages before they reach your systems. These settings secure the default setup. There’s another use case of package updates: receiving security fixes to address security risks. This process involves updating packages to a more recent version. I also show you how to override the cooldown configuration so you can install the latest security patches while newly installed package updates are delayed. We recommend that you assess the severity of code defects and apply security fixes if there’s known risk. Handling security fixes based on their severity—and how to specify SLAs for these fixes based on severity—is beyond the scope of this blog post.
Background: Two risks pull in opposite directions
Software delivered by Amazon Linux packages go through review by Amazon package maintainers and pass guardrails before release. Open source software is developed and maintained with similar processes and guardrails. The npm and PyPI registries have open publishing access and don’t enforce reviews. Unexpected packages are potentially added to the registries because of risks like impersonation or stolen credentials. You’re caught between two risks: older software accumulates unpatched vulnerabilities, while new packages potentially contain unexpected vulnerabilities that haven’t been detected yet. The best approach is to stay current without adopting the newest releases immediately, while applying recommended security fixes. The following diagram illustrates the relation between the two types of risks in an abstract way, where the supply chain risk is highest immediately after a package is published, because unexpected updates can potentially bypass guardrails. After a package is published, auditing can review it and identify potential defects over time. If no security fixes are applied, the risk of all the code defects adds up.
Figure 1: Software risk over lifetime. Unpatched vulnerabilities risk increases over time. Very recent software also carries more supply chain risk.
The problem: The first day presents the highest risk
Supply chain events follow a consistent pattern. An unexpected author publishes an unexpected package or package version and waits for automated systems and users to pull it in. Security researchers and automated scanners typically detect and remove these packages within hours, but by then, systems have been exposed to the risk.
Datadog’s 2026 State of DevSecOps report found that 54% of JavaScript applications install at least one dependency within a day of its release. That’s the time window that presents the highest supply chain risk. Recent events show how fast detection happens:
Event
Exposure window
Nx s1ngularity (Aug 2025)
4–5 hours
axios (Mar 2026)
2–3 hours
Bitwarden CLI (Apr 2026)
93 minutes
TanStack (May 2026)
30 minutes
node-ipc (May 2026)
less than 24 hours
The solution: Skip packages published today
A dependency cooldown tells your package manager to skip recently published versions. If a version hasn’t existed on the registry for the configured timespan, for example, 1 day, it won’t be installed, giving the security community time to detect and remove unexpected versions.
A 1-day cooldown blocks each event listed in the preceding table. Notably, several of these events produced valid provenance attestations and passed build verification. These provenance checks alone didn’t stop them. A cooldown works independently of authorization mechanisms, because it blocks by age rather than by trust.
Both npm (v11.10.0+) and pip (v26.1+) support cooldowns . Amazon Linux 2023 ships these packages in NodeJS 24 and Python 3.14 since release 2023.11.20260608.
If you use lockfile-based installations through npm ci or pip install -r requirements.txt with pinned versions, you won’t pull latest package updates. The cooldown doesn’t apply to those installations. The cooldown only affects resolution of new or updated packages. See the Lockfile-based installs and the cooldown section for details.
Prerequisites
To implement the following solution, you first need to have the following prerequisites in place:
Node.js 24 with npm 11.10.0 or later (in nodejs24-24.14.1-1.amzn2023.0.1 or later).
Python 3.14 with pip 26.1 (in python3.14-pip-26.1.1-1.amzn2023.0.1 or later)
pip-audit (tool to scan python packages required for defect-based override scripts). Use python3.14 -m pip install pip-audit to install.
Future versions of Node.js and Python will bring new commands. The following tool commands work for Amazon Linux 2023 with Node.js 24 and Python 3.14. The provided commands target specific package versions. Adjust the commands if you use later releases.
To set up the npm cooldown
Create the global configuration directory, depending on your NodeJS version. sudo mkdir -p /usr/lib/nodejs24/etc
Add the npm configuration file with the cooldown setting. sudo npm-24 config set min-release-age 1 --location=global
Check that the cooldown is active by running the next command. npm-24 config list
You will see before = "<timestamp from 24 hours ago>" in the output, confirming npm converted the 1-day cooldown into a date filter. For more information, see the npm min-release-age documentation.
To set up the pip cooldown
Create the system-wide pip configuration file with the cooldown setting. sudo python3.14 -m pip config set --global global.uploaded-prior-to P1D
Verify the configuration (for Python 3.14 and pip 26.1+). python3.14 -m pip config list
You will see global.uploaded-prior-to='P1D’ in the output.
This configuration is safe to deploy immediately, because older pip versions (25.x) silently ignore the setting.
To install a package’s latest version without cooldown
What if you want to install the latest version of a package, for example to receive security fixes? The following sections describe how to override the flag using the tool command line. To identify which packages need urgent updates, run the appropriate audit command for your package manager.
npm auditor python3.14 -m pip_audit
For npm packages
Install the package with the cooldown override. npm-24 install <package-name> --min-release-age=0
For pip packages
Install the package with the cooldown override. python3.14 -m pip install <package-name> --uploaded-prior-to="P0D”
Update packages that need urgent updates
We recommend that you apply security fixes for packages that have known security risks. You don’t need to turn off the cooldown entirely to apply security fixes. Use the audit tools to identify packages with known issues, then override the cooldown for only these packages.
Prerequisites: Ensure you have Python 3 and pip-audit installed (python3.14 -m pip install pip-audit).
Important: These scripts demonstrate the concept. For production use, add error handling, logging, and testing. Review packages before updating them in automated pipelines.
For npm packages
The following script demonstrates the required steps to identify npm packages with a known security fix. The npm audit command prints these packages as JSON. Next, packages in this list are updated with an npm install command, where their cooldown is overridden so that the latest version is picked up.
npm audit --json | python3 -c "
import json, sys, subprocess
data = json.load(sys.stdin)
for pkg in data.get('vulnerabilities', {}):
subprocess.run(['npm-24', 'install', f'{pkg}@latest', '--min-release-age=0'])
"
For pip packages
The following script demonstrates the required steps to identify pip packages with a known security fix. The pip_audit command prints these packages as JSON. Next, all packages in this list are updated with an pip install command that overrides the cooldown so that the latest version can be picked up.
If you use npm ci or pip install -r requirements.txt with pinned versions, the cooldown doesn’t apply. These commands install what the lockfile specifies, regardless of package age. The cooldown only affects resolution of new or updated packages.
Industry adoption: Cooldowns are now used across PyPI and NodeJS
Major package managers and enterprises have started to adopt dependency cooldowns. As of May 2026, several popular package management tools now include cooldown features: pnpm (a fast Node.js package manager), Renovate (an automated dependency update tool), and StepSecurity (a supply chain security platform).
pnpm 11 ships with minimumReleaseAge enabled by default. It’s one of the first major package manager to make cooldowns opt-out rather than opt-in.
Renovate’sconfig best-practices preset has included a 3-day npm cooldown since 2025 and is widely adopted across enterprises.
StepSecurity Secure Registry uses a configurable cooldown period for enterprise customers. StepSecurity recommends a 10 day delay as default.
How AWS is helping protect the open source supply chain
AWS scans upstream package registries to catch unexpected packages before they reach customers.
Amazon Inspector, a security management service that continuously scans workloads for software vulnerabilities and network exposure, uses AI-assisted detection rules to scan upstream package registries. In 2025, Amazon Inspector researchers identified over 150,000 unexpected npm packages linked to a token farming campaign.
Unexpected packages are typically caught within hours of publication. A 1-day cooldown ensures you don’t install them during that detection window.
Recommendations
To secure your Amazon Linux 2023 configuration:
Set a 1-day cooldown for npm and pip as shown in the preceding sections. External registries don’t have human review, so give the defenders time to catch problems.
Override when needed for urgent security patches using the per-command flags.
Run npm audit or pip_audit regularly to identify packages that need immediate attention.
Set up the cooldown with one line of configuration, and the protection is immediate.
Conclusion
By implementing the solutions presented in the post, you secure your npm and PyPI environment from most instances of unexpected code. The update delay of 1 day protects your environment, while still allowing to apply the latest security fixes. To learn about how to protect your environment further, see the following resources:
One-time passwords (OTPs) are the backbone of modern user verification, from account creation to wallet additions, to password resets. But for businesses operating globally, delivering OTP messages reliably across dozens of countries is complex. It has traditionally meant navigating a maze of country-specific registrations, originator provisioning, and carrier compliance requirements. AWS End User Messaging Notify changes that equation entirely by removing the complexity of global OTP delivery.
Before we dive into the technical implementation, let’s explore what Notify is, why it matters for businesses sending verification codes at scale, and how you can go from zero to sending OTPs in minutes, not weeks.
The problem with traditional OTP sending
Sending OTP messages through traditional SMS channels requires significant upfront investment. For each country you want to reach, you need to research which origination identity types are supported: short codes, long codes, toll-free numbers, sender IDs, or 10DLC. Each has different registration timelines, throughput limits, and costs. Some countries require pre-registered message templates. Others mandate sender ID registration or risk having your messages displayed as “LIKELY-SCAM” to recipients.
For a business targeting multiple countries, that complexity multiplies fast. Each market carries its own registration process, approval timeline, and compliance requirements. A US short code can take 8–12 weeks to provision. India requires Distributed Ledger Technology (DLT) registration with the Telecom Regulatory Authority of India (TRAI), including entity verification, template approval, and header registration. Even straightforward markets like the UK now require sender ID registration to prevent carrier filtering.
The result? Businesses spend weeks or months on registration paperwork before sending their first verification code. And that assumes every registration is approved on the first attempt.
What is Notify?
Notify is a fully managed OTP and verification code sending service within AWS End User Messaging. Instead of provisioning your own phone numbers, managing carrier registrations, and building routing logic, you create a single Notify configuration and AWS handles the rest: the origination identities, carrier compliance, message routing, and even fraud protection.
The core benefit behind Notify is simplicity. You provide your brand name, select your target countries, choose a pre-approved message template, and start sending. AWS automatically validates your account, assigns the appropriate origination identities for each destination country, configures SMS Protect for fraud prevention, and routes your messages through the optimal delivery path. What previously required weeks of registration work now takes minutes.
The business case for Notify
The value of Notify becomes clear when you look at the traditional OTP implementation journey. A business targeting the United States, United Kingdom, Japan, and India would face four completely different registration processes:
United States: Choose between toll-free (15 business days, best case), 10DLC (moderate timeline), or short code (8-12 weeks or more). Sender IDs are not supported.
United Kingdom: Sender ID registration is required. Unregistered sender IDs risk being displayed as “LIKELY-SCAM.” Short codes are available through an AWS support case.
Japan: Sender IDs are still supported without pre-registration, but number display isn’t guaranteed across all carriers (particularly KDDI). Dedicated short codes are available through an AWS support case.
India: Requires DLT registration including entity verification, content template approval, and header (sender ID) registration. Traditionally one of the longest lead-time markets to onboard.
With Notify, your OTP use case in all four of these countries can be operational in minutes. For markets that previously required weeks, you can send immediately while working through dedicated registrations in parallel.
To illustrate the business impact across industries:
Hospitality and entertainment: Theme parks and resorts with global audiences can verify guest accounts across 50+ countries without managing individual country registrations, so international visitors can add passes to digital wallets, create accounts, and complete elevated security actions.
Ecommerce: Global marketplaces can onboard customers from any supported country with verified OTPs, eliminating the registration bottleneck that delays international expansion.
Financial services: Banks and fintech companies can deploy multi-factor authentication globally, with built-in fraud protection that automatically filters artificially inflated traffic and SMS pumping.
Healthcare: Patient portals can verify identities across international locations, with pre-approved templates that help maintain compliance without custom message body management.
Where to start?
When evaluating Notify for your OTP program, we recommend starting with your highest-volume verification use case. OTP and code verification messages follow predictable patterns, making them ideal candidates for Notify pre-approved templates. More importantly, they provide the most immediate return on investment (ROI) by eliminating the registration overhead for your most critical customer touchpoint.
Consider a phased approach:
Phase 1 – Quick win with Notify: Set up Notify for your primary markets. Get OTP sending operational in minutes. Use this to validate your integration and monitor deliverability.
Phase 2 – Evaluate and scale: As volumes grow, evaluate whether specific high-volume countries benefit from dedicated origination identities for higher throughput or branding purposes. Notify supports a hybrid approach in which you can use your own originators alongside Notify managed identities.
Phase 3 – Complex markets: For countries with strict registration requirements, begin the dedicated registration process in parallel. After they’re approved, integrate those dedicated identities into your sending architecture while Notify continues to handle the rest.
Now that you have decided where to start, the next sections walk you through setting up your first Notify configuration, from prerequisites to sending your first OTP.
Prerequisites
Before you begin, make sure that you have the following prerequisites in place:
A brand display name for your Notify configuration.
A planned list of target countries for your OTP messages.
Note: The display name must contain only letters, numbers, spaces, hyphens, or underscores, and can be up to 15 characters.
Understanding Notify tiers
Notify offers two tiers, each designed for different stages of your OTP implementation:
Feature
Basic tier
Advanced tier
Transactions per second (TPS)
1
25
Daily message limit
200 messages/day
Unlimited
Country availability
30 pre-approved low-risk countries
Full country list
Short code access
No
Yes
Fraud protection (SMS Protect)
Mandatory – AWS managed
Mandatory – AWS managed
Compliance verification
Trust-based with audit
AWS verifies opt-in compliance
The Basic tier is designed for getting started, testing, and low-volume use cases. It provides immediate access with conservative limits, which is well suited to validating your integration before scaling to production.
The Advanced tier unlocks higher throughput, unlimited daily sending, and access to the full list of supported countries. To upgrade, you complete a streamlined verification process where AWS confirms your opt-in compliance. This is significantly faster than traditional carrier registration because the verification stays within the AWS boundary. No downstream carrier approval is required.
The Basic tier is live almost immediately. The Advanced tier upgrade requires a brand verification registration that demonstrates a compliant opt-in flow, and most requests are processed within 3-5 business days, which is still much faster than traditional carrier registrations.
Spend limits: Notify has a separate spend limit from standard SMS sending. AWS auto-approves more generous limits for Notify, but plan accordingly if you anticipate high volumes at launch.
Testing tip: While testing, keep in mind there is a per-recipient cap of 10 messages per day per Notify configuration (and 10 per day per account) for any single destination phone number. If repeated test sends to your own phone suddenly stop arriving, this limit, not a configuration error, is the likely cause.
A closer look at Advanced tier country coverage
The Advanced tier documentation states it supports all countries available on AWS End User Messaging SMS. While technically accurate, there is an important distinction not immediately obvious from the console: not all countries are fully managed with Notify.
At the time of writing, 68 of the 247 countries available on the Advanced tier (28%) require you to provide your own origination identity. For those destinations, you still need to go through number provisioning and carrier registration. AWS does not automatically assign managed identities for these countries.
Look for the CustomerOwnedIdentityRequired field in the response.
Coverage type
Count
Percentage
Fully managed by AWS
179
72%
Customer-owned identity required
68
28%
Total Advanced tier countries
247
100%
Note: These figures reflect coverage at the time of publication. As AWS continues expanding Notify fully managed coverage, these numbers may change – always use the API for the most current breakdown.
Generating the countries list as a CSV
Because Notify’s country coverage evolves over time as AWS expands fully managed support, we recommend always generating the latest list with the AWS CLI, in a spreadsheet-friendly format, whenever you need it.
For 179 countries, coverage is turnkey. Create your configuration and start sending immediately.
For 68 countries, you need a hybrid setup. Provision origination identities and associate them through a phone pool. Notify still handles routing, templates, and fraud protection.
This reinforces the phased approach: start with the 179 fully managed countries, then register for the remaining 68 in parallel.
Eight countries support SMS only, with no voice: Austria, China, France, Gabon, Germany, Italy, Pakistan, and Slovenia.
Pro tip: Before committing to your target country list, run the ListNotifyCountries API call and look for CustomerOwnedIdentityRequired: true to identify which markets need additional lead time.
How Notify works
Notify simplifies the OTP sending workflow into three steps:
Create a Notify configuration – Provide your brand display name and select your use case (currently code verification). Optionally configure target countries, preferred templates, and channel settings.
Enable countries and channels – Select which countries you want to send to and enable SMS, voice, or both. AWS automatically configures the appropriate origination identities and fraud protection for each country.
Send messages – Use the SendNotifyTextMessage API to deliver OTPs. Pass your Notify configuration ID, the destination phone number, and your template variables (such as the OTP code). AWS handles identity selection, template resolution, and message delivery.
Registration steps
To create your first Notify configuration, open the AWS End User Messaging SMS console, choose Notify configurations, and choose Create configuration.
Enter your brand display name, which cannot be changed after creation.
Select the Code verification use case. Optionally, under Advanced, set your target countries, channels, language code, a default template, and a phone pool (for a hybrid setup).
After your configuration is active, you can send a test message directly from the console:
Navigate to your Notify configuration and choose the Test tab.
Enter a destination phone number.
Select a message template and language.
(Optional) Configure the OTP code value and expiration.
Choose Send test message.
The recipient receives a message similar to the following:
“[YourBrandDisplayName], your one-time password verification code is [PASSWORD]. Please do not share this message. Sent by Notify.”
Note: This post focuses on delivering the OTP message itself. Generating a secure OTP code and verifying the code the customer submits back are outside the scope of this post. For guidance on that side of the architecture, including code generation, storage, and verification, see Build a Secure One-Time Password Architecture with AWS.
Before running, replace the placeholder values with your own:
notify-config-1234567890 – Your Notify configuration ID.
+15555550123 – The destination phone number in E.164 format.
your-template-id – The template ID you created earlier.
123456 – Your OTP or variable value.
Run the script:
python3 send_notify_message.py
You receive output similar to the following:
Message ID: msg-1234567890abcdef0
Note: Notify uses the same pinpoint-sms-voice-v2 API namespace as standard AWS End User Messaging SMS. If you have an existing integration, the migration path is straightforward. The primary difference is the API endpoint and parameters.
Notify and standard AWS End User Messaging SMS: Better together
Notify works alongside your existing AWS End User Messaging SMS infrastructure. This hybrid approach combines the strengths of both:
Notify for rapid global coverage: Use Notify for countries where you don’t have dedicated origination identities. Eliminate registration delays and start sending immediately.
Dedicated identities for high-volume markets: For countries where you need maximum throughput, custom branding, or two-way messaging, continue using your own short codes, toll-free numbers, or registered sender IDs.
Pool-based prioritization: Associate your dedicated numbers in a phone pool with your Notify configuration. Notify prioritizes your numbers first and falls back to AWS managed identities only when needed.
This approach works well for businesses expanding internationally. You can launch OTP verification in new markets on day one with Notify, then transition to dedicated identities as volume and requirements justify the investment.
How Notify compares to standard AWS End User Messaging SMS
The following table compares standard SMS sending with Notify across the capabilities that matter most when planning your OTP program.
Country rules: Control which countries can receive messages. We recommend disabling all countries by default and enabling only your target markets. You can adjust these settings at any time through the console, API, or CLI.
Fraud filtering: AWS automatically filters artificially inflated traffic and SMS pumping. On the Basic tier, filter settings are AWS managed. On the Advanced tier, you retain the same protections with additional controls.
Spend controls: Notify includes a dedicated spend limit separate from your standard SMS spend limit. Monitor your usage through Amazon CloudWatch metrics and set up billing alerts to track spending.
You can view and manage your country rules directly within the Notify configuration. To allow or block specific countries:
Navigate to your Notify configuration.
Choose the Countries tab.
Toggle countries between Allow and Block status.
Note: SMS Protect is part of all AWS End User Messaging, not exclusive to Notify. If you decide to transition from Notify to dedicated origination identities for specific countries, your Protect configurations carry over without additional configuration.
Important considerations
Display name is permanent: Your display name appears in every message and cannot be changed after creation. If you need to experiment, create a test configuration first, then create a production configuration with your finalized brand name.
Templates are pre-approved: You cannot create custom OTP message bodies with Notify. This is by design. Pre-approved templates help maintain carrier compliance and reduce the risk of message filtering. If you need custom message content, use standard SMS sending with your own origination identities.
Architecture compatibility: Notify uses the same V2 API (pinpoint-sms-voice-v2) as standard AWS End User Messaging SMS. If you have an existing architecture using Amazon API Gateway, AWS Lambda, and Amazon Simple Queue Service (Amazon SQS) for message routing, integrating Notify requires only parameter changes, not an architectural redesign.
Message feedback: Notify supports the Message Feedback API for tracking OTP conversion rates. Pass the MessageFeedbackEnabled parameter when sending to track whether recipients successfully verify their codes. This provides first-party data for measuring authentication success rates across countries and carriers.
Managing costs and usage
Notify pricing includes a per-message service fee on top of standard SMS transport rates. The service fee covers origination identity management, fraud protection, and routing optimization.
Monitor your Notify message volume through Amazon CloudWatch metrics and the analytics dashboard within your Notify configuration. Set up billing alerts to track spending against your budget. For the latest rates, see AWS End User Messaging pricing.
Conclusion
In this post, we showed you how AWS End User Messaging Notify eliminates the registration complexity of global OTP sending. You get a fully managed verification code service with built-in fraud protection, pre-approved templates, and coverage across 200+ countries, operational in minutes instead of weeks.
Evaluate your current OTP sending workflow and identify the countries where registration overhead is delaying your go-to-market. Consider starting with Notify for those markets to establish immediate coverage, then layer in dedicated origination identities for high-volume countries as your program scales.
Get started today
Ready to implement Notify? Here are your next steps:
If you’re ready to start: Open the AWS End User Messaging SMS console and create your first Notify configuration. You can be sending test OTPs within 5 minutes.
Fanatics Commerce is a leading designer, manufacturer, and retailer of licensed consumer products, including fan gear, jerseys, lifestyle and streetwear products, headwear, and hardgoods. Whether it’s a championship jersey or a last-minute gift, fans trust Fanatics to deliver and that trust extends to every digital touchpoint along the way.
Every order confirmation, shipping notification, and account update represents a moment of connection with a fan. Fans check their inbox after buying a jersey, track a package before game day, and verify their account when they sign up. These emails are the backbone of the fan experience.
The Fanatics Commerce engineering team built a modern, scalable email platform on Amazon Simple Email Service (Amazon SES), designed from the start for high deliverability, operational efficiency, and seasonal scale that comes with serving more than 100 million fans. When events like Super Bowl, NBA Finals, or World Series drive a surge in orders, the platform has to keep up without missing a beat.
This post walks through what drove the decision, the platform architecture, migration, the key engineering decisions, and what comes next for a large transactional email platform running on Amazon SES.
The case for change
As Fanatics Commerce grew, the engineering team saw an opportunity to elevate their email infrastructure by using Amazon SES capabilities purpose-built for operating at scale.
Dedicated IP addresses for full reputation control. With dedicated IPs in Amazon SES, Fanatics Commerce could own their sending reputation entirely removing dependency on shared infrastructure and gaining direct control over deliverability outcomes.
Granular traffic segmentation. Amazon SES offered the ability to treat transactional and marketing email as distinct, independently managed streams each with its own configuration sets, sending identities, and performance tuning rather than routing everything through a single pipeline.
Real-time deliverability visibility. At the scale of millions of fans, the team needed domain-level insight into open rates, bounce rates, and complaint rates in real time. The built-in analytics and Virtual Deliverability Manager in Amazon SES gave them the detail to diagnose shifts quickly and act decisively.
Domain-level isolation and authentication. SES enabled the team to assign dedicated subdomains and authentication policies (DKIM, SPF, DMARC) per email type ensuring high-priority transactional messages maintain protected, independent reputations.
Operational automation at scale. IP warming, reputation monitoring, and sending pattern adjustments could be managed programmatically through SES rather than requiring manual intervention keeping pace with Fanatics Commerce’s volume growth.
The team recognized the opportunity to move beyond incremental fixes. Rather than continuing to adapt an existing system, they set out to build a purpose-built transactional email platform on AWS that addressed all of these needs from the ground up.
Why Fanatics Commerce chose Amazon SES
After evaluating their requirements against several email service providers, the Fanatics Commerce team chose Amazon SES for its combination of reputation control, native observability, and tight integration with their existing AWS infrastructure. Several capabilities stood out during their evaluation.
The priority was reputation control. SES supports dedicated IP pools with separate pools for high-priority transactional, account, and lower-priority traffic, ensuring noisy streams cannot contaminate critical flows.
Visibility was equally important. As Rajat Banerjee, Fanatics’ engineering leader, explains:
“SES emits detailed JSON events for every send, delivery, bounce, complaint, open, and click into S3, and we model that data directly in our warehouse. Tagging each event with order, site, and mailbox provider, plus the user agent SES captures on opens and clicks, lets us slice deliverability at the level we need to run at Fanatics Commerce scale. That granularity is what let us refine our NPS survey email, power order attribution reporting, and debug real production issues over the last few months.”
The team also valued owning the full delivery path, from provider through messaging queue, internal processing, and status store, with rendered email HTML stored in-house. This end-to-end visibility strengthens support and debugging workflows.
The migration scope is strictly transactional, service, and survey email with high but predictable baseline volume and large event-driven spikes. SES is purpose-built for this pattern, with configurable IP warm-up strategies and the flexibility to choose between standard and managed dedicated IPs.
Finally, SES integrates natively with AWS metrics, notifications, queues, and storage, allowing monitoring, alerting, and failure handling to follow the same patterns used elsewhere in the Fanatics stack.
“If SES works for Amazon at scale, I figured it would work for us. We had also seen SES handle our load before, during a failover from our primary provider on a shared IP setup. That gave us the confidence to commit early and design around it.”
Platform architecture
The Fanatics Commerce team designed their email platform with the same engineering rigor they apply to their commerce systems. The architecture reflects a technology first approach to email operations.
Figure 1 — Fanatics Commerce transactional email platform on Amazon SES
Application layer
The Fanatics application connects to Amazon SES through IAM role-based authentication, with no stored credentials anywhere in the pipeline. This approach simplified security management and eliminated credential rotation as an operational concern.
Managed dedicated IPs
Fanatics started with dedicated IPs and pivoted to managed dedicated to let SES handle IP warming and management. Managed IPs let Amazon SES handle reputation optimization automatically, adjusting sending patterns, warming new IPs, and responding to reputation signals without manual intervention. This was a deliberate engineering decision: the team wanted to invest their time building great fan experiences, not managing IP reputation.
“We started with standard dedicated IPs and managed warming ourselves. Reputation management at our scale became more challenging than we wanted to own, so on AWS architects’ recommendation we moved to managed dedicated IPs. We would rather have our engineers enhancing the fan experience than tuning IP reputation.”
Domain and subdomain strategy
The domain architecture reinforces sender reputation through isolation. Transactional email sends from a dedicated subdomain with its own DKIM signing, SPF records, and DMARC policy. This ensures mailbox providers evaluate transactional email reputation independently, protecting the deliverability of order confirmations and shipping notifications regardless of what other email streams do.
Multi-tenant email design
The team designed a multi-tenant architecture that separates email streams into distinct tenants with independent configuration sets, dedicated IPs, and domain strategies. Each tenant maintains its own reputation, its own IP warming schedule, and its own deliverability metrics. If one tenant has reputation challenges, that specific tenant will be paused without disrupting other tenants.
This isolation is a core design principle. Transactional email, the email fans depend on, runs in its own tenant with dedicated infrastructure. Commercial email operates in a separate tenant. The architecture ensures each stream scales independently and maintains its own deliverability profile.
Real-time observability
Amazon SES Virtual Deliverability Manager (VDM) gives the Fanatics Commerce team a real-time, centralized view of key deliverability metrics including open rates, bounce rates, and complaint rates at the tenant or configuration set level. With VDM, the team is able to spot deliverability shifts early, diagnose issues with confidence, and take action before fans ever notice a problem in their inbox.
Scaling with the seasons
Sports merchandise is inherently seasonal. The platform needed to handle volume swings, from baseline traffic to peak holiday and playoff demand, without degrading deliverability.
During the 2025 holiday season, the platform scaled sending volume by 48x in five months, from initial rollout to full peak capacity across Black Friday, Cyber Monday, and the holiday gifting season. The architecture handled this surge while preserving deliverability, demonstrating that the multi-tenant design and managed dedicated IPs absorb seasonal spikes while maintaining consistent inbox placement rates.
The team phased their rollout by email type and volume, monitored deliverability metrics at each stage, and adjusted sending patterns based on real time feedback from mailbox providers. As the volume scales rapidly, this methodical approach ensured deliverability remained high.
The partnership model
This platform succeeded because of the partnership between Fanatics Commerce and AWS. The engagement brought together an account team TAM, a Solutions Architect, and a Worldwide Specialist SA, each contributing a different perspective.
The TAM coordinated the engagement by connecting Fanatics Commerce engineering with AWS specialists and driving architecture reviews from initial planning through peak holiday season scale.
Rajat Banerjee, Senior Manager of Engineering at Fanatics Commerce, led this initiative end to end from platform design all the way through production rollout. He and his team designed the domain and subdomain strategy that protects sender reputation across brands and ran a phased migration that scaled sending volume to full peak capacity without any disruption to delivery. SES built real-time analytics and reporting pipelines that give team visibility into delivery rates, bounces, and engagement. That visibility transformed incident response and helped the team optimize sending behavior at scale.
This model, customer engineering plus a cross-functional AWS team, accelerated decisions and shortened the feedback loop between architecture questions and production answers. The team had direct access to SES product expertise whenever they needed it, which enabled them to make timely informed decisions.
What’s next
Tenant-level isolation within SES – Handling each tenant’s sending, reputation, and operational signals independently end to end.
Deep linking from transactional emails into the Fanatics mobile app so fans can tap a link in an order or shipping email and land directly on the right screen in the app instead of the web.
Conclusion
Fanatics Commerce set out to build an email platform that matches the speed and reliability fans expect from the brand. By choosing Amazon SES and investing in purpose-built architecture, multi-tenant isolation, managed dedicated IPs, domain-level reputation control, and real-time observability, the team eliminated the operational trade-offs that come with scaling large email systems.
The results speak for themselves: the platform scaled sending volume 48x in five months, maintained high inbox placement rates through peak holiday and playoff demand, and gave the engineering team the visibility to diagnose and resolve deliverability issues in minutes rather than days.
More importantly, this platform frees the Fanatics Commerce team to focus on what matters most, building great fan experiences rather than managing IP reputation and chasing deliverability problems. Every order confirmation that lands in a fan’s inbox on time is a moment of trust earned.
Whether you’re sending millions of emails or only beginning to outgrow your current setup, the patterns in this post apply at any scale. Start by identifying where your current email infrastructure makes you choose between deliverability and growth. Amazon SES is built so you don’t have to.
For legal and compliance teams, auditing a repository of thousands of contracts for a single regulatory obligation shouldn’t take weeks. But with keyword search, it often does. A search for “inadvertent access notification” returns exact matches while missing functionally equivalent clauses such as “security incident disclosure” or “unauthorized access reporting.” This creates two problems:
Discovery gap: Critical risk exposure goes undetected because keyword search cannot match semantically equivalent terms across different contracts.
Review latency: After finding relevant contracts, legal counsel must manually scan lengthy documents to locate the specific clauses that matter. This process can stretch from minutes to hours per document.
Amazon OpenSearch Service is a fully managed search and analytics service that configures, manages, and scales OpenSearch clusters in the AWS Cloud. It supports use cases from log analytics and application monitoring to full-text search and real-time security analytics. It also supports AI-powered semantic search.
Amazon OpenSearch Service addresses both problems through two capabilities:
Semantic search retrieves contracts based on meaning rather than exact keyword matches, closing the discovery gap.
Semantic highlighting pinpoints the exact clauses within retrieved contracts that answer the query, reducing review time from hours of manual scanning to seconds of targeted reading.
In this post, you build a contract compliance search system that combines semantic search with semantic highlighting in Amazon OpenSearch Service. You deploy the solution using two AWS CloudFormation stacks, test it with synthetic contract documents, and see how a single query surfaces both the right contracts and the right clauses within them.
Solution overview
The solution uses a two-stage retrieval and extraction pipeline. First, semantic search identifies relevant contracts across the repository. Then, semantic highlighting marks the specific clauses within those contracts that match the query intent.
The following diagram illustrates the solution architecture:
Upload contracts toAmazon Simple Storage Service (Amazon S3) – Contract documents (JSON format) are uploaded to an Amazon S3 bucket, which serves as the centralized document repository.
Amazon OpenSearch Ingestion (OSI)reads from S3 – A serverless OSI pipeline detects new documents in the S3 bucket and reads them for processing.
OpenSearch ingest pipeline generates embeddings through Amazon Bedrock – As documents arrive, the ingest pipeline’s text_embedding processor invokes Amazon Titan Text Embeddings V2 through an ML Commons Bedrock connector. This converts contract text into 1024-dimension vector representations, stored in a k-NN index that uses the faiss engine.
User submits a search query – A user queries the system with natural language (for example, “data protection regulations”) through a test AWS Lambda function that forwards the request to OpenSearch using the neural query type.
OpenSearch generates the query embedding – OpenSearch converts the user’s natural language query into a vector embedding using the same machine learning (ML) Commons Amazon Bedrock connector and Amazon Titan V2 model.
Amazon OpenSearch Service performs semantic search – OpenSearch uses k-NN vector similarity to retrieve contracts that are semantically relevant to the query, even when exact terminology differs.
Amazon SageMaker AI performs semantic highlighting – The opensearch-semantic-highlighter-v1 model, hosted on an Amazon SageMaker AI GPU endpoint, scores sentence relevance using cross-encoder inference and wraps the matching clauses in <em> tags for targeted reading.
How semantic search and semantic highlighting work together
The system processes queries in two steps:
Step 1 – Semantic search (document discovery): You query the contract corpus using natural language. The system retrieves contracts with semantically similar concepts, even when exact terminology differs. For example, searching for “force majeure” returns contracts discussing “natural disasters” or “unforeseeable circumstances” because the system understands these concepts are related.
Step 2 – Semantic highlighting (clause identification): After relevant contracts are retrieved, semantic highlighting automatically marks the clauses that semantically match your search intent. Instead of scanning pages of legal text, you immediately see the specific paragraphs that answer your question.
Keyword highlighting wraps individual matching words: <em>termination</em> and <em>rights</em>.
Semantic highlighting wraps entire relevant clauses: <em>Upon termination, the consultant must return all confidential information and proprietary materials within 15 business days.</em>.
This reduces false positives, cuts review time, and provides explainability for why each document was retrieved.
Semantic highlighting model deployment
Before the system can highlight clauses based on meaning, the opensearch-semantic-highlighter-v1 model must be deployed to an Amazon SageMaker AI GPU endpoint and registered with the OpenSearch ML Commons plugin through a remote connector.
Stack 2 of the CloudFormation deployment automates this process. It performs the following steps:
Downloads the model artifact from an AWS-managed source and deploys it to an Amazon SageMaker AI endpoint (ml.g5.xlarge).
Creates a remote ML Commons connector in OpenSearch that points to the SageMaker endpoint.
Registers the model with the QUESTION_ANSWERING function so that OpenSearch can use the model’s cross-encoder capabilities to score sentence relevance at query time.
The equivalent manual registration call (handled automatically by the stack) is:
You don’t need to run this manually. The deployment script and CloudFormation stack handle model registration end-to-end. The resulting model ID is automatically passed to the query Lambda function for use in semantic highlighting requests.
Index configuration
The index uses a k-NN vector field with 1024 dimensions (matching the Amazon Titan V2 output) and the faiss engine with HNSW method. The mapping includes both a knn_vector field for semantic retrieval and a standard text field for keyword matching and highlighting. When you search for “liability limits,” OpenSearch first retrieves documents through vector similarity, then uses the Amazon SageMaker AI model to identify and wrap the specific relevant sentences in <em> tags.
This section walks you through deploying the solution using two AWS CloudFormation stacks and two shell scripts. You first set up the core infrastructure (OpenSearch, ingestion pipeline, and ML Commons Bedrock connector), then deploy the semantic highlighting model on Amazon SageMaker AI.
Prerequisites
To deploy this solution, you need:
An active AWS account with permissions to create Amazon S3 buckets, AWS Lambda functions, Amazon SageMaker AI endpoints, Amazon Bedrock model access, Amazon OpenSearch Ingestion pipelines, Amazon OpenSearch Service domains, and AWS Identity and Access Management (IAM) roles (including iam:PassRole and sts:AssumeRole). For the exact least-privilege policy, see iam-deployer-policy.json in the repository. Both CloudFormation stacks require the CAPABILITY_NAMED_IAM acknowledgement.
Amazon Bedrock model access enabled for Amazon Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0).
Estimated deployment time: approximately 35 minutes.
Estimated cost: approximately USD $ 2.00–3.00 for a quick demo. Delete the stacks promptly after testing.
This post uses US East (N. Virginia) as the deployment AWS Region. Verify service availability in your preferred Region before deploying.
Deploy the solution
The solution deploys using two AWS CloudFormation stacks and two shell scripts. The demo includes synthetic contract documents covering common contract types including software licenses, data processing agreements, managed services, and software as a service (SaaS) subscriptions.
Clone the repository and run the deployment script:
git clone https://github.com/aws-samples/sample-contract-compliance-search-amazon-opensearch.git
cd sample-contract-compliance-search-amazon-opensearch
./deploy.sh
The deployment script creates the following resources across two stacks:
Stack 1:
An Amazon OpenSearch Service domain with fine-grained access control.
An Amazon OpenSearch Ingestion (OSI) pipeline that reads contracts from S3 and sends them to OpenSearch for indexing.
An ML Commons Bedrock connector and ingest pipeline that automatically generates 1024-dimension vector embeddings through Amazon Titan Text Embeddings V2 during document indexing.
A test Lambda function for querying the OpenSearch index using keyword, neural, or hybrid search with semantic highlighting support.
An S3 bucket for storing contract documents.
IAM roles for Lambda functions, the OSI pipeline, and OpenSearch access.
Stack 2:
An Amazon SageMaker AI endpoint hosting the semantic highlighting model.
A Lambda function that creates an ML Commons remote connector in OpenSearch and registers the highlighting model.
After both stacks deploy, the script automatically configures OpenSearch (role mappings, Amazon Bedrock connector, embedding model, k-NN index), ingests the sample contract data, and registers the semantic highlighting model.
The total deployment takes approximately 35 minutes to complete.
(Optional) Automated deployment with Claude Code CLI
If you have Claude Code CLI installed, you can deploy the solution using an AI-assisted workflow that creates a least-privilege IAM role scoped to this demo before deploying:
git clone https://github.com/aws-samples/sample-contract-compliance-search-amazon-opensearch.git
cd sample-contract-compliance-search-amazon-opensearch
./scripts/create-deployer-role.sh
export OS_DEMO_DEPLOYER_ROLE=arn:aws:iam::<ACCOUNT_ID>:role/os-demo-deployer-role
export AWS_DEFAULT_REGION=us-east-1
claude "Deploy the OpenSearch semantic search demo following README.md"
Claude Code reads the repository instructions, assumes the deployer role, deploys both CloudFormation stacks in order, runs the setup scripts, and verifies the deployment end-to-end. The deployer role restricts actions to resources prefixed with os-demo-*, following the principle of least privilege.
Test the solution
After the deployment succeeds, follow these steps to test the solution.
On the Lambda console, choose Functions in the navigation pane.
Choose the function that has os-demo-query in its name.
On the Test tab, in the Event JSON paste this keyword search query {"query": "data protection regulations?", "type": "keyword", "k": 3}
Choose Test to run the Lambda function.
The following screenshot shows the Lambda function test configuration on the AWS Management Console with the keyword search query.
The function processes the query in two ways depending on the search type:
For keyword search (enter: keyword): The function sends a standard match query to OpenSearch, which returns documents containing the exact query terms. The highlight fragments wrap individual matching words like <em>termination</em> and <em>rights</em>.
For neural search (enter: neural): The function sends a hybrid query to OpenSearch combining k-NN (semantic similarity) with keyword matching. OpenSearch automatically generates the query embedding through the ML Commons Amazon Bedrock connector using the same Amazon Titan V2 model. This returns semantically related documents even if they don’t contain the exact query terms. The SageMaker endpoint powers the semantic highlighting, identifying the most relevant clauses within each retrieved document. It wraps entire passages like <em>Upon termination, the consultant must return all confidential information and proprietary materials within 15 business days.</em>.
Download the highlight viewer HTML file and open it in the browser. This file helps you view the highlighted text.
Copy the entire execution output of the Lambda execution, paste it into the placeholder in the HTML file, and then choose Load Results.
The following screenshot shows that only the matching keywords are highlighted.
Next, paste the neural search query as input to the Lambda function to see how semantic highlighting works: {"query": "data protection regulations", "type": "neural", "k": 1}
Choose Test to run, and then paste the entire output into the HTML viewer.
The viewer now displays entire sentences highlighted instead of individual keywords.
Optimizing for scale: batch semantic highlighting
In a standard search, a query might return dozens of relevant contracts. Using the default single inference mode, OpenSearch makes a separate ML call for every document in the result set. For a compliance officer reviewing 50 contracts, this sequential processing introduces noticeable latency.
OpenSearch 3.3 introduced batch inference mode to address this. Batch inference collects the matching documents and processes them in a single ML inference call. In the contract compliance use case, this shifts the performance characteristic from multiple sequential roundtrips to a single parallel execution on the Amazon SageMaker AI GPU.
To enable batch inference, first configure the cluster setting:
PUT _cluster/settings
{
"persistent": {
"search.pipeline.enabled_system_generated_factories": ["semantic-highlighter"]
}
}
Then add batch_inference: true to your highlight options. The following query searches for data privacy clauses across the contracts and highlights the top 10 results using a single batch call:
Follow these recommendations to optimize performance, security, and cost-efficiency when deploying the contract compliance search system in production.
Experiment with overlapping chunk sizes (for example, 500 characters with a 10 percent overlap) in your OSI pipeline to verify that context is preserved for long indemnification or liability clauses.
Verify that your Amazon S3 buckets and OpenSearch domains are encrypted using AWS Key Management Service (AWS KMS). For production workloads containing sensitive data, make sure that all traffic stays within your virtual private cloud (VPC) through interface endpoints.
This demo uses simplified configurations for learning purposes. For production deployments, implement VPC isolation, AWS KMS encryption with customer-managed keys, and multi-AZ OpenSearch clusters.
Clean up resources
To avoid ongoing charges, delete the AWS CloudFormation stacks and associated resources:
On the AWS CloudFormation console, choose Stacks in the navigation pane.
Select the os-demo-highlighting stack (Stack 2) and choose Delete. Wait for deletion to complete.
Select the os-demo-search stack (Stack 1) and choose Delete. Stack deletion takes approximately 10–15 minutes to complete.
The stack deletion will automatically remove:
OpenSearch domain.
SageMaker model and endpoint.
Lambda functions.
IAM roles and policies.
After both stacks are deleted, manually delete the S3 bucket (opensearch-cfn-semantic-highlighting-us-east-1-<ACCOUNT_ID>) created for model artifacts. This bucket is provisioned at deploy time and is not managed by CloudFormation. Replace <ACCOUNT_ID> with your AWS account ID in the bucket name.
Conclusion
In this post, you built a contract compliance search system that combines semantic search with semantic highlighting in Amazon OpenSearch Service. The system helps close the discovery gap by retrieving contracts based on meaning rather than exact keywords, and it reduces review latency by highlighting the specific clauses that answer your query.
While we focused on legal agreements, the architecture described here is a blueprint for domains requiring high-stakes document discovery, including:
Regulatory filings: Identifying specific compliance mandates in financial reports.
Technical documentation: Pinpointing troubleshooting steps across massive product manuals.
Research and academia: Isolating specific methodologies within thousands of scientific papers.
Internal knowledge bases: Empowering employees to find exact policy language instantly.
Application-layer distributed denial of service (DDoS) attacks are difficult to detect because they closely resemble legitimate traffic. HTTP request floods are now among the most common vectors targeting web applications, using valid-looking requests that blend in with normal user activity.
In June 2025, AWS launched the AWS WAF Anti-DDoS managed rule group, built specifically for application-layer (L7) DDoS protection. AWS Shield Advanced is adopting it as the default application-layer protection, and in time as the only one. On July 27, AWS Shield Advanced begins adding the Anti-DDoS managed rule group to eligible web access control lists (ACLs) in Count mode. It will not cause any interruption to your traffic alongside your existing L7 automatic mitigation and WAF rules. In this blog post, we provide details regarding the Anti-DDoS managed rule group and when the change is expected to reach your web ACLs. You will understand the phases and steps that you need to take before the finish date, including how the monitoring and metrics will change.
Anti-DDoS managed rule group features
The Anti-DDoS managed rule group builds on what Shield Advanced automatic mitigation already provides. It profiles your traffic, learns what normal traffic looks like for your application, and establishes a baseline in minutes rather than hours. When an attack starts, it reacts within seconds and there are no health checks to configure. The rule group adds a Challenge action to the Block and Count actions you already use. Challenge decisions are driven by the AMR labels that mark the suspicion level of each inspected request. One option is a silent browser challenge, which has a background verification that runs in the visitor’s browser with no interstitial page, so legitimate users are never interrupted while automated traffic is filtered out. You can also exclude workload paths that don’t support Challenge, which fall back to Block mitigations instead. Sensitivity is configurable to Low, Medium, or High, and you set it separately for Block and Challenge. Block and Challenge are tuned independently; meaning you can run Challenge at high sensitivity to catch more suspicious traffic while keeping Block low to avoid dropping legitimate requests or reverse it for a stricter posture.
The rest is about cost and visibility:
It uses less capacity than before. The rule group needs 50 web ACL capacity units (WCUs), down from the 150 the previous protection required, providing you with capacity for the rest of your rules.
The dashboard ships in the AWS Management Console for AWS WAF. It’s there now, showing live DDoS events, match metrics, and the top URIs, geographies, and IP addresses driving traffic.
It labels everything it inspects. Requests get labels for event-detected, graduated suspicion levels, and specific rules. Match on those labels in your own AWS WAF rules when you need logic the rule group doesn’t cover.
You don’t pay for the attack traffic. During active mitigation, blocked DDoS requests drop out of your monthly count. That exclusion covers AWS WAF request fees, Anti-DDoS managed rule group request fees, and Shield Advanced request charges.
AWS Shield Advanced isn’t required to use any of these features. Shield Advanced subscribers get the rule group included with AWS WAF and any customer can turn it on independently. See AWS WAF pricing for more information on costs.
Implementation details
Shield Advanced upgrades application-layer DDoS protection in five phases. The following dates are when AWS will act automatically, not the earliest date when you can act. After the rule group is deployed in Count mode on July 27, 2026, you can begin migrating right away rather than waiting for the October auto-upgrade. There’s no window where protection lapses. Your current automatic mitigation stays active through every phase until the Anti-DDoS managed rule group takes over. That handoff happens in a single operation, with no cutover window and no gap for your traffic flows.
Phase 1: Anti-DDoS managed rule group deployed in Count mode (rolling out July 27–August 7, 2026)
AWS adds the Anti-DDoS managed rule group in Count mode to every web ACL eligible for this rollout. Eligible means any Shield Advanced web ACL with at least one resource using application-layer automatic mitigation that isn’t already running the Anti-DDoS rule group. This is a broader set than the web ACLs eligible for the October auto-upgrade (Phase 3), which applies a stricter test. The deployment rolls out gradually, starting July 27 and expected to finish by August 7, 2026, so different web ACLs might be updated on different days. There’s no impact to your traffic because the rule group watches and labels requests without acting on them while your existing automatic mitigation keeps running. Throughout the evaluation period, you receive DDoS events, metrics, and AWS WAF labels at no additional charge.
Phase 2: Free evaluation period (July 27–September 30, 2026)
The existing automatic mitigation and the Anti-DDoS managed rule group run side by side each detecting independently. Automatic mitigation continues to protect your resources while the rule group operates in Count mode. To compare their detection results, use the DDoSAttackRequests metric, AWS WAF labels, and the Anti-DDoS dashboard. All Anti-DDoS managed rule group charges are waived during this period, including the subscription fee, per-request fees, and WCU consumption costs for the eligible web ACLs from phase 1.
Phase 3: Auto-upgrade begins (October 1, 2026)
For eligible web ACLs, the auto-upgrade mirrors your existing automatic mitigation configuration. The rule group inherits your current setting, so a Block configuration comes up in Block mode and a Count configuration comes up in Count mode in a single, atomic operation. The rule group takes over in the same step that disables automatic mitigation, so protection never drops for an instant. This is a handoff rather than a cutover with no window where your resources are unprotected. If you’d rather not upgrade you can opt out by contacting AWS Support before the auto-upgrade date.
Phase 4: Guided migration (available July 27–December 31, 2026)
You don’t have to wait for the October auto-upgrade to migrate. As soon as the rule group is deployed in Count mode between July 27 and August 7, 2026, you can move to it on your own schedule. This is the path to use for web ACLs that aren’t eligible for the Phase 3 auto-upgrade, meaning mixed-mode web ACLs or ones with resources that don’t have automatic mitigation enabled. Work with your AWS account team and AWS Support at any point in this window to plan and complete the migration. Eligible web ACLs are also upgraded automatically starting October 1 (Phase 3), so guided migration is mainly for the web ACLs the auto-upgrade can’t cover.
As of January 1, 2027, the Shield Advanced application-layer automatic mitigation feature will no longer be available. Resources that haven’t migrated to the Anti-DDoS managed rule group will lose automatic application-layer DDoS mitigation.
The existing automatic mitigation and the Anti-DDoS managed rule group use separate Amazon CloudWatch namespaces and metric structures. The rule group gives you three tiers of observability: tier 1 tells you an attack is happening, tier 2 shows which requests it flagged and why, and tier 3 shows what it did about them. You don’t need all three on day 1 because most customer teams start at tier 1 to confirm detection is working, then add the others as they tune.
Tier 1: Event detection alarms
You can detect DDoS events using two CloudWatch metrics, each with its own namespace.
DDoSDetected (Shield)
DDoSAttackRequests (Anti-DDoS managed rule group)
Namespace
AWS/DDoSProtection
AWS/WAFV2
Requires Shield Advanced
Yes
No
Scope
L3, L4, and L7 events
L7 events only
Value during event
Binary (0 or 1)
Count of requests observed
Value outside event
Reported once daily (keeps metric alive)
Absent (no data points)
Dimensions
ResourceArn
Resource, ResourceType
What this means for your existing alarms:
After the application-layer automatic mitigation feature is sunset, DDoSDetected still fires for infrastructure layer 3 and layer 4 events, so your existing network layer and transport layer alarms remain valid. For the full list, see AWS Shield Advanced metrics.
DDoSAttackRequests is the Anti-DDoS managed rule group equivalent for application-layer event detection. Alarm on Sum >= 1 to detect any event, or set a volume threshold (for example, more than 10,000 requests per minute) for severity-based alerting.
During the evaluation period, both metrics fire independently and you can validate detection parity before migrating your application-layer alarms.
Because DDoSAttackRequests is absent when there are no active DDoS events, set treat-missing-data to missing or notBreaching for alarms on this metric.
Tier 2: Detection labels for custom monitoring
Every request the Anti-DDoS managed rule group evaluates gets a label. Where tier 1 tells you an attack started, tier 2 shows which requests looked suspicious and how confident the rule group was. The labels surface as AWS WAF metrics in the AWS/WAFV2 namespace: AllowedRequests, BlockedRequests, and CountRuleMatch. Each carries the LabelName and LabelNamespace dimensions under the awswaf:managed:aws:anti-ddos: namespace.
event-detected – Requests observed during a detected DDoS event
ddos-request – Requests identified as part of the attack
challengeable-request – Requests eligible for browser challenge
Chart suspicion-level trends on a CloudWatch dashboard to see how an attack builds. Match on the labels in your own AWS WAF rules or dig into them in your AWS WAF logs with CloudWatch Logs Insights or Amazon Athena when you need to understand a specific event after the fact.
Tier 3: Mitigation action metrics
Where tier 2 shows what the rule group flagged, tier 3 shows what it did about those requests during an event. You’ll find these metrics as ChallengeRequests, BlockedRequests, and CountRuleMatch, each scoped by the rule label that produced it.
ChallengeAllDuringEvent – Requests challenged during an active event
ChallengeDDoSRequests – Suspected DDoS requests challenged based on suspicion level
DDoSRequests – Requests blocked (or counted in Count mode)
Watch these during a live event to see whether mitigation is keeping up. If you’re challenging far more requests than you’re blocking, your configuration might be too cautious, and you can raise the sensitivity level after you trust the numbers.
Observability summary
Tier
Automatic mitigation
Anti-DDoS managed rule group
Event alarm
DDoSDetected in AWS/DDoSProtection (binary, L3/L4/L7)
DDoSAttackRequests in AWS/WAFV2 (request count, L7)
Your Shield Advanced subscription includes the Anti-DDoS managed rule group for up to 50 billion requests per month, counted across your whole organization at the payer account level. For most customers that ceiling is well above normal traffic, so you won’t see a line item here unless you’re operating at very high volume. For the exact rates, see AWS WAF pricing and Shield Advanced pricing.
You aren’t charged for DDoS traffic while the Anti-DDoS managed rule group is actively mitigating, which means Block or Challenge mode rather than Count. This applies to AWS WAF request fees, Anti-DDoS managed rule group request fees, and Shield Advanced request charges. Leaving the rule group in Count mode past the evaluation period costs you the protection without the billing relief, so avoid staying in Count mode longer than you need to validate.
During the evaluation period (July 27 to September 30, 2026), the eligible web ACLs AWS auto-enrolled don’t incur per-request fees or WCU consumption, even when configured in Count mode.
The Anti-DDoS managed rule group works at the web ACL level, so every resource you associate with a web ACL shares that coverage. Before assuming a single resource accounts for the whole cost, look at how many resources sit behind each web ACL. A web ACL fronting 20 resources bills differently from one fronting 2, so check that count first and familiarize yourself with the workload protected by each web ACL.
Adding the Anti-DDoS managed rule group to a web ACL yourself isn’t part of the upgrade path, so standard pricing applies from the moment you enable it. The same is true for any resource that was already running the rule group before the rollout. To get the free evaluation, let the automatic rollout reach your web ACLs rather than adding the rule group ahead of it. There’s no penalty for adding it yourself; you just don’t receive the waiver on that web ACL.
Update your infrastructure as code
If you manage web ACLs with AWS CloudFormation, AWS Cloud Development Kit (AWS CDK), Terraform, or other infrastructure as code (IaC), the auto-upgrade changes your infrastructure configuration outside your templates. Your code is still the source of truth, so you need to do two things. First, change where the protection is declared. Today you enable application-layer automatic mitigation through the Shield API (EnableApplicationLayerAutomaticResponse), configured per protected resource. The Anti-DDoS managed rule group is configured through the AWS WAF API instead (CreateWebACL and UpdateWebACL), as a managed rule group statement inside the web ACL, scoped per web ACL rather than per resource. In IaC terms, you remove the Shield automatic-response block (for example, Terraform’s aws_shield_application_layer_automatic_response) and add the WAF managed rule group statement shown in the following section. Second, pull the upgraded web ACL back into your tooling before your next deploy, or your pipeline will try to revert the change.
For the full statement in Terraform, CloudFormation, and the AWS CDK, plus how to sync state after the auto-upgrade (terraform plan, CloudFormation drift detection, cdk diff), see the iac-webacl-examples helper.
Update your AWS Firewall Manager policy
If you run a Shield Advanced policy in AWS Firewall Manager today, that policy is what enabled application-layer automatic mitigation across your accounts. To keep that protection, add the Anti-DDoS managed rule group to an AWS WAF Firewall Manager policy. Your Shield Advanced policy still handles L3 and L4, while the application-layer piece moves to the AWS WAF policy. The migration is straightforward: add or reuse an AWS WAF Firewall Manager policy, put the Anti-DDoS managed rule group in it, and scope it to the same accounts and resources your Shield Advanced policy covers.
You can’t add the rule group from the Shield console or by editing an account-level web ACL directly, because Firewall Manager owns the web ACLs it creates and overwrites local edits. Instead, add the rule group to the AWS WAF policy and Firewall Manager pushes it to every in-scope account.
You can make this change in the console or as code. If you manage your Firewall Manager policies as code, don’t edit them in the console: add a new AWS WAF policy or update an existing one in your templates with the Anti-DDoS managed rule group included, and deploy it using the following Firewall Manager policies using IaC steps. Otherwise, use the console.
In the console, follow Creating an AWS Firewall Manager policy for AWS WAF to create the policy and reach the Edit policy rules page. Add the Anti-DDoS rule group, listed there as AWS AntiDDoS Protection for Layer 7 attacks (AWSManagedRulesAntiDDoSRuleSet), as a new rule group under First rule groups so it evaluates before your other managed groups, but below any Allow custom rules you use to fast-path known-good traffic.
If you protect CloudFront distributions, make this change in your Global policy, and repeat it in each AWS Regional policy for regional resources. Save the policy, and Firewall Manager rolls the change out to in-scope accounts, which can take a few minutes.
After being added, the rule group appears as the first rule group in the policy, as shown in the following screenshot:
Figure 1: AntiDDoS enabled
Firewall Manager policies using IaC
If you manage Firewall Manager policies as code, make the change in your template instead of the console. The Anti-DDoS managed rule group goes into the AWS WAF policy’s ManagedServiceData, a WAFV2 policy definition carried as a JSON string, added to the first rule groups so it evaluates early. For the ManagedServiceData JSON with CloudFormation, Terraform, and AWS CDK examples, see the firewall-manager-examples helper.
Whichever path you take, scope the policy to the same accounts and resources your Shield Advanced policy already covers, so no resource loses application-layer protection during the move.
Getting started
Between July 27 and August 7, 2026, AWS will add the Anti-DDoS managed rule group in Count mode to Shield Advanced web ACLs that have resources using application layer automatic mitigation but not yet the Anti-DDoS rule group. After it reaches your web ACL, you can evaluate it, and migrate whenever you’re ready, without waiting for the October auto-upgrade.
Review the Anti-DDoS dashboard in the AWS WAF console. The dashboard shows real-time DDoS events, match metrics, and top traffic sources.
Compare event detection side by side. During Count mode, both systems detect independently. Check the DDoSDetected metric in AWS/DDoSProtection alongside DDoSAttackRequests in AWS/WAFV2 to validate detection parity for your resources. You can deploy the CloudWatch comparison dashboard from the AWS Samples repository to view both systems on a single dashboard.
Explore AWS WAF labels. Enable AWS WAF logging and query for labels in the awswaf:managed:aws:anti-ddos: namespace. Look at suspicion levels (low-suspicion-ddos-request, medium-suspicion-ddos-request, high-suspicion-ddos-request), event-detected, and challengeable-request to see per-request visibility into detected events.
Start with Low sensitivity for Block actions during evaluation to minimize false positive risk. Tune up as you gain confidence from the Anti-DDoS dashboard and AWS WAF label data.
Plan your configuration. Review sensitivity levels, URI exemptions for non-HTML paths, and web ACL priority placement. The Anti-DDoS managed rule group should run at the highest priority in your web ACL, or right below any custom rules with the Allow action.
Sync your IaC templates. After the auto-upgrade adds the Anti-DDoS managed rule group to your web ACL, fetch the current state into your IaC tooling (Terraform refresh, CloudFormation drift detection, AWS CDK import) before your next deployment.
Conclusion
The Anti-DDoS managed rule group profiles your traffic within minutes and mitigates within seconds, where the automatic mitigation it builds on established its baseline over hours, and it gives you granular visibility into what it’s doing. The evaluation period exists so you can watch both systems run on your own traffic before anything changes. Spend the first few weeks in Count mode confirming the new detection matches what you see today, then move your alarms over and pick a sensitivity level you’re comfortable with. If you run a web ACL across several resources, or you manage rules through AWS Firewall Manager, contact AWS Support before you start so you don’t have to unwind anything later. The Shield Advanced application-layer automatic mitigation feature retires on January 1, 2027, and anything still relying on it needs to be migrated by then.
The new Amazon GuardDuty investigation agent (now in public preview) investigates security findings across your Amazon Web Services (AWS) environment, reducing investigation time from hours to minutes.
GuardDuty is our managed threat detection service that continuously monitors your AWS accounts and workloads for suspicious, potentially malicious activity, and unauthorized behavior, delivering detailed security findings for visibility and remediation.
Whether you’re investigating a single suspicious finding or assessing security posture across your entire organization, the investigation agent provides structured assessments providing risk levels, confidence scores, and actionable recommendations.
Security teams can spend hours investigating security findings and correlating data across multiple tools. The GuardDuty investigation agent automates this correlation, providing actionable intelligence, built directly into GuardDuty and accessible on demand through the AWS Management Console, AWS Command Line Interface (AWS CLI), AWS APIs, or AWS SDKs.
This post shows you how to:
Enable the investigation agent in your GuardDuty console.
Create your first investigation through the console or AWS CLI.
Use the investigation agent with the AWS MCP server for AI-assisted security operations
Key features of the GuardDuty investigation agent
The GuardDuty investigation agent provides APIs using the same patterns you already know from GuardDuty. Each completed investigation returns a risk level, confidence assessment, MITRE ATT&CK®technique mapping, resource mapping, and prioritized recommendations.
You can scope investigations from the console for a specific finding, an account, or all accounts across your organization. Alternatively, the AWS CLI and API accept a free-form trigger prompt of up to 2,048 characters, so you can describe what to investigate in natural language and guide the analysis of the agent by specifying areas of concern, suspected root causes, or priorities for the investigation.
The investigation agent APIs are also available through the official AWS MCP server, part of the Agent Toolkit for AWS, enabling integration into your existing security toolchains and AI-powered workflows. You don’t need to manage or interact with the agent directly. Call API endpoints, and the agent investigates findings, correlates evidence, and delivers an assessment without the overhead of managing complex configurations.
How the investigation agent analyzes findings
When you create an investigation, the agent uses cross-Region inference to process your findings based on scope and produces a structured output.
Cross-Region inference – GuardDuty investigation uses the Cross-Region Inference Service (CRIS), which selects the optimal AWS Region within your geography to process the investigation assessment. Your data remains stored only in the Region where the investigation request originates. However, investigation data and summary results might be processed outside that Region. Data is transmitted encrypted across the secure network provided by Amazon.
For more information about which inference Regions your request might be routed to see the Cross-Region inference routing table located in the investigation section of the Amazon GuardDuty User Guide.
Investigation output – Each completed investigation produces the following insights: Risk level (Info, Low, Medium, High, or Critical), Confidence (Unknown, Low, Medium, or High), Summary (description of findings and key observations), Investigation Details (additional context), and Recommended Actions (detailed actions including AWS CLI commands).
Account scoping – Account specification is required only when investigating a specific member account. For broaderscopes such as your entire organization, no account ID is needed. The agent will only investigate findings within accounts you’re authorized to access per the authorization model that follows.
Prerequisites
Before you get started, make sure you have the following prerequisites in place:
Amazon GuardDuty enabled in your account
AWS account in a supported Region (see Availability section)
Required IAM permissions
You will need three new permissions: guardduty:CreateInvestigation to start new investigations, guardduty:GetInvestigation to retrieve results, and guardduty:ListInvestigations to view investigations for a given detector.
Administrator accounts can create investigations, retrieve results, and view investigation lists for themselves and their member accounts. Member accounts can only retrieve results and view investigation lists for their own account. Member accounts can’t create investigations and can’t access investigations belonging to other accounts or the administrator account. Account specification is required only when investigating a specific member account. For your own account or accounts across your organization, no account ID is needed.
To enable and create your first investigation
Before you begin, verify you have the required IAM permissions as described in the prerequisites .
Open the AWS Management Console in the desired supported Region and navigate to Amazon GuardDuty.
In the navigation pane, choose Investigations.
Figure 1: GuardDuty investigation dashboard
If investigations aren’t enabled choose Go to Settings and then enable investigations by choosing Enable.
After investigations are enabled, navigate back to the investigations page.
In the navigation pane, choose Initiate Investigation.
Figure 3: GuardDuty initiate investigation
Select a scope for your investigation:
Enter a GuardDuty Finding ID: Use when you want to investigate a specific GuardDuty finding in depth
Enter an AWS Account ID: Use when you want to assess the overall security posture of a specific AWS account
All accounts: Use for organization-wide security assessment or when investigating potential lateral movement
Choose Initiate investigation.
Figure 4: GuardDuty investigation setup
Wait for the investigation to complete (typically 2–5 minutes for account level and 10–12 minutes for specific finding investigations during preview). The status updates automatically.
When the investigation completes, select the investigation title to view the full assessment.
Figure 5: GuardDuty investigation completed menu
The investigation assessment contains detailed information about the investigation including general information, a summary of the investigation, mapping, assessment of the threat, and recommended actions.
The General Information section displays the investigation ID, status, triggered-by account, and creation timestamp.
Figure 6: General information section of the assessment
The summary section provides a narrative of key observations and findings.
Figure 7: Summary section of the assessment
The mapping section shows attack techniques and affected AWS resources.
Figure 8: MITRE ATT&CK mapping section of the assessment
The Threat Assessment section displays the risk level, confidence score, and detailed threat analysis.
Figure 9: Threat assessment section
The Recommended Actions section lists prioritized remediation steps.
Figure 10: Recommended actions section of the assessment
Investigations can also be conducted with the AWS CLI or SDK using the following API endpoints:
CreateInvestigation – Initiates a GuardDuty investigation that automatically analyzes security findings, correlates related activity, performs account-level analysis, and produces a structured investigation summary with recommended next steps.
GetInvestigation – Retrieve the status and results of a specific investigation, including the assessment from the agent, correlated evidence, and recommended actions when completed.
ListInvestigations – View investigations across your environment with filtering and pagination.
To run investigations using the AWS CLI
Investigations are asynchronous because the agent queries multiple data sources, correlates findings across services, and performs AI-based analysis. After creating an investigation, you’ll need to check its status periodically until it completes.
Step 1: Find your detector ID
Each GuardDuty deployment has a unique detector ID per-account and per-Region that identifies your specific GuardDuty configuration. You will need this for all AWS CLI operations, especially if you have GuardDuty enabled in multiple Regions. You can find your detector ID in the GuardDuty console under Settings, or by running the following command and specifying the Region. For example, if the GuardDuty detector of interest were in the us-east-1 (N. Virginia) Region
To investigate findings across an entire organization:
aws guardduty create-investigation –-region=us-east-1 \
--detector-id 12abc34d567e8fa901bc2d34eexample \
--trigger-prompt “Investigate findings across my AWS Organization”
Step 3: Check investigation status
Check the status of the investigation shown here using the AWS CLIquery command to filter and list only the Status section of the output for simplicity:
Timing –Investigation times can very. Checking status every 30 seconds should be sufficient to yield results.
If status shows FAILED –Review the error message in the response and verify your permissions match the authorization model requirements.
To list all investigations for a given detector run the following, the max-results command is optional but useful to filter the number of returned results.
Beyond running investigations manually, the API-first design addresses a common customer pattern: sending GuardDuty findings to third-party tools. You can now add automated investigation to those existing pipelines, so your team receives enriched, prioritized intelligence rather than raw alerts.
Consider a customer that routes GuardDuty findings through Amazon EventBridge to their Security Information and Event Management (SIEM) platform, where analysts manually investigate each alert. With the investigation agent, an AWS Lambda function can be placed into the pipeline that calls CreateInvestigation with the finding ID, waits for completion, and forwards the enriched results (risk level, confidence score, MITRE ATT&CK mapping, and recommended actions) to their SIEM alongside the original finding. Critical findings route directly to the customer incident response queue for further analysis or automation. Low-risk findings with high confidence get auto-closed or batched for weekly review. The analyst’s time shifts from repetitive log correlation to validating assessments and acting on confirmed threats.
This pattern works with SIEMs, ticketing systems, or automation platforms that can be customized to use the API or EventBridge messaging. The investigation agent fits into the pipeline as a processing step, not a destination.
The agent is fine-tuned on investigating GuardDuty findings. It’s distinct from other AWS frontier agents such as the AWS Security Agent and AWS DevOps Agent. The scope of the investigation agent is focused to deliver specialized analysis of GuardDuty findings.
Integration with the AWS MCP server
The Model Context Protocol (MCP) is an open standard that allows AI assistants to securely connect to external data sources and tools. Because the AWS MCP server implements this standard for AWS services, you can use it to add GuardDuty investigations into AI-powered workflows using tools like Kiro, Anthropic’s Claude, or other MCP-compatible clients.
To configure the AWS MCP server
Configure your MCP client to connect to the AWS MCP server.
Use natural language to invoke investigations (for example,“Investigate the recent Unauthorized Access finding for account 123456789012″).
Review the investigation results returned through your MCP client. These results can vary depending on the model or agent being used, configuration, and the non-deterministic nature of AI.
Integrate the results into your existing agent automation or take manual action based on the findings.
Additional usage examples
“Investigate the latest high-severity finding in my production account”
“Create an investigation for finding ID abc123 in account 987654321098 and summarize what happened”
“List investigations from the last 24 hours and flag those that need human review”
How the investigation agent relates to AWS Security Incident Response
At re:Invent 2024, AWS launched AWS Security Incident Response (AWS SIR), a managed service that you can use to quickly prepare for, respond to, and recover from security incidents. AWS SIR and the GuardDuty investigation agent address different stages of your security workflow. The GuardDuty investigation agent provides an on-demand assessment capability. When your team needs deeper context on a specific finding, an account security posture, or the overall security posture of your organization. You create an investigation and receive a structured assessment with risk levels, confidence scores, MITRE ATT&CK® technique mappings, and actionable recommendations. Security analysts can use this to quickly understand the scope and severity of what GuardDuty has detected.
When you create an AWS-supported case through AWS SIR, a SIR investigation agent activates, working in parallel with AWS Security Incident Response engineers to gather evidence and deliver an investigation summary within minutes. AWS SIR is purpose-built for active security events where you need both AI-powered automation and human expertise to coordinate containment and recovery.
Security teams can use these capabilities to assess and prioritize findings on demand using the GuardDuty investigation agent, escalate confirmed issues to stakeholders with supporting evidence, and create or update an AWS-supported case to accelerate involvement from the AWS SIR team when additional support is needed.
Availability and pricing
Public preview of the GuardDuty investigation agent is available in 10 AWS Regions including US East (N. Virginia), US East (Ohio), US West (Oregon), Canada (Central), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), Europe (Stockholm), and Asia Pacific (Tokyo).
During public preview, the investigation agent is available at no charge. Usage is limited to 10 investigations per account per day, with a cumulative limit of 100 investigations per account during the preview period. Failed investigations do not count toward these quotas.
Start investigating findings today
The Amazon GuardDuty investigation agent reduces investigation time from hours to minutes, letting your security team focus on confirmed security events rather than manual correlation.
Get started by:
Enabling the investigation agent in your GuardDuty console
Creating your first investigation using a recent GuardDuty finding
Reviewing the structured assessment, including risk level and recommended next steps
For organizations using the AWS MCP server, you can also invoke investigations through natural language in your AI assistant of choice.
Organizations running large-scale Apache Spark workloads often face a trade-off between achieving lower cost and job reliability. These tradeoffs are more prominent when using Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances or when their jobs process highly skewed datasets. Three shuffle-related challenges drive the pain:
Spot interruptions trigger costly recomputation: Spot instances can reduce compute spend by up to 90 percent compared to On-Demand instances, but they can be reclaimed with only two minutes of notice. When a Spark executor on a Spot Instance is interrupted, its local shuffle data is lost, and Spark must recompute entire upstream stages to regenerate that data. For shuffle-heavy jobs processing terabytes of data, frequent interruptions cause cascading recomputation and runtime delays, quickly eroding the savings that made Spot attractive in the first place.
Local shuffle storage causes cluster-wide over-provisioning: In YARN-based Hadoop architectures, including Amazon EMR on EC2, the External Shuffle Service (ESS) stores shuffle data locally on each Node Manager’s node alongside the Spark executor that produced it. Every node must carry large memory and disk allocations to accommodate shuffle output, yet only a few EC2 nodes perform most of the shuffle work. The rest sit oversized and underused. This is a classic coupled storage-compute problem. By decoupling shuffle storage to a dedicated, storage-optimized tier, you can right-size your compute for actual demands.
Shuffle data protection leaves compute idle: To guard against local shuffle data loss, Spark’s scaling logic prevents nodes that still hold shuffle data from scaling down. EC2 nodes sit idle long after Spark tasks complete. Data skew amplifies this effect: tail-end tasks run far longer than typical ones, delaying shuffle reads and postponing scale-down across the cluster.
Together, these challenges force a difficult choice: cheaper infrastructure or predictable jobs. In this post, we show how Apache Celeborn resolves this trade-off for Amazon EMR on EKS and Amazon EMR on EC2, improving job reliability while unlocking additional cost savings.
What is Apache Celeborn?
Apache Celeborn is an open-source Remote Shuffle Service (RSS) that solves the preceding problems by decoupling shuffle data from the executor lifecycle entirely. It uses a Leader-Worker-Client architecture: Leader nodes manage metadata, Workers read and write shuffle blocks, and Clients integrate with compute engines. Instead of writing shuffle output to local disks, Spark executors push data to a shared, storage-optimized Celeborn cluster that persists shuffle data independently of executor location. This means EMR executor nodes can run on 100% Spot Instances. Spot reclamations no longer cause shuffle data loss. Executors scale in and out freely without triggering upstream recomputation.
Celeborn also provides Raft-based high availability, per-job data replication, and a pluggable shuffle manager that replaces Spark’s default mechanism with minimal configuration changes. With its push-based model, Spark executors send shuffle data directly to Celeborn workers, which cache and consolidate partitions. This reduces the N×M network connections during the read phase, improving both performance and stability at scale.
Image 1: Push-based Remote Shuffle Service for Spark on EMR
Overview of solution
In this post, we show you how to deploy a Celeborn cluster alongside EMR on EKS and EMR on EC2. The solution also includes an observability stack to provide operational visibility into the Celeborn cluster. Metrics are collected by the AWS Distro for OpenTelemetry (ADOT) collector and routed to two monitoring paths. The AWS managed option uses Amazon Managed Service for Prometheus and Amazon Managed Grafana. The open source option uses self-managed Prometheus with a built-in Grafana.
Apache Celeborn can be deployed in several ways depending on your operational requirements and scale. These two deployment patterns are the main ones:
Co-located on the same cluster: Celeborn runs on the same compute environment as Spark. This is the most straightforward operational model, with no cross-cluster networking. The main constraint is shared cluster lifecycle: any upgrade or termination affects Celeborn and running Spark jobs simultaneously.
Separate Celeborn cluster: Celeborn runs on its own EC2 or EKS cluster, fully isolated from Spark compute. This is the most operationally flexible model and is the focus of this post.
In this solution, Celeborn runs on a dedicated Amazon Elastic Kubernetes Service (Amazon EKS) cluster, separate from the EMR Spark environments. The two workloads have different resource profiles. Celeborn is storage and network I/O intensive, while Spark is CPU and memory intensive. By separating them, each cluster can use instance types optimized for its workload. It also improves independent lifecycle management, so you can upgrade or scale Celeborn clusters without disrupting Spark jobs, and the other way around.
Image 2: Solution Architecture
As the architecture diagram shows, two types of EMR deployment models, EMR on EKS and EMR on EC2, connect to a shared Celeborn cluster through an internal Network Load Balancer (NLB). Behind the scenes, Spark executors register their shuffle partitions to Celeborn workers through this connection, while reducers read consolidated data back during the fetch phase.
The following are the key design considerations for this solution:
Streamlined operation by a shared RSS model: Celeborn runs on a dedicated EKS cluster. This provides lifecycle independence, allows each cluster to use workload-optimized instance types, and allows a single Celeborn cluster to serve multiple EMR clusters as a shared service.
Cross-cluster connectivity: Clusters reside in the same Amazon Virtual Private Cloud (Amazon VPC) and share private subnets. The AWS Load Balancer Controller on the Celeborn cluster provisions an internal NLB exposing its active primary pods on ports 9097 (RPC) and 9098 (dashboard). The NLB DNS name is VPC-resolvable, so any EMR cluster in the same VPC can reach Celeborn by setting spark.celeborn.master.endpoints to the NLB address.
Restricted and secured networking: The Celeborn cluster only allows inbound traffic from the EMR on EC2 and EMR on EKS clusters, sending shuffle data and metrics over the network to the Celeborn cluster.
State persistence: Primary nodes maintain Celeborn’s coordination state through Raft consensus, which requires storage that survives pod restarts. Deploying them as StatefulSets with EBS-backed persistent volume claims (PVC) lets a restarted primary pod recover its Raft log and identity from durable storage rather than starting from scratch. Workers keep shuffle data on local NVMe instance store for performance, but this is ephemeral. To protect data loss on workers, each shuffle partition is replicated to two other workers by setting spark.celeborn.client.push.replicate.enabled=true.
Observability: ADOT collector is deployed on the Celeborn EKS cluster. It scrapes Prometheus metrics from Celeborn’s pods, and simultaneously remote writes them to two monitoring backends. Option 1 uses Amazon Managed Service for Prometheus as the metrics store, with Amazon Managed Grafana surfacing pre-built dashboards for Celeborn cluster health and Java Virtual Machine (JVM) metrics. This option requires AWS IAM Identity Center. Option 2 deploys a self-managed Prometheus stack with a built-in Grafana on the Celeborn EKS cluster, with Prometheus configured as a remote-write receiver for the same ADOT collector. This option suits environments without IAM Identity Center or those preferring a single open-source tooling.
Critical configurations
The following two tables list the key configurations that make up the solution.
Spark configuration (RSS client): Tells the Spark client to use Celeborn as the shuffle manager in place of Spark’s built-in implementation.
Celeborn configuration (RSS server): Controls how the primary and worker Celeborn pods operate on Kubernetes.
Note: The values in the following tables are reference defaults used in this walkthrough. Adjust them based on your workload requirements and cluster sizing.
1. Spark configuration (RSS client)
The following table highlights some key Spark configurations required to use Celeborn as the shuffle manager. You can refer to these configurations applied for each Spark submission method in the scripts below:
Registers Celeborn’s data I/O plugin alongside the shuffle manager
spark.celeborn.client.push.replicate.enabled
true
Default: false
OPTIONAL: if shuffle performance has a higher priority than job stability, turn off the data replication at a job level. This setting replicates shuffle data across multiple Celeborn workers for fault tolerance.
COMPULSORY: to make sure we don’t see intermittent writes to local and remote shuffles.
2. Celeborn configuration (RSS server)
The following table provides the server-side settings that control how the Celeborn primary and worker pods operate on Kubernetes. These values are set in the Helm values.yaml file.
Parameter
Value
Purpose
master.replicas
3
Number of Celeborn primary node replicas for HA. A minimum 3 required for Raft quorum
worker.replicas
3
Number of Celeborn workers to store shuffle data, should be less than EC2 node number.
master.volumeClaimTemplates
gp3, 5 GiB
Persistent storage for Raft consensus state
worker.volumes
4 × hostPath (/mnt/nvme/disk1-4)
Shuffle data stored on local NVMe instance store is ephemeral but significantly faster than Amazon Elastic Block Store (EBS) volumes for shuffle I/O
master/worker tolerations
celeborn-dedicated
Schedules Celeborn pods only on dedicated tainted nodes
master/worker podAntiAffinity
preferred, w=100
Spreads replicas across different nodes to limit failure blast radius
image.tag
0.6.2
Pinned Apache Celeborn version
Deploy the solution
This solution contains six layers, each of which is dependent on the previous deployments. See the details in the following deployment steps:
Shared Infrastructure (Step 2).
Celeborn Remote Shuffle Service installation (Step 3).
Prepares sample data and creates EMR compute (Step 4, 5).
Observability Layer (Step 6-7 and Step 9).
Submit job (Step 8).
Cleanup (Step 10).
Note: This walkthrough creates billable AWS resources, including Amazon EKS clusters, EC2 instances, Amazon Managed Grafana, Amazon Managed Service for Prometheus, and a Network Load Balancer. To avoid ongoing charges, follow the cleanup instructions at the end of this post.
Prerequisites
Before you deploy this solution, make sure the following prerequisites are in place:
Clone the repository to your local machine and set the AWS_REGION:
git clone https://github.com/aws-samples/sample-emr-celeborn-shuffle-service.git
cd sample-emr-celeborn-shuffle-service
export AWS_REGION=<AWS_REGION>
Step 2: Deploy the shared infrastructure
This step creates the core AWS resources, including the VPC, AWS Key Management Service (AWS KMS) key, security groups, and Amazon Simple Storage Service (Amazon S3) bucket.
./shared-infra/deploy.sh
Step 3: Deploy the Celeborn cluster
This step provisions a dedicated EKS cluster for Celeborn and exposes it through an internal NLB. It follows security best practices by keeping the EKS endpoint private, allowing public access only from a deployment workstation IP, and enabling encryption for secrets plus logging for all cluster components.
./celeborn/deploy.sh
Step 4: Prepare the sample data
Execute the following script to generate sample data:
./spark-jobs/setup-data.sh
Step 5. Deploy a compute engine (choose one or both)
Create at least one of the following EMR deployment models to run Spark jobs.
Option A: EMR on EKS
./emr-on-eks/deploy.sh
Option B: EMR on EC2
./emr-on-ec2/deploy.sh
Step 6. Deploy observability
To monitor the Celeborn cluster, deploy one of the observability options. After deployment, a Grafana URL and login details are available in the .environment-info file located at the repository’s root directory. Follow the instructions in Step 9 to sign in to your Grafana dashboard. You will see two pre-built dashboards on the Grafana web UI:
Celeborn Cluster Overview: Active shuffles, worker status, disk usage, and memory utilization.
Celeborn JVM Metrics: Heap usage, garbage collection, and thread activity.
Option A: AWS managed services
This option deploys an Amazon Managed Service for Prometheus workspace for metrics storage and an Amazon Managed Grafana workspace to host dashboards. Amazon Managed Grafana requires IAM Identity Center, so first enable it at the organization level and create an SSO user:
This option deploys a Prometheus stack, including a built-in Grafana service, onto the Celeborn EKS cluster in the monitoring namespace.
./observability/prometheus-grafana/deploy.sh
Step 7: Deploy ADOT Collector
The ADOT collector scrapes Prometheus metrics from Celeborn pods and remote-writes them to all active monitoring backends: Amazon Managed Service for Prometheus, open-source Prometheus, or both.
./observability/adot/deploy.sh
Step 8: Submit a Spark job
The sample job is a PySpark word-count application that creates shuffle through groupBy and orderBy operations. Both EMR deployment options below are configured with Celeborn as the shuffle manager.
Option A: Using EMR on EKS
Submit the job using the StartJobRun API:
./emr-on-eks/submit-emr-api.sh
This code snippet shows the core part of the script (see the full version):
Alternatively, submit the job using a Spark Operator:
./emr-on-eks/submit-spark-operator.sh
The script automatically generates a SparkApplication manifest then applies it to EMR on EKS. For example, kubectl apply -f your-job-manifest-name.yaml
Option B: Using EMR on EC2
Submit the job as an EMR Step through the Steps API:
./emr-on-ec2/submit-job.sh
The following snippet shows the core API call used by the script:
Step 9: Review the Grafana dashboard for remote shuffle metrics
The Grafana endpoint is dynamically generated at deploy time and is unique to your deployment. To access it, open the .environment-info file at the repository root. This file contains the Grafana URL along with login instructions. Sign in using the credentials listed there, then navigate to the Celeborn Cluster Overview dashboard to observe remote shuffle metrics in real time. A sample Grafana dashboard screenshot is shown below:
Image 3: Grafana dashboard for Celeborn Metrics
Step 10. Cleaning up
To avoid incurring future charges, run the cleanup script from the root directory:
./teardown.sh
This script detects which components are deployed and tears them down in reverse dependency order, automatically skipping components that are not present. Shared infrastructure is always deleted last, since other components depend on it.
WARNING: This will permanently remove all resources created previously, including any data stored in S3 buckets and configurations. The action cannot be undone. Make sure you have backed up any data you wish to retain before proceeding.
Considerations for production implementation
This post demonstrates a working end-to-end architecture for integrating Celeborn with Amazon EMR. Before taking this pattern to production, consider the following areas.
1. Security
The following considerations help you secure a Celeborn deployment across data isolation, encryption, and access control.
1.1. Shuffle data isolation between teams
Celeborn partitions shuffle data by application ID, which is designed to prevent jobs from accidentally reading each other’s data. This is sufficient when all jobs share the same trust boundary. For multi-team deployments where data privacy is required, a dedicated Celeborn cluster per team is the most effective isolation boundary: each team gets its own NLB and security group, and shuffle data never co-mingles at the infrastructure level.
1.2. Data in transit
In our implementation, shuffle data travels over plain TCP between Spark executors and Celeborn workers. Access is restricted to nodes using security groups. The internal NLB isn’t reachable outside the VPC, and security group rules block all other intra-VPC traffic.
For workloads requiring encryption in transit, Celeborn supports TLS on both RPC and data channels through the celeborn.ssl.* configuration. This is a cluster-wide setting that applies to all jobs on the cluster and must be enabled server-side in celeborn-defaults.conf.
1.3. Data at rest
EBS volumes (used for Celeborn leader pod Raft state) are encrypted with the shared AWS KMS key. NVMe instance store volumes on Celeborn worker nodes are ephemeral and not encrypted by default. Shuffle data written to NVMe is not protected by AWS KMS. For compliance requirements mandating encryption at rest, consider using EBS-backed worker storage instead of instance store, where worker pods mount PersistentVolumeClaim (through volumeClaimTemplates) that reference the encrypted gp3 StorageClass. This comes at the cost of lower I/O throughput compared to local NVMe.
1.4. EKS secrets encryption
EKS clusters encrypt Kubernetes secrets at rest using AWS KMS (EncryptionConfig in the cluster CloudFormation templates). This covers Kubernetes API objects but not application-level shuffle data.
1.5. Application-level authorization
Our deployment enforces access control in the network layer (that is, VPC and security group rules). Celeborn also provides an application-level authorization framework, which is disabled by default. Turning it on adds a second level of security control where only applications presenting valid credentials can register with the cluster. This is recommended for production deployments where multiple workloads or teams share the same VPC subnets, ensuring that network proximity alone does not grant shuffle service access.
Celeborn workers register with a primary node when they start, so scaling out is safe even while the cluster is running. Scaling in, however, requires more care. Removing a worker pod during an active job may cause shuffle data loss unless celeborn.client.push.replicate.enabled=true is enabled. To reduce the risk of accidental disruption during node scale-in or upgrades, add a Pod Disruption Budget to prevent multiple workers from being evicted at the same time.
Spark executors: Dynamic Resource Allocation (DRA) is enabled in sample Spark jobs (spark.dynamicAllocation.enabled=true). This means the number of executors can scale automatically within the configured minExecutors and maxExecutors range.
3. Resiliency
Primary Node HA: 3-replica Raft quorum with EBS-backed durable state is designed to tolerate one leader node failure without job interruption.
Worker replication: the setting celeborn.client.push.replicate.enabled=true copies each shuffle partition to two workers, designed to tolerate a single worker failure mid-job. Without replication, a worker failure causes a fetch failure and job retry.
Pod Disruption Budgets (PDB): not configured in this demo. Add PDBs for Celeborn’s StatefulSets to prevent simultaneous eviction during node upgrades or scale-in.
Multi-AZ placement:podAntiAffinity (preferred, weight 100) spreads pods across nodes and Availability Zones. For strict Availability Zone isolation, switch to requiredDuringSchedulingIgnoredDuringExecution.
4. Monitoring and alerting
Consider extending the Grafana dashboards with the alerting rules on:
Active shuffle partition count: indicator of job load.
Worker disk utilization: prevent shuffle storage exhaustion.
Push failure rate: early signal of connectivity or capacity issues.
In this post, we showed how to deploy Apache Celeborn as a Remote Shuffle Service on Amazon EKS and integrate it with Amazon EMR on EKS and Amazon EMR on EC2. By decoupling shuffle storage from Spark’s compute, this architecture delivers resilience to node failures, eliminates disk contention, and enables independent scaling of storage and compute tiers.
Running Celeborn on a dedicated cluster gives you lifecycle independence from Spark, lets multiple EMR clusters share a single shuffle service, and provides fault tolerance through push-based shuffling, Raft-based high availability, and per-job data replication. It also provides automatic fallback to Spark’s built-in shuffle during maintenance windows. Stop choosing between cost and reliability. With Celeborn on Amazon EMR, you get both.
As AI agents and automated tools increasingly access web applications, distinguishing legitimate bot traffic from malicious attempts has become a critical security challenge. Traditional approaches such as IP-based filtering and reverse DNS lookups fail in multi-tenant systems (such as Amazon Bedrock AgentCore) where thousands of distinct workloads share the same IP space. Attackers can easily spoof user agents, and manual allowlists don’t scale with growing demand.
Web Bot Authentication (WBA), available in AWS WAF Bot Control since November 2025, solves this challenge by implementing cryptographic signatures that provide tamper-proof verification of bot identities. WBA uses asymmetric cryptography to verify that a request comes from an authorized automated agent, relying on two active Internet Engineering Task Force (IETF) drafts: a directory draft for sharing public keys, and a protocol draft defining how keys attach crawler identity to HTTP requests.
With WBA, you can confidently identify trusted automated access while maintaining granular control through WAF labels, creating a more secure and manageable ecosystem for both bot operators and website owners. AWS WAF Bot Control respects WBA verification status by default, automatically allowing verified AI agent traffic.
This post provides a deeper technical guide to implementing WBA with AWS WAF. You learn how WBA works, explore the new labels and capabilities it introduces, and walk through a step-by-step implementation—including signing code—to authenticate bot traffic using cryptographic signatures.
How Web Bot Authentication works with AWS WAF
WBA uses asymmetric cryptography to verify bot identities through HTTP message signatures. The process works as follows:
Bot registration – Bot operators publish their public keys in a signature directory. AWS WAF regularly polls these directories and maintains a valid key registry.
Request signing – Each bot operator’s request is signed using their private key following the IETF standard HTTP Message Signatures (RFC 9421).
Verification – AWS WAF verifies signatures against known public keys associated with the bot operator and appends labels related to verification status.
A typical WBA-signed request includes headers like the following:
The following sequence diagram shows how AWS WAF verifies bot signatures and applies labels for allow or block decisions.
Figure 1 – AWS WAF Web Bot Authentication verification flow
The workflow shown in figure 1 includes the following steps:
A bot sends a signed request to Amazon CloudFront and is inspected by AWS WAF Bot Control
AWS WAF Bot Control retrieves the bot operator’s public key from the signature directory
AWS WAF Bot Control verifies the ed25519 signature
AWS WAF Bot Control appends a verification label (verified, invalid, expired, or unknown_bot)
AWS WAF Bot Control evaluates rules using the label to allow or block the request.
New capabilities added to AWS WAF
With the addition of WBA, the following capabilities were added to AWS WAF.
Cryptographic bot verification
When a bot sends a request, it includes HTTP message signatures that AWS WAF validates at the edge using the AWS WAF Bot Control rule group (version 4.0 and later). This validation process adds minimal latency to requests while providing cryptographic certainty about the bot’s identity. HTTP Message Signatures is an open IETF standard (RFC 9421) that defines a mechanism for signing and verifying HTTP messages using asymmetric keys—in practice, this means a bot cryptographically signs specific headers and metadata of each request, and the receiver can verify the signature using the bot’s published public key.
New labels within AWS WAF for granular control
AWS WAF automatically validates signatures, and successfully validated traffic is immediately marked as verified. This verification status can be used in WAF rules and bot management policies, giving you the ability to write your own rules based on the new functionality.
AWS WAF now automatically allows verified AI agent traffic
AWS WAF Bot Control now respects WBA verification status by default, automatically allowing verified AI agent traffic. This includes two specific behavior changes:
Category:AI rule update – Previously, the Category:AI rule under common Bot Control blocked unverified bots. Bot Control now checks WBA verification status before applying this rule.
TGT_TokenAbsent rule update – The TGT_TokenAbsent rule, which detects requests without a WAF token, no longer matches requests that carry the web_bot_auth:verified label.
Key benefits for AWS WAF customers
WBA with AWS WAF delivers several advantages for organizations managing automated traffic at scale.
Enhanced bot visibility – Clear identification of distinct bots operating from multi-tenant platforms like Amazon Bedrock AgentCore, providing transparency into automated traffic sources. The AWS WAF console includes a new AI activity dashboard that provides a centralized view of AI bot and agent traffic across your protected resources.
Enhanced security – Cryptographic verification of bot identities using industry-standard signing mechanisms.
Reduced false positives – Accurate distinction between legitimate and malicious automated traffic, particularly in shared IP environments.
Industry alignment – Alignment with industry standards and major content delivery network (CDN) providers for consistent bot authentication across platforms.
Customer use cases for WBA with AWS WAF
Across industries, organizations use WBA to grant automated agents secure, controlled access to their web applications. The following scenarios highlight where this capability delivers real-world value:
Verified customer support agents – Authenticate AI-powered chat and support bots so websites can recognize them as approved, registered agents. This enables seamless customer service automation while maintaining security controls and audit trails.
Automated crawling and indexing – Allow search engine crawlers and content indexers to fetch pages with clear identity and scoped permissions. This reduces false-positive blocks, improves crawl efficiency, and helps legitimate bots access your content without triggering security controls.
Partner integrations – Third-party agents can access customer portals and APIs with explicit consent and granular, scoped access controls. This facilitates secure business-to-business (B2B) integrations while maintaining visibility into partner bot activity.
Enterprise automations and agents – Internal automation tools—including monitoring systems, QA bots, continuous integration and delivery (CI/CD) pipelines, and robotic process automation (RPA) solutions—get authenticated access to web applications with least-privilege access principles and full auditability.
Availability
WBA was introduced in Bot Control rule group Version_4.0 (November 2025) for Amazon CloudFront distributions, with continued support in later versions. With Version_6.0, WBA is available for resource types supported by AWS WAF across standard commercial AWS Regions.
Getting started: Developers or agents quick start
Whether you’re implementing WBA yourself or working with an AI coding assistant, the following steps walk you through deploying WBA, signing requests, and writing custom rules.
Step 1: Deploy the WBA-enabled Bot Control
Add the AWS WAF Bot Control rule group to your CloudFront-associated web ACL using static Version_4.0 or Version_5.0—both include WBA support for cryptographic bot verification. Version_5.0 (released February 2026) covers more than 650 unique bots and agents spanning categories including AI search engine crawlers, AI data collectors, AI assistants, and large language model (LLM) training crawlers.
Important: You must explicitly select one of these static versions.
The following example CloudFormation YAML snippet shows a bot control rule set configuration:
# Bot Control rule group with WBA support
ManagedRuleGroupStatement:
VendorName: AWS
Name: AWSManagedRulesBotControlRuleSet
# Use Version_4.0 or higher for WBA support
Version: Version_5.0
ManagedRuleGroupConfigs:
- AWSManagedRulesBotControlRuleSet:
# COMMON level provides WBA verification
# TARGETED level adds additional bot-specific protections
InspectionLevel: COMMON
Step 2: Sign requests from your bot
If your agent runs on Amazon Bedrock AgentCore Browser, request signing is handled automatically—no additional configuration is required.
For agents running outside of AgentCore, registration APIs are on the roadmap that you can use to sign requests independently by:
Alert on – awswaf:managed:aws:bot-control:bot:web_bot_auth:expired
Step 4: Monitor WBA traffic
Use AWS WAF metrics and logs to monitor authenticated bot traffic:
Review Amazon CloudWatch metrics for Bot Control rule group matches and set up alarms for anomalous or unexpected spikes in invalid or expired verification attempts.
Analyze AWS WAF logs to identify patterns in bot authentication attempts and filter on web_bot_auth labels.
Use the AI Activity Dashboard in the AWS WAF console for a centralized view of AI bot traffic. Visualize traffic trends, identify top bots and frequently targeted paths, and filter by verification status to decide which bots to allow, rate-limit, or block.
Conclusion
WBA with AWS WAF provides a cryptographically secure, standards-based approach to authenticating legitimate AI agent traffic. By moving from IP-based allowlisting to signature-based verification, you gain accurate bot identification that works across multi-tenant environments.
Looking ahead, our focus is to simplify bot authentication and make it safer by default. Registration APIs that agent owners can use to cryptographically verify bot identity and intent are on the roadmap, helping website owners quickly distinguish trusted automation from unknown traffic.
If you own an agent, adopt WBA and register your agent to receive verified status. In parallel, AWS continues to actively participate in the IETF web-bot-auth working group, advocating for complementary approaches—using both identifying and anonymous verification protocols—and will incorporate these standards into products as they mature to help your deployments stay aligned with the broader ecosystem.
Marketing teams running large-scale campaigns often send the same message across SMS, WhatsApp, and email regardless of how each customer engages or how many messages they’ve already received that week. This pattern wastes budget on channels customers ignores and pushes promotional content toward frustrated or message-fatigued customers. A MarketingSherpa study found that 45% of consumers who unsubscribe from email marketing cite messages being too frequent as the reason. Over-messaging therefore erodes the audience a brand has paid to acquire. This post shows how to build a campaign orchestrator on AWS End User Messaging and Amazon Bedrock. The orchestrator predicts the best channel for each customer, adapts content per channel, and holds back messages to fatigued or unhappy customers.
In this post, we describe the following capabilities for enterprise marketing teams:
Channel prediction that selects SMS, WhatsApp, or email for each customer based on engagement history
Content adaptation that takes a single campaign brief and produces channel-appropriate variants: a 160-character SMS, a longer WhatsApp template message, and an HTML email
Sentiment-aware suppression that holds back promotional messages when a customer’s stored sentiment score is negative
Frequency tracking across channels that lowers send rate when a customer shows disengagement signal
Natural-language campaign launch that turns a typed instruction such as “Send the Andaman package to Mumbai customers who haven’t booked in six months” into a segmented, channel-routed send
Amazon Bedrock model access granted for an Anthropic Claude model in your AWS Region
(Optional) An Amazon SageMaker AI endpoint for channel prediction. The orchestrator calls the endpoint when it’s configured and falls back to the customer’s stored preferred channel otherwise.
Solution overview
A marketer types a plain-language instruction into the campaign launcher. Amazon API Gateway forwards the instruction to an AWS Lambda function, which starts an AWS Step Functions state machine. The state machine walks the campaign through seven stages. Each stage moves the campaign closer to dispatching the right message on the right channel. The stages read and write customer state in Amazon DynamoDB and call Amazon Bedrock for language tasks. The final stage dispatches messages through AWS End User Messaging or Amazon Simple Email Service (Amazon SES).
When you turn on semantic segmentation, the state machine also queries an Amazon OpenSearch Serverless collection. The collection holds customer embeddings.
To deploy the sample in your account, refer to the GitHub repository.
Figure 1 shows the campaign orchestration system.
Message processing
When a marketer submits an instruction, the launcher Lambda function starts a Step Functions execution. The state machine then runs the stages in order. Each stage reads the output of the previous one, applies its own logic, and passes its result forward. The state machine retries transient failures within a stage, so a Bedrock throttle or a DynamoDB timeout doesn’t restart the whole campaign. A choice state redirects the workflow straight to the recording stage when no customers pass the safety check, so empty campaigns skip the content adaptation step. This decoupled design gives operators three things:
If one stage fails, the workflow retries that stage without rerunning earlier work
You can add new stages — for example, a translation step — without changing the others
The system scales with campaign volume
AI conversation engine
Amazon Bedrock does two distinct things in the orchestrator, and they happen at different stages. The parse stage runs first. It takes the marketer’s plain-language instruction and asks the model to return a small JSON object. The JSON has fields such as location, package, age range, and a short semantic query when the instruction implies a lifestyle or affinity. That JSON is what every downstream stage works against, so the parse output sets the shape of the campaign.The parse stage sends the following prompt to Anthropic Claude on Bedrock through the InvokeModel API:
You parse marketing campaign instructions into structured fields.
Instruction:
{instruction}
Return JSON with these fields:
- "sku" (string or null): product SKU or package name
- "location" (string or null): city or region
- "category" (string or null): one of "Electronics", "Travel", "Apparel", "Home"
- "min_age" (integer or null), "max_age" (integer or null)
- "min_purchases" (integer or null)
- "lookback_days" (integer or null)
- "has_cart_items" (bool or null)
- "semantic_query" (string or null): free-text lifestyle/affinity descriptor
Output ONLY the JSON object, no prose.
For the instruction “Send the Andaman package to budget-conscious families in Mumbai”, the model returns:
The adapt stage runs later, after segmentation and safety. It takes a single campaign brief and asks Bedrock to produce one variant per channel: a 160-character SMS, a longer WhatsApp template message, and an HTML email. The model never sees customer-level data at this point; the brief and the channel are the only inputs.The orchestrator stores one prompt per channel. The SMS prompt enforces a hard character limit; the WhatsApp prompt allows a longer message; the email prompt asks for structured HTML:
# SMS
Write a single SMS for the campaign brief below. Hard limit: 160 characters.
No emojis, no links unless the brief explicitly includes one. Plain text only.
# WhatsApp
Write a WhatsApp message for the campaign brief below. Up to 1024 characters.
Friendly tone, optional emoji where natural.
# Email
Write an HTML email body for the campaign brief below. Include a single <h1>,
two short paragraphs, and a call-to-action link placeholder {{CTA_URL}}.
No <html> or <body> wrappers.
Each prompt is formatted with the campaign brief and sent to Bedrock; the response becomes that channel’s variant for every approved customer in the segment.
The safety stage runs after the parse stage and before the adapt stage, and it is rule-based rather than model-based. It reads each customer’s stored sentiment score and rolling send count from DynamoDB, and drops customers below the sentiment threshold (default -0.3) or above the fatigue limit. The fatigue limit is a per-customer count of sends over a rolling window, for example five sends in the previous seven days. You set the fatigue window and the sentiment threshold as Step Functions input parameters. You populate the sentiment score upstream. For example, you can run a daily Amazon Comprehend Custom Classification job that scores recent support transcripts and writes the result back to the customer record.
The fatigue check reads the customer’s recent send timestamps from the rate-limits table and counts the entries inside the rolling window:
NEGATIVE_THRESHOLD = Decimal("-0.3") # configurable
MAX_MESSAGES_PER_WINDOW = 5
WINDOW_SECONDS = 7 * 24 * 60 * 60 # 7 days
def _is_fatigued(customer_id):
item = _rate_limits.get_item(Key={"limiter_key": f"customer:{customer_id}"}).get("Item")
if not item:
return False
cutoff = int(time.time()) - WINDOW_SECONDS
recent = [t for t in item.get("recent_sends", []) if int(t) >= cutoff]
return len(recent) >= MAX_MESSAGES_PER_WINDOW
Each successful send writes its timestamp into the customer’s recent_sends list, so the next campaign sees an up-to-date fatigue count without a separate ETL step.
Orchestration
Each stage in the campaign workflow is a small AWS Lambda function. The state machine invokes them in sequence: parse the instruction, segment customers, predict channels, check safety, adapt content, deliver messages, and record results. The predict stage reads each customer’s per-channel engagement history from DynamoDB and picks the channel with the highest historical engagement rate. When you wire an Amazon SageMaker AI endpoint into the stack, the stage calls that endpoint instead and uses its score as the channel ranking signal.The state machine, not the functions, owns the control flow. New stages (for example, a translation step before adapt content) can be inserted without changing the existing handlers. The Step Functions definition lives in statemachine/campaign_orchestrator.asl.json. Refer to it in the GitHub repository for the exact state graph and retry policy.
Semantic search
Consider a marketer who types “Send the Andaman package to budget-conscious families interested in beach vacations.” A keyword filter against the customer table won’t match a profile tagged “economy package, kid-friendly, coastal”, because the words don’t overlap even though the meaning does. To bridge that gap, the seed script embeds each customer profile with Amazon Titan Text Embeddings v2 and writes the vector into an OpenSearch Serverless Vector search collection. The segment stage then embeds the marketer’s phrasing at query time and runs a k-nearest-neighbor search against the collection.
The orchestrator intersects those matches with the structured DynamoDB filter. The final segment respects both the hard constraints (location, age, recency) and the soft ones (lifestyle, affinity). OpenSearch Serverless scales the collection’s compute units to zero when idle, so this capability adds near-zero cost when no campaigns run.
Deployment
To deploy the sample in your AWS account, clone the GitHub repository and run the SAM-based deploy script:
git clone https://github.com/aws-samples/sample-ai-campaign-orchestrator.git
cd sample-ai-campaign-orchestrator
./scripts/deploy.sh --guided
The script prompts you for an AWS Region, a stack name, and the orchestrator parameters (your WhatsApp phone number ID and optional SES sender). It then runs sam build followed by sam deploy, and prints the API endpoint and stack outputs when the deployment finishes.
Test the solution
After the stack finishes deploying, seed the customer profiles table with one sample customer and run a campaign against it:
Then submit a campaign instruction to the API endpoint that the deploy script printed:
curl -X POST $ENDPOINT -H 'content-type: application/json' \
-d '{"instruction": "Send the Andaman package to Mumbai customers"}'
From here you can:
Watch the campaign execution in the AWS Step Functions console.
Query the delivery tracking table in Amazon DynamoDB to see which customers the safety stage approved or suppressed, and which channel the orchestrator picked for each.
Check the recipient’s phone for the WhatsApp template message that the deliver stage sent.
Sample conversation
The recording in this section shows a marketer using the campaign launcher to send an Andaman travel promotion to a Mumbai segment. It opens with the marketer typing the natural-language instruction and the parse stage extracting structured filters. The segment stage then matches customers in DynamoDB. The safety stage suppresses a customer with a low sentiment score. The predict stage assigns a channel per remaining customer. The recording ends with the adapt stage producing one message variant per channel and the deliver stage dispatching them through AWS End User Messaging.
Clean up
To avoid incurring future charges, delete the resources you created. The sample includes a cleanup script in the GitHub repository. Run ./scripts/cleanup.sh to empty the deployment bucket and delete the stack. The stack deletion removes the AWS Step Functions state machine, AWS Lambda functions, Amazon DynamoDB tables, and (when configured) the Amazon OpenSearch Serverless collection.
Conclusion
You can combine AWS End User Messaging, Amazon Bedrock, and AWS Step Functions to build a campaign orchestrator. The orchestrator routes each message to the channel a customer is most likely to open. It also holds back sends to fatigued or unhappy customers.
The same pattern fits other business-initiated messaging workflows where per-recipient channel and content decisions matter. Examples include transactional banking notifications, appointment reminders, and logistics status updates. To deploy the sample in your account, refer to the GitHub repository. To learn more about AWS End User Messaging, refer to the service documentation.
If you’re applying this pattern, start with the safety check and frequency tracking. Those two stages reduce the risk of damaging customer relationships and produce the engagement data that channel prediction depends on. Once that data is in place, add the prediction and content adaptation stages. Use this implementation as a reference for production messaging on AWS.
Healthcare organizations seeking HITRUST i1 certification increasingly rely on Amazon Web Services (AWS) as their cloud foundation. The HITRUST i1 assessment covers 182 curated controls at the Implemented level and is the most widely required HITRUST certification tier in healthcare vendor contracts and Business Associate Agreements required by health plans, hospital systems, and business associates as a condition of working with them.
This guide is designed to close the gap between understanding what HITRUST i1 requires and knowing how to implement it on AWS. It walks cloud architects, security engineers, compliance leads, and assessment preparation teams through the full lifecycle of an i1 engagement from defining the assessment boundary to implementing controls across each technical domain.
What the guide covers
The guide addresses 11 HITRUST i1 technical control domains, with supporting AWS implementation components relative to these domains. The domains include access control, endpoint protection, configuration management, vulnerability management, network protection, transmission protection, incident management, data protection and privacy, audit logging and monitoring, password management, and business continuity and disaster recovery.
The guidance is grounded in a fictional but realistic connected healthcare platform deployed on AWS Landing Zone Accelerator. The scenario is used to make abstract HITRUST concepts concrete, not to suggest that the same architecture or control choices apply universally. HITRUST i1 scoping is inherently organization-specific. The assessment boundary, applicable controls, and evidence requirements are determined by each organization’s system scope and delivered through the HITRUST MyCSF portal. Readers should treat the guidance as a starting point and work with a HITRUST Authorized External Assessor to validate what applies to their specific environment. This guide doesn’t constitute a compliance certification advisory.
AWS HITRUST assurance documentation and the Customer Responsibility Matrix are available through AWS Artifact. For assessment readiness support, visit AWS Security Assurance Services.
If you have feedback about this post, submit comments in the Comments section below.
In addition, AWS is introducing several new security and governance tools, including: new global condition keys for OAuth, token introspection and revocation, dynamic client registration, new AWS CloudTrail elements, and a new API for headless OAuth connectivity. All of this is compatible with your existing IAM configuration including permissions, roles, and federated access.
In this post, you’ll learn how to connect your agents to the AWS MCP Server, understand how AWS Sign-In authorizes agent access, and manage access using new security and governance capabilities.
How to connect an agent to the AWS MCP Server
This walkthrough uses Claude Code, but the same steps apply to any agent that supports Model Context Protocol (MCP) such as Kiro, Codex, and Gemini. See Setting up the AWS MCP Server for how to connect the AWS MCP Server to an agent.
Prerequisite permissions
To connect an agent to the AWS MCP Server, you’ll need the IAM permissions required for OAuth-based sign-in. The following AWS CLI command adds a managed policy with required permissions to your IAM role (remember to replace <MyRole> with your IAM role):
aws iam attach-role-policy \
--role-name <MyRole> \
--policy-arn arn:aws:iam::aws:policy/AWSMCPSignInOAuthAccessPolicy
Step 1: Configure the AWS MCP Server on your agent
Run the following command to add the AWS MCP Server endpoint to your agent’s configuration as shown in Figure 1:
claude mcp add --transport http aws-mcp https://aws-mcp.us-east-1.api.aws/mcp
Figure 1: Adding the AWS MCP Server endpoint to Claude Code
Step 2: Review the authorization request
The first time your agent needs to access the AWS MCP Server, it opens a browser and redirects you to an AWS Sign-In page, shown in Figure 2. Authenticate as you would on AWS console or AWS CLI, review the authorization request, and approve access. You should receive an Authorization successful message.
Figure 2: Review authorization request
Note that if you already have an active AWS Sign-In session (e.g., because you previously signed in to the console earlier in the day), you can reuse that session without needing to sign in again.
Step 3: Start using AWS tools
After connecting your agent to the AWS MCP Server, you can begin invoking tools provided by the server. To verify that Claude Code is connected to the AWS MCP Server, start Claude Code and run the following command:
/mcp
The command displays the configured MCP servers and confirms that the AWS MCP Server is connected and ready to use with your AWS credentials.
Figure 3 shows an example of a successful connection to the AWS MCP Server.
Figure 3: Verifying the AWS MCP Server connection in Claude Code
After the connection is established, you can ask Claude Code to invoke tools provided by the AWS MCP Server. For example, enter the following prompt:
Deploy a sample serverless web application into my development AWS account
Claude Code uses the AWS MCP Server to identify the active AWS account, confirm the target account, and describe the deployment it plans to perform before invoking AWS services on your behalf.
Figure 4 shows Claude Code confirming the active AWS account and outlining the resources that will be deployed.
Figure 4: Using Claude Code to deploy a sample serverless application through the AWS MCP Server
Authorization models and how they work
AWS Sign-In supports two authorization models for connecting agents to the AWS MCP Server:
Interactive authorization for developer’s AI agents using browser based authentication
Non-interactive (headless) authorization for applications and AI agents that already have AWS credentials and don’t have access to a browser
Note that authorizing an agent allows it to access the AWS MCP Server on your behalf. It doesn’t grant the agent additional AWS permissions. Every request is still evaluated using your existing IAM policies, SCPs, RCPs, permission boundaries, and other organizational controls.
Interactive access
In the interactive case, the agent first discovers the AWS Sign-In OAuth server and then registers itself as an OAuth client using Dynamic Client Registration (DCR). It then redirects you to an AWS Sign-In page where you authenticate and authorize access (step 2 in the preceding section). After successful authorization, AWS Sign-In then issues short-lived access tokens and refresh tokens that authorize the agent to access the AWS MCP Server on your behalf. AWS Sign-In automatically manages token issuance and token refresh, enabling authorized agents to continue accessing the AWS MCP Server without requiring you to repeatedly sign in.
The interactive authorization model supports three distinct sign-in methods: native AWS IAM credentials for individual developers, managed access through AWS IAM Identity Center for enterprises, and seamless federated access via third-party providers like Okta and Ping Identity for larger organizations
OAuth server metadata and DCR
Before an agent can request authorization, it must discover the AWS Sign-In OAuth endpoints and register itself as an OAuth client. AWS Sign-In supports OAuth metadata discovery and DCR, allowing supported agents to configure themselves automatically without requiring developers to manually provision OAuth client IDs and client secretsWhen an agent connects to the AWS MCP Server for the first time, it retrieves the AWS MCP Server’s protected resource metadata (RFC 9728) and the AWS Sign-In OAuth metadata (RFC 8414). The agent then uses RFC 7591)) to register with AWS Sign-In, obtain a client ID, and initiate the standard OAuth authorization code flow.
AWS Sign-In supports OAuth discovery and DCR for agents running on local workstations and supported hosted environments. For the current list of supported agents and environments, see Supported redirect URIs for the AWS MCP Server.
Non-interactive access to the AWS MCP Server
Non-interactive (headless) authorization is for agents and applications that run without a browser or human in the loop, and thus don’t require interactive sign-in. This allows agents that already have AWS credentials to obtain OAuth access tokens and connect to the AWS MCP Server. The following is an example of how to obtain an access token.
In the non-interactive case, AWS Sign-In implements the OAuth client credentials grant using AWS security credentials instead of a static client secret. Applications authenticate to the AWS Sign-In token endpoint using SigV4 creds, and AWS Sign-In returns a short-lived OAuth access token that can be used to access the AWS MCP Server.
Please note you may have to update the SDK and AWS CLI, please refer to CLI guide.
Managing OAuth access
AWS Sign-In extends the existing IAM authorization model with capabilities for governing OAuth access to the AWS MCP Server. Administrators can use familiar IAM policies together with new OAuth-specific controls.
Granting OAuth permissions
OAuth access is governed using IAM policies and requires the following IAM actions:
signin:AuthorizeOAuth2Access – Allows users to sign in interactively using the OAuth authorization code flow
signin:CreateOAuth2Token – Allows applications to obtain OAuth access tokens by exchanging authorization codes, refresh tokens, or using client credentials
When an application requests access, AWS Sign-In creates an OAuth authorization grant between the agent and the AWS MCP Server. This grant is represented as an IAM resource, which the preceding AWS Sign-In actions are authorized against.
OAuth authorization grants are represented as IAM resource enabling administrators to use standard IAM policy constructs, including global condition keys, together with OAuth-specific condition keys to control how authorization grants are created and used.
Governing OAuth access
AWS Sign-In introduces OAuth-specific condition keys that allow administrators to govern how agents obtain OAuth authorization. The following examples demonstrate common governance patterns.
To restrict OAuth authorization to localhost:
In addition to accessing the AWS MCP Server with agents on your local workstation, AWS supports signing into the AWS MCP Server on select hosted providers through dynamic client registration. Click here to view the list of supported remote providers. Many organizations want to allow developers to authorize agents running on their local workstations while preventing OAuth tokens from being delivered to untrusted redirect URIs or using unsupported authorization flows. The following policy allows only the OAuth authorization code and refresh token flows for the AWS MCP server and restricts token delivery tolocalhost.
Use the aws:SignInSessionArn global condition key to deny authorization associated with a specific sign-in session. This allows administrators to contain a suspicious or compromised authorization session without affecting other active sessions.
These examples demonstrate common governance patterns. Additional IAM and SCP examples are available in the AWS Sign-In condition keys reference.
Revoking OAuth tokens
AWS Sign-In provides OAuth token introspection and token revocation APIs that allow administrators to build custom tools for token validation and revocation. Access to these APIs is controlled through the signin:IntrospectOAuth2Token and signin:RevokeOAuth2Token permissions. IAM principals with permissions are allowed to introspect and revoke tokens for the same account.
The introspection API can be used to determine whether a token is active and obtain information about the associated authorization. The revocation API allows administrators and security tools to revoke individual refresh tokens without affecting other active sessions. For example, if an organization needs to invalidate access for a specific OAuth authorization, account admins can revoke the associated refresh token without affecting other active sessions.
Monitoring OAuth activity
OAuth-related activities are recorded in AWS CloudTrail, including authorization requests, token issuance, token revocation, and token introspection events. CloudTrail logs also capture details such as the OAuth client, target the AWS MCP Server, redirect URI, authorization flow, and associated sign-in session. In addition, AWS API calls made using OAuth access tokens include the associated aws:SignInSessionArn context, allowing organizations to correlate API activity with the originating OAuth sign-in session.
This allows security teams to monitor OAuth usage, investigate authorization activity, detect anomalous behavior, and integrate OAuth events into existing auditing, compliance, and incident response workflows alongside other AWS activity.
Here’s a CloudTrail sample for an AuthorizeOAuth2Access event:
AWS Sign-In support for OAuth enables you to securely connect to the AWS MCP Server using industry-standard authorization. This release simplifies application and agent integration with AWS while supporting your existing IAM setup, governance, and auditing capabilities.
Over a dozen major economies have now published post-quantum cryptography (PQC) adoption guidance. As a CISO, you’re probably well into your migration plan and know the most difficult part has little to do with changing algorithms. The real leadership challenge is driving coordinated change across a large, complex organization where asymmetric cryptography is embedded in every protocol, every vendor dependency, and every legacy system that quietly handles key exchange or digital signatures. This guide provides the regulatory context and the strategic playbook for CISOs, CTOs, or any senior leader who needs to deliver a program that meets compliance deadlines while modernizing your organization’s security governance.
Overview for busy executives
There are five key takeaways to the information presented in this post:
Start at the top. Secure board-level sponsorship by framing cryptographic modernization as enterprise risk reduction with a defined timeline and measurable milestones. Stand up a centralized program office that owns the mandate, sets prioritization criteria, and coordinates delivery across business units.
Classify dependencies, don’t inventory everything. At the workload level, you need to understand three things: what your providers will upgrade on your behalf, what they won’t upgrade in time and needs replacing, and what you own and must address directly. The fastest path to reduce your migration scope is to shift cryptographic responsibility to the first category (what providers will upgrade for you) wherever possible.
Invest in cryptographic telemetry. Build visibility and monitoring in parallel with your migration work. Although this capability is critical, it shouldn’t come at the cost of momentum. Track algorithm usage, PQC coverage percentage, and migration velocity at the workload level. Telemetry sustains board sponsorship over a multiyear program and gives your centralized team the feedback loop to set priorities.
Build for agility, not one-time compliance. Your goal should extend beyond deploying PQC one time. Build the organizational muscle to rotate protocols, algorithms, and key lengths as standards evolve, because cryptographic migration will be a recurring operational requirement.
Treat this as security and governance modernization. Strong patching discipline, reliable continuous integration and delivery (CI/CD), and automated lifecycle management are capabilities that will outlast your PQC migration. They’re the same capabilities you need to respond to AI-accelerated threats, where vulnerability discovery timelines are compressing from weeks to hours. An organization that can rotate algorithms on demand can also patch against novel AI-driven exploits.
Read on for the full playbook.
Global regulatory landscape
In August 2024, NIST published the first three post-quantum standards covering key encapsulation (ML-KEM), lattice-based digital signatures (ML-DSA), and hash-based signature alternatives (SLH-DSA). These standards now serve as the baseline that most jurisdictions reference when setting migration deadlines. The United States, European Union, United Kingdom, Germany, France, Australia, Canada, Japan, South Korea, India, Singapore, and the UAE have all published formal guidance. Industry groups like FS-ISAC in financial services and GSMA in telecom have their own additional timelines.
These timelines vary by jurisdiction, but all follow the same direction. Most regions require PQC readiness for new procurement by 2027, with full migration deadlines falling between 2030 and 2035 depending on industry and geography. For any organization operating across borders, navigating the specific requirements in each jurisdiction where you do business is critical to both compliance and competitive positioning. Amazon Web Services (AWS) maintains a detailed breakdown of regional mandates and timelines in the FAQ section of the Migration to quantum-resistant cryptography page.
Scoping your migration
Historically, cryptographic migrations have taken far longer than you might expect. The deprecation of SHA-1 took nearly twenty years from the first published vulnerability until major browsers finally rejected it. MD5, 3DES, and RC4 all followed the same pattern of slow organizational response despite clear technical consensus that migration was overdue. Those transitions also happened without the modern cloud infrastructure, automated orchestration, and real-time telemetry that exists today. Organizations that use these capabilities can migrate faster while simultaneously building a future-ready security foundation.
The migration scoping challenge splits cleanly into two families. The first is software systems that negotiate algorithms as part of short-lived authentication or encryption protocols, such as TLS, IPsec, or SSH. For these workloads, cloud-centered lifecycle management, automated patching, and centralized library upgrades make this more straightforward than previous cryptographic migrations. Managed services can handle upgrades transparently and telemetry tooling gives real-time visibility into algorithm usage across endpoints. CI/CD pipelines enable incremental rollout with clean rollback paths. Organizations with modern cloud infrastructure have never been better positioned to execute this side of cryptographic transition at speed.
The second family of things to migrate are long-lived embedded systems, which are devices with burned-in firmware that contain keys and algorithm code that can’t be updated in place. The fastest way to reduce this surface area is to offload their cryptographic workloads to managed services, where your provider absorbs the hardware refresh cycle and every migrated workload is one fewer device you need to plan around. For what remains on dedicated hardware, build quantum readiness into your annual capex review. Because quantum advances don’t arrive on a fixed schedule, evaluate embedded cryptographic assets yearly against developments in quantum hardware. Some devices will stay operationally sound for years, whereas others will need accelerated replacement as threat timelines compress. Annual evaluation means early deprecation becomes a planned business decision rather than an unbudgeted emergency.
The strategic playbook
The following playbook outlines a strategic approach to PQC migration that you can adapt to your organizational context. Each step is designed to build enterprise-wide alignment, replace ambiguity with actionable frameworks, and deliver measurable progress to keep your program funded and on track.
Secure board-level commitment
CISOs need to bring PQC to the board as a business risk conversation anchored to regulatory compliance and competitive exposure rather than a technical briefing on lattice-based algorithms. During this process, it’s important to battle misconceptions. One common misconception at the board level is that PQC migration requires re-encrypting all stored data. It does not. Data encrypted at rest using standard 256-bit symmetric encryption is not vulnerable to a quantum computer. This distinction significantly narrows the actual scope of change and should be communicated early to prevent over-scoping.
Present the regulatory timeline with specificity. For example, explain how CNSA 2.0 mandates PQC for new products by January 2027 and that these timelines will function as procurement gates in regulated industries like financial services, healthcare, government, and defense. You can also quantify the organizational exposure by mapping revenue and workloads that sit in regulated verticals. This could be using existing contracts and pending opportunities with public sector customers as the quantifiable data for business at risk.
Here’s an example of what this could look like in practice. First, identify existing contracts in regulated verticals where PQC compliance language is appearing or will appear at renewal. Calculate the revenue attached and flag renewal dates within 18 months as compliance cliffs. Second, look at your open pipeline. Do you have RFPs, vendor questionnaires, or procurement requirements already referencing post-quantum readiness? That pipeline value is at risk of disqualification if you can’t demonstrate compliance and a competitor can. Third, size the total addressable opportunity in verticals where mandates are taking effect and frame what share becomes inaccessible without readiness. With customers writing PQ readiness requirements into vendor contracts, organizations that can’t demonstrate compliance risk being disqualified from future business.
Finally, request dedicated headcount and vendor budget with board-level sponsorship. This can’t be a side project absorbed into existing security operations. Prioritize executive reviews with quantifiable outcomes tracked quarterly at the leadership level.
Assign single-threaded migration leaders
Stand up a cryptography center of excellence with a cross-functional mandate that spans security, engineering, compliance, and procurement. Appoint a migration lead with direct executive reporting who owns the program end-to-end. Staff the team with representation from networking, identity, application development, vendor management, and compliance because PQC touches all these domains simultaneously.
Give the team authority to set organizational standards for cryptographic policy, library usage, and migration timelines. Align this body with vendor and supplier engagement so there’s one accountable team driving the cloud provider and third-party vendor relationships on PQC readiness.
Fund this team to drive centralized remediation patterns that individual business units adopt rather than reinvent. They own the reference implementations, the approved library versions, the testing frameworks, and the rollout playbooks. When one team solves a migration pattern for a given workload type, the centralized team packages that solution and distributes it across every similar workload in the organization.
Classify dependencies and reduce migration surface area
Beware of guidance that recommends a comprehensive bottom-up cryptographic inventory, except in jurisdictions where it’s explicitly required. That exercise can consume months and delay actual migration. Instead, classify your dependencies into three categories:
Workloads where someone else will upgrade for you. Managed cloud services, software as a service (SaaS) providers, and infrastructure vendors with active PQC roadmaps fall here. Your job is to validate their timelines and hold them accountable.
Workloads where someone else owns the stack but won’t upgrade in time. These are vendor dependencies that you need to replace, potentially before the end of their planned useful life. Flag them now so replacement decisions enter your procurement and capex cycles early.
The third is workloads you own and must upgrade yourself. For these, the decision is whether to upgrade in place or modernize into the cloud where the cryptographic layer becomes managed for you.
The first two categories fall into a vendor risk assessment program. The third category is the workstream that must be managed within your own organization and driven to completion on a workback schedule. Track which dependencies have been validated, which replacements are in flight, and which of your self-managed stacks have active upgrade plans. The three-category model gives your centralized team a clear decision framework instead of going into an unbounded discovery exercise.
Build observability and continuously monitor progress
Visibility into your cryptographic posture is a necessity for planning, execution, and demonstrating compliance to auditors. However, observability shouldn’t be a prerequisite to migrating workloads and should be viewed as a parallel workstream so it doesn’t come at the cost of momentum. After your visibility tooling is in place, it will retroactively show all previous work completed and give a real-time view of progress at the organization level.
Many organizations start with TLS because it’s typically the broadest deployment of cryptography and the primary mechanism protecting sensitive data in transit across web applications, APIs, and microservices. Sponsor TLS metric dashboards that show algorithm usage across all endpoints, differentiating between post-quantum and classical TLS traffic using metadata fields in service logs. The PQC Readiness Scanner serves as an example of how to build and deploy this type of visibility tooling. Over time, extend the same observability to other transport protocols like IPSec, SFTP, and SSH.
Establish a continuous evaluation program with company-wide KPIs, which can feed into executive reviews. Beyond discovery, telemetry provides the executive-level progress metrics that sustain board sponsorship over a multiyear program. Some examples include:
Percentage of TLS connections using TLS 1.3 and ML-KEM key exchange
PQC coverage percentage across your defined categories
Ratio of validated vendor timelines to unconfirmed ones
Time-to-remediation when a new dependency is flagged as noncompliant.
Track PQC coverage percentage at the workload and organization level. These metrics turn PQC migration from a one-time project into an ongoing governance function, the same way you already govern patching cadence, vulnerability SLAs, and compliance posture. The goal is to develop a standing capability that absorbs future cryptographic transitions as routine operational work rather than requiring a new program each time.
Align with vendors, regulators, and industry groups
PQC migration crosses organizational boundaries and requires coordinated movement across your supply chain. Engage your cloud providers on their PQC roadmaps and understand which services already support PQ-TLS, which are on the roadmap, and when support is expected. Engage third-party software vendors and SaaS providers with explicit questions about PQC support timelines and write PQC readiness into procurement requirements and vendor contracts going forward.
Engage regulators and standards bodies in your jurisdictions to understand the specific timelines, compliance mechanisms, and audit expectations that apply to your industry. Participate in industry forums because financial services, telecom, healthcare, and critical infrastructure each have sector-specific PQC working groups where peer organizations are sharing approaches and lessons learned. This collaborative approach can also help you get the investment you need for a migration when you have unwilling stakeholders across the business.
Prioritize and roadmap the workloads you own
Adopt a phased approach rather than attempting to migrate everything all at once. Prioritize workloads based on risk and use case. The AWS post-quantum cryptography migration plan blog post provides an example of this prioritization. As you execute on your roadmap, build reliable release and rollback mechanisms at every stage. PQC algorithms have different performance and size characteristics that might surface unexpected behavior under production load. Identify legacy dependencies before they become migration blockers. Systems running custom TLS libraries or hardcoded cipher suites need to be flagged early in the process.
The fastest path to reducing your PQC surface area is eliminating custom cryptographic stacks entirely. Every workload you migrate to a managed service is one fewer workload that your team must upgrade manually. AWS has already delivered post-quantum key exchange across several service endpoints with imperceptible performance impact, and post-quantum signing through AWS Key Management Service (AWS KMS) and AWS Private Certificate Authority. For bespoke code on cloud compute or on premises, open source cryptographic libraries like AWS-LC provide production-ready, FIPS 140-3 validated PQC implementations that your teams can adopt immediately.
Transition to a crypto agile enterprise
Crypto agility is the operational capability to rotate algorithms, update protocols, and absorb cryptographic change as business as usual rather than a dedicated program. Cryptographic standards will continue to evolve. Algorithms will be deprecated and replaced. The organizations that build the ability to do this now won’t need a new program next time.
Crypto agility demands excellence at four disciplines:
Patching and upgrade discipline: If you can’t maintain consistent patching cadences across your fleet today, PQC migration will surface that gap at enterprise scale. Mature vulnerability management programs adopt PQC as a natural extension of existing operations.
Incremental release with clean rollback: PQ algorithms carry larger signatures, larger keys, and different performance profiles. You need to be able to deploy changes incrementally, validate behavior in production, and rollback cleanly when something doesn’t perform as expected.
Consistent CI/CD pipelines: Every application touching asymmetric cryptography will need to be evaluated and potentially rebuilt and redeployed with updated algorithms or libraries. Fragile or manual deployment processes will impede the entire migration.
Automated security lifecycle management: Certificate lifecycle, key rotation, secrets vaulting, signature operations, and compliance validation must all operate at machine speed. Manual processes that function today will fail as security requirements evolve.
These aren’t necessarily PQC-specific investments. They’re the foundational capabilities of a well-run security organization. With AI accelerating the speed at which vulnerabilities are discovered and exploited, organizations that have built crypto agility into their operational posture are better positioned to respond to AI-accelerated threats. Savvy security leaders can use PQC as a forcing function to build the operational resilience your organization needs as the threat landscape evolves.
Conclusion
PQC migration will define how the next generation of enterprise security programs are built and measured. The technical tooling exists to execute this transition faster than any previous cryptographic migration. The organizations that move now will shape procurement requirements and set the competitive baseline for their industries. Those that defer will inherit compressed timelines, increased costs, and diminished optionality.
If you have feedback about this post, submit comments in the Comments section below.
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
Note: CloudTrail events typically take 5–15 minutes to appear. Wait at least 5 minutes after the deployment failure before running the troubleshoot tool for the most complete analysis.
