A second delivery of the same message can reveal more about an application than its happy-path demo. I like tools that make those cases easier to investigate while a change is still small. Last week’s launch of AWS Emulator brings part of that work onto your own machine: developers and coding agents can exercise AWS API interactions, inspect results, and try again without provisioning cloud resources for every attempt.
I wrote about the Emulator’s launch last week, and it’s my main highlight in this edition. There are also new ways to isolate workloads on shared infrastructure, give an agent narrowly scoped access to a backend, and prepare search applications for a regional disruption. Now, let’s get into this week’s AWS news…
AWS Emulator runs locally alongside your code and development tools. Image: AWS.
Headlines Test your applications against AWS locally with AWS Emulator introduces a generally available local environment for AWS API interactions. Point the AWS CLI or AWS SDKs at its HTTPS endpoint, and your application keeps its normal SDK calls. The walkthrough shows a coding agent creating an Amazon DynamoDB table, writing an item, and reading it back. You don’t need an AWS account for emulated operations, and there is no additional service fee. The getting-started guide covers Linux packages, Docker for macOS and Windows, and agent-assisted setup; the source is on GitHub.
What I like is giving an agent concrete feedback while it works: run calls, inspect results, and revise the code. Matt Wood’s perspective on feedback for coding agents explores why that loop matters. Check the published fidelity results for your operations, because coverage varies and unimplemented calls return HTTP 501. The emulator doesn’t reproduce IAM and resource-policy enforcement or cross-account and cross-Region behavior, so test the complete application and its security configuration on AWS before production.
AWS Nitro Sandboxes bring hardware isolation to pods, tasks, and VMs. You can give an Amazon EKS pod or Amazon ECS task its own VM boundary while sharing an Amazon EC2 instance. That is useful for tenant workloads, build jobs, and agents executing generated code. Isolated mode separates workloads from their neighbors; confidential mode adds protection from node operators and cryptographic attestation. The Compute Blog deep dive explains both modes and their integrations. Nitro Sandboxes have no additional charge on supported instances; EC2 and applicable EKS Auto Mode or ECS Managed Instances fees still apply.
Last week’s launches Here are some launches and updates from this past week that caught my attention:
Amazon Cognito supports OAuth 2.0 token exchange – You can now exchange a user’s token for one scoped to a downstream service in Amazon Cognito. The derived token can have narrower permissions and a single audience, with no refresh token; supplying an actor token also identifies the intermediary in an act claim. For an agent calling a backend on someone’s behalf, that makes delegation more explicit without forwarding the original token. The feature is available on Essentials and Plus plans in all Cognito Regions.
Claude Fable 5.2 is available on AWS – You can access the model through Amazon Bedrock or Claude Platform on AWS. Anthropic positions it as its most capable generally available model for technical work, including extended engineering tasks, scientific research, and quantitative finance. Bedrock has model-specific retention requirements: Fable 5.2 traffic is retained for up to 30 days for abuse detection, with a temporary zero-retention exception for eligible internal use.
Amazon ElastiCache Serverless adds search and aggregations – You can now combine vector, full-text, and hybrid search with server-side aggregations in Amazon ElastiCache Serverless for Valkey. That gives product discovery, recommendations, and agent-memory retrieval another option for querying frequently changing data. The capability requires Valkey 9, so create a compatible cache or upgrade an existing one. It is available in commercial Regions, AWS GovCloud (US), and the AWS European Sovereign Cloud.
Amazon OpenSearch Service adds managed cross-Region disaster recovery – You can now pair Amazon OpenSearch Service domains running version 3.7 or later in two Regions within one account. AWS specifies recovery objectives under one minute for data loss and under 20 minutes for recovery time. Planned switchovers preserve all writes; unplanned failovers can lose unreplicated data. The deep dive covers the remaining work: keeping configuration aligned and redirecting application writes after switching roles, plus secondary-domain and data-transfer charges.
Amazon CloudWatch Logs adds account-level retention policies – You can now set retention across all log groups or groups matching a name prefix in Amazon CloudWatch, with policies covering existing groups or only new ones. When policies overlap, the longest retention period applies. A platform team can set a baseline and retain security logs longer without updating every group individually. The feature has no additional charge and is available in commercial Regions except Middle East (Bahrain) and Middle East (UAE).
The next generation AgentCore runtime is now the default – New Amazon Bedrock AgentCore runtimes now use the next-generation runtime by default, so agent sessions start in under 2 seconds, scale up to 8 vCPU and 32 GB, scale to zero when idle, and bill up to 68% less memory, while each session still runs in its own isolated microVM.
For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.
Other AWS news Here are some additional posts and resources that you might find interesting:
Optimizing cost per task and proving agent ROI on Amazon Bedrock – This guide follows an agent’s cost through repeated calls, reasoning, caching, and human review. I like its emphasis on measuring a completed task and comparing outcomes against a documented baseline. Its worked prices are illustrative; use your own workload and reconcile estimated costs with billing data.
How Zillow runs 100 million daily notifications on AWS Step Functions – Zillow describes reusable content modules and configuration-driven AWS Step Functions Express Workflows for more than 100 million daily notifications. The useful engineering details include idempotent modules, batching, campaign isolation, and testing expressions with the TestState API before deployment. Those patterns apply well beyond notification systems.
Upcoming AWS events Check your calendar and sign up for upcoming AWS events:
AWS re:Invent – Join the community in Las Vegas from November 30 to December 4. Browse more than 2,200 sessions, including workshops and deep dives, or register for the free livestream to follow keynotes and selected sessions remotely.
AWS Community Days – Find community-led technical talks, workshops, and hands-on learning near you. Check each organizer’s page for current dates and registration details.
Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS-led in-person and virtual events and developer-focused events.
That’s all for this week. Check back next Monday for another Weekly Roundup!
– Daniel Abib
This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!
Every day, Zillow sends home shoppers more than 100 million personalized email and push notifications: saved search alerts, home recommendations, property updates, seller reports, and reminders. These messages are one of Zillow’s core ways to engage home shoppers, and Zillow delivers them on time with consistent behavior across every campaign. Today, each of those notifications runs as its own short-lived workflow execution on AWS Step Functions Express Workflows.
The underlying shape of this work isn’t unique to real estate. Many systems run the same kind of workflow at high volume: take an event, enrich it from several services, apply rules, assemble an output, and send it. This post shows how Zillow built one configuration-driven orchestration system to run that pattern at scale, and what teams across many domains can reuse from it.
Historically at Zillow a separate generator service produced each of these campaigns, and each service carried its own workflow, data models, and business rules. As the number of notification campaigns grew, so did duplicated engineering work, inconsistent behavior, and operational cost. Product teams needed to launch campaigns faster, reuse shared capabilities, and run consistent experiments without system-specific work.
To meet that need, the Zillow Comms Experience team, working with AWS, built two things. The first is the Campaign Orchestrator, which is a configuration-driven orchestration layer that runs each notification as a short-lived workflow execution on Step Functions Express Workflows. The second is the Universal Content Generator (UCG), a library of reusable, schema-driven content modules that those workflows call. Together they make campaign creation faster and more consistent, and they give AI-assisted tooling a declarative format to work with.
The challenge: why the original system needed replacing
The previous content generation model slowed product delivery. Zillow’s Comms Experience supported more than 20 email and push campaigns generated by more than 10 microservices. Each microservice owned its own workflow, models, business rules, and assembly logic.
Most campaigns followed the same pattern: receive a trigger, hydrate data, apply rules, assemble content, and send the message. Each campaign service implemented that pattern separately, creating duplicate engineering work and inconsistent behavior. It also added operational overhead across releases, monitoring, logs, dashboards, and on-call support.
Each campaign was built as its own service because, when these systems were first built, scale was the main challenge. As Zillow added more products, the tradeoff changed. Similar features ended up implemented differently across campaigns, so every new capability had to be rebuilt and retested in each service. That made it hard to roll out features across products and slowed development. The push to centralize came from that cost: we needed to move faster and stay consistent across every customer experience. With one shared implementation, a feature could be built once and reused everywhere, while each campaign kept the flexibility to tailor its experience. Before evaluating workflow engines, we first had to extract shared logic into reusable modules and move campaign decisions into configuration.
The following diagram shows the original architecture, where each campaign generator carries its own copy of the workflow, data models, and business rules.
Figure 1: Original system
Competitive evaluation
We evaluated Temporal, Flowable, an in-house engine, and Step Functions Standard and Express Workflows against four criteria. Those criteria were reliability, total cost of ownership, development speed, and how well the service would scale across teams. The engine had to be managed, configuration-driven, and able to orchestrate both newer and older services. Pricing also had to fit our volume.
Temporal and Flowable would have added another infrastructure component to operate. That worked against two of our core goals: cutting the release, monitoring, and on-call overhead that the original system created, and getting pilot projects running quickly. Neither is available as a managed AWS service, so we would have had to run and scale them ourselves. Step Functions is serverless, so we had no servers or worker fleets to run, and a team could start a pilot without first setting up infrastructure. An in-house engine offered control but required custom lifecycle management, worker pools, queue integration, and monitoring. Step Functions met the core orchestration needs at roughly a quarter of the estimated build effort and a quarter of the ongoing maintenance effort. It was the clear choice for the minimum viable product (MVP).
Express mode compared with Standard mode
Our planning model assumed high-throughput, low-latency workflows. The MVP plan ramped from 14 million workflows per day to more than 100 million across all campaigns. At full scale that means several thousand executions per second at peak. Each workflow runs for only a few seconds, and a notification has to arrive while it’s still relevant. We evaluated both Step Functions execution modes, and for our use case Express Workflows were the better fit.
At least once, sufficient with idempotent modules.
Our workflows are high volume, short-lived, finish well within the five-minute Express limit, and are designed around idempotent module calls. We validated quota, latency, and operational behavior during the traffic ramp-up.
The architecture
Campaign Orchestrator is a configuration-driven workflow platform. Each message is input data combined with persisted campaign configuration. The configuration is relatively static for each workflow, while the input data stays flexible based on how the configured steps are assembled.
A shared orchestration layer reads queued campaign events, groups work where appropriate, and invokes the corresponding Step Functions Express Workflow for each campaign. Campaign metadata describes product constants, ownership, and Zillow environment features. Workflow configuration describes step topology, step-level configuration, mappings, branching, delay behavior, and send behavior.
The following JSON shows a simplified workflow input, combining the message payload with the platform, campaign, and configuration metadata that drive the workflow:
The following diagram shows the new architecture, where a single shared orchestration layer and one library of reusable modules serve every campaign. A message moves through the diagram from left to right. The ingress service queues it, and the router applies campaign controls and starts the workflow. Campaign Orchestrator runs the Step Functions steps, each step calls a UCG module, and the delivery service sends the result.
Figure 2: The new system
Router
Our campaigns need a set of controls applied before a workflow starts: queue polling, batching, idempotency checks, pause and resume behavior, replay coordination, speed control, and campaign routing decisions. Some of these controls are specific to how we order and throttle campaign sends, so they live outside Step Functions.
We centralized these controls in an in-house router service that reads from Amazon Simple Queue Service (Amazon SQS) and starts the right Step Functions workflow. Keeping them in one place gives us a single point to pause a campaign, replay a batch, or adjust send speed without changing any workflow definition.
Shared modules (UCG)
The Universal Content Generator is the modular content generation layer behind Campaign Orchestrator. Instead of each campaign owning a separate generator, UCG organizes shared capabilities into reusable modules. Before UCG, this logic was spread across more than 10 separate generators. Centralizing it in one service also smoothed its traffic. Some daily campaigns used to run for about four hours and leave their generator idle for the other 20. Now all campaigns share one fleet, so the load spreads across the day.
Modules handle capabilities such as property hydration (loading listing details such as price, photos, and address for each home in a message), hidden homes (listings a user has chosen to hide, which we filter out of notifications), common data (the subscriptions, saved homes, eligibility flags, and propensity scores every campaign loads for a recipient), recommendations, A/B testing, and content assembly. They are stateless and schema-driven.
Direct service invocation with HTTP Invoke
A key design choice was to call UCG directly through Step Functions HTTP Invoke. This fits the workload for two reasons. First, our content generation logic already ran as long-lived containerized services inside our private service environment, with network access, service discovery, and authentication established. Second, routing workflow steps to UCG added no new compute cost.
With Step Functions HTTP Invoke, we call UCG modules for data hydration, rule checks, and content assembly as they already run. Authentication runs through an Amazon EventBridge connection. Each HTTP Invoke task points to the connection with a ConnectionArn. The connection holds an API key, and UCG checks that key on every request. Amazon EventBridge keeps the key in AWS Secrets Manager and adds it to the request at call time. The workflow definition never contains the key, and neither does our application code. When the key rotates, we rotate it once in the connection and every workflow picks it up.
To avoid repeating the same HTTP Invoke settings in every workflow, we introduced component inheritance. Components are written in a slightly modified version of the Step Functions task definition we deploy through AWS CloudFormation. A base component holds the settings that many steps share, and other components extend it. The following base component, content_module_call, defines how every workflow step calls UCG services directly through HTTP Invoke: the resource, method, headers, and the ConnectionArn for the connection described earlier.
Business modules extend the shared HTTP component and supply only the module endpoint. Before deployment, our build script replaces each component with a standard Task state: it loads the component named in Extends, merges the step’s own fields on top, and writes the result into the Terraform we deploy. Workflow definitions stay focused on product intent rather than network plumbing:
Step Functions state machine definitions, JSONata expressions, and infrastructure definitions are powerful, but they are not the right interface for every campaign owner. We added a higher-level configuration layer so campaign owners can define their own workflows without having to understand Step Functions CloudFormation configuration. Each campaign config describes a workflow: which modules run, in what order, and how data moves between them. The system translates that config into the Step Functions definition and CloudFormation template that gets deployed. Campaigns don’t share one generic workflow that picks modules at runtime. Each campaign gets its own workflow, built from the same shared modules and components. As a next step, we are considering a generic workflow that assembles modules in real time from config. Each campaign would still run its own copy of that workflow, so ownership stays isolated per campaign.
Campaign owners specify the modules needed for an email template or push notification. Each module exposes input, output, and configuration schemas. Campaign owners define module outputs to satisfy the template input schema. Module inputs define what triggers need to provide.
With this structure, campaign owners onboard through a declarative process. They work with configuration instead of raw workflow code or JSONata expressions. They can also run controlled experiments and give AI tools a format they can inspect, generate, and modify with appropriate validation.
The following simplified A/B test config is adapted from a real Campaign Orchestrator config. It injects an A/B test module before email transformation and maps the experiment result into the next step without directly touching Step Functions or CloudFormation definition.
Because a campaign is declared as schema-validated configuration, the same contracts that protect human changes also apply to AI-assisted tooling. Our internal assistants already draft module selections and mappings from a product brief, and the output must pass the same schema checks, JSONata tests, and Step Functions validation before it can be deployed.
Batch mode optimization
Campaign Orchestrator supports batching, which improves overall throughput while reducing cost. Instead of launching one workflow per message, the router can group messages and invoke one workflow for the batch.
For example, processing 100 messages individually could produce 100 workflow executions and 400 HTTP requests to UCG services. With batching, the same messages can be processed in one workflow with four HTTP requests, assuming each step supports bulk input and output. That matters at our volume for two reasons. Express Workflows charge for each request and for run time. Batching cuts the number of requests, but each batch runs longer. A rule of thumb: a batch of 10 messages costs less than 10 separate runs as long as it takes no more than 10 times as long as one run. In the preceding four-step example, one message takes about 1.2 seconds, so a batch of 10 saves money if it finishes within 12 seconds. In practice, a batch runs only 10 to 20 percent longer than a single message, about 1.4 seconds instead of 12. That makes a batch of 10 about 90 percent cheaper than 10 separate runs. The batched payload stays under the 256 KiB limit, so memory is still billed at the 64 MB minimum. Fewer executions and HTTP requests also use less of the StartExecution rate, HTTP Invoke concurrency, and logging throughput we size for. That leaves headroom to send more notifications on the same quotas.
The batching model appears in the Step Functions definition and JSONata expressions. This simplified example shapes a batched payload into a bulk UCG request while preserving each tracking ID:
Batching works only if every step in the workflow can handle a batch rather than a single message. For UCG, that requirement is built into the module API. Each module’s schemas define its input, output, and config models, and the module API accepts those models as a batch keyed by tracking ID. A module that doesn’t follow that API won’t build. Our JSONata mappings and upstream and downstream service contracts already accepted bulk input internally, so adding batching didn’t require reworking them.
Batching also can’t cost us visibility into individual messages. Because each batch is keyed by tracking ID, as the preceding example shows, UCG returns a result from each module for each tracking ID. The router uses those per-message results to deliver, retry, or drop each message individually, so one failed message doesn’t fail the whole batch.
The following diagram shows how the router groups messages into a single batched workflow execution.
Figure 3: Batching mode
Scaling: Performance and organizational
At our target throughput, service quotas are part of the architecture. We reviewed Step Functions StartExecution rate limits, HTTP Invoke concurrency, and Amazon CloudWatch logging throughput before traffic ramped. HTTP Invoke concurrency needed attention first, because its default quota is lower than the rate at which Express Workflows can start. We sized the required concurrency from our capacity model and worked with AWS to raise the quota before each rollout phase, so throughput limits never gated the traffic ramp.
We planned the rollout in phases, from early traffic to more than 100 million daily executions. With this approach, we could validate cost, quotas, latency, and operations before full scale.
The architecture also supports organizational scaling. Product teams own campaign modules, steps, configuration, and the Step Functions cost tied to their traffic, so they can launch and change campaigns without waiting on the central team. Central teams own shared infrastructure such as the router service, Step Functions patterns, Amazon SQS integration, and deployment tooling, so one improvement reaches every campaign.
Shared code doesn’t mean shared failure. Every campaign’s Express state machine is generated from the same templates and modules, so isolation adds no duplicate code. Each campaign still runs its own state machine and its own Amazon SQS queue, and the router gives each queue its own polling threads, worker pool, and circuit breakers. When one campaign fails or backs up, only its own breaker trips, and we can pause its queue without touching the others. A traffic spike or downstream failure in one campaign doesn’t become a noisy neighbor for the rest.
Testing
Configuration-driven workflows require strong validation because much of the behavior lives in JSONata expressions and campaign configuration. We keep shared JSONata expressions in dedicated files and require unit tests for them. These tests are not offline string evaluations. The test runner builds a small Step Functions state and executes it through the Step Functions TestState API, validating expressions against the same runtime used by Step Functions.
For example, this shared user profile expression converts a batch of campaign messages into UCG module requests:
Every test case for the expressions we reference through @jsonata runs through the TestState API in continuous integration (CI), and a failing case blocks the deploy. With this approach, teams get reusable mappings and can reduce the risk of malformed payloads, runtime expression errors, or configuration changes that fail inside Step Functions.
Conclusion: Measured impact
The largest impact of the new architecture is product velocity. By replacing repeated custom engineering with reusable modules and configuration-driven workflows, Zillow shortened development cycles for new products and features, opened up greater self-service, and reduced cross-team dependencies. Teams now iterate through configuration, shared modules, and validated JSONata mappings instead of building or modifying a dedicated generator for each idea.
In the original system, testing a new variant or launching a campaign meant changing a dedicated generator service, and often several. Now a team edits a config, reuses modules other teams have already built, and validates the workflow before it reaches production.
The orchestration service also reduced workflow engine build and maintenance cost. Zillow’s internal engineering estimates put the build at 9 development weeks, down from 32 for a comparable in-house engine. Ongoing maintenance fell along the same lines, from roughly 8 hours per week to 2. We sized the system to scale from early traffic phases to more than 100 million workflows per day.
Most of what made this work can be reused by teams with high-volume, short-lived workflows: describe each message as input data plus configuration, keep modules stateless and tested, make every step accept batches, and plan service quotas before traffic ramps. Step Functions Express made those practices practical to run at Zillow’s scale.
The agentic launches that stayed with me last week had nothing to do with how capable a model is. They were about reach. AWS Step Functions, which already orchestrates over 220 AWS services, can now be handed to an agent as a tool, so workflows teams already trust became callable without a rewrite. In the same week, the AI blog published how one engagement platform exposed roughly 400 internal API operations the same way. Neither story involves rebuilding anything, only describing what exists and letting the agent find it.
The other thread was cost. OpenAI’s newest Terra model reached Amazon Bedrock priced well below the model beside it, and two post-quantum launches landed for a deadline nobody can put on a calendar yet. Now, let’s get into this week’s AWS news…
Headlines OpenAI GPT-6.1 Terra is now generally available on Amazon Bedrock, and what interests me is where it sits rather than what it tops. Terra is the everyday-workload line of the GPT-6 family. In OpenAI’s Agents’ Last Exam evaluation it scored 15 percentage points higher than GPT-5.6 Terra, performed on par with GPT-6 Astra, and completed tasks more than three times faster than both. Those are OpenAI’s figures, not an independent test. The price is what I would check first: input and output tokens are at least 50% lower than GPT-6.1 Sol and GPT-5.6 Terra. It takes up to 1 million input tokens with text and image input, and you can reach it from the Bedrock APIs or ChatGPT Work and Codex.
AWS Step Functions workflows are now available as AI agent tools through Amazon Bedrock AgentCore Gateway, which connects agents to tools over the Model Context Protocol (MCP). The division of labor is the idea: the agent decides when to run a workflow and what input to pass; Step Functions runs each step exactly as you defined it. A support agent can trigger a refund, and the eligibility check, the manager approval, and the credit still happen in that order every time, because the ordering was never the model’s to improvise. Your existing Standard and Express workflows qualify as connector targets, and agents can also look up runs. It’s available in all Regions where AgentCore Gateway is, at standard rates.
Last week’s launches Here are some launches and updates from this past week that caught my attention:
AWS Key Management Service now supports ML-KEM for post-quantum key encapsulation – You can now establish a shared secret with a post-quantum algorithm using AWS KMS keys, with private keys in FIPS 140-3 Level 3 validated HSMs. All three NIST FIPS 203 parameter sets are supported through Encapsulate, which produces a random 32-byte shared secret and its ciphertext, and Decapsulate, which recovers it. The threat is harvest now, decrypt later: an adversary records today’s key exchange and recovers the secret once quantum computers are powerful enough. It’s available everywhere KMS runs except the Middle East (UAE and Bahrain) and China Regions. Amazon Aurora PostgreSQL added post-quantum TLS key exchange the same week, on versions 18 and higher.
AWS IAM Identity Center now supports cross-Region trusted identity propagation – You can now carry a user’s identity with their request to AWS services in another Region. This removes a genuine architectural constraint: trusted identity propagation used to require every service accessing that identity to sit in the same Region, which left two unappealing options, broad IAM roles that cannot distinguish individual users, or consolidating all your services into a single Region. Now the identity travels with the request automatically, so you grant access by user and group membership and audit who accessed what. It’s available at no additional cost everywhere IAM Identity Center runs.
Amazon ECS now supports GPU sharing with NVIDIA Multi-Process Service (MPS) – You can now run multiple Amazon ECS tasks concurrently on a single NVIDIA GPU, the practical answer to work that only ever uses a fraction of one, like small-model inference and multi-model serving. You declare the GPU memory each container needs and can optionally cap its share of GPU compute, and ECS places tasks on GPUs with available capacity as your services scale, with no code changes. It works on ECS Managed Instances and ECS on Amazon EC2, and on Managed Instances it watches the MPS control daemon and replaces the instance if it turns unhealthy. It’s available in all commercial and AWS GovCloud (US) Regions.
AWS Lambda supports Seekable OCI (SOCI) lazy loading for Lambda Managed Instances – You can now have AWS Lambda fetch only the files needed at each stage of startup instead of pulling a whole container image first, which AWS puts at up to 70% off container startup time. You create a SOCI index, push it to Amazon ECR, point the image URI at the indexed tag, and Lambda detects it automatically. The technique also reachedAmazon SageMaker AI training and processing jobs, with benchmarks: lazy loading held at 3 to 11 seconds regardless of image size, against a median 3 minutes 42 seconds for a 13.2 GB image, a range they report as 8x to 46x.
Amazon EKS Auto Mode now supports Kubernetes Gateway API – You can now declare how traffic reaches your applications with Gateway and HTTPRoute resources on Amazon EKS without installing a separate controller, and EKS Auto Mode provisions and manages the underlying AWS load balancing infrastructure. You get HTTP and gRPC routing, cross-namespace routing, weighted traffic splitting, header-based routing, and automatic TLS certificate discovery through AWS Certificate Manager. Existing Ingress workloads keep working alongside them, so this is additive rather than a migration. Gateway API support is included with EKS Auto Mode; standard load balancing charges apply.
For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.
Other AWS news Here are some additional posts and resources that you might find interesting:
How Nubank automated disaster recovery with the AWS DevOps Agent – Nubank serves more than 140 million customers across Brazil, Mexico, and Colombia, and classifies over 300 business flows as critical, each tested on a regulator’s schedule. Nubank reports that handing evidence collection and report writing to the AWS DevOps Agent cut a test cycle from about 4.3 hours to roughly 40 minutes, and quadrupled flow coverage in 2026 to more than 160. One execution terminated more than 900 EKS instances under synthetic traffic with AWS Fault Injection Service, producing the evidence its regulator needed.
How MoEngage turned over 400 APIs into agent tools with AgentCore Gateway – Roughly 400 internal REST API operations, built by 25 teams, sitting in private VPCs across seven Regions, with tenant identity that could never become a parameter the model gets to set. MoEngage estimates the managed path saved about six months of platform engineering. The detail worth stealing: a full tool listing would have crossed 20,000 tokens per connection, so agents search for tools by describing a task instead of receiving the catalog.
Reducing petabyte-scale S3 storage costs with evidence-based lifecycle rules – The hard part isn’t deciding to act, it’s knowing which data has gone cold and where it can safely live. This uses S3 Storage Lens groups and Storage Class Analysis to prove access patterns before moving anything. In the engagement described, 5.8 PB of logs in Amazon S3 Standard at approximately $241,000 per month showed an access ratio below 0.1% past 90 days, and a 90-day transition to Glacier Instant Retrieval cut the bill by roughly 70%, a figure the author ties to that workload.
Upcoming AWS events Check your calendar and sign up for upcoming AWS events:
AWS re:Invent – re:Invent returns to Las Vegas from November 30 to December 4, with more than 2,200 sessions. The event catalog is live and reserved seating is open, so register and claim your spots before they fill up.
AWS Community Days – Community-led conferences planned and delivered by community leaders. The 2026 season is winding down as re:Invent approaches, so check the hub for what’s still on.
Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development. Browse here for upcoming AWS-led in-person and virtual events and developer-focused events.
That’s all for this week. Check back next Monday for another Weekly Roundup!
– Daniel Abib
This post is part of our Weekly Roundup series. Check back each week for a quick roundup of interesting news and announcements from AWS!
With AI agents, you can automate medical coding while keeping the workflow deterministic, auditable, and subject to human approval. The key is division of labor: the workflow engine owns sequencing and approval gates, while agents reason within individual, bounded steps. Handing the entire workflow to an agent would make the runtime path a property of the model’s reasoning, which is difficult to review and replay.
Medical coding makes this concrete. Teams must translate free-text clinical terms like “migrane” (a typo) or “flutters in my chest sometimes” into standardized codes before they can analyze or report the data. The target vocabularies include controlled medical dictionaries such as MedDRA for adverse events and WHODrug for medications. This work is safety-critical because a coding error can send an adverse event to the wrong safety signal. Yet it’s mostly repetitive work, with skilled coders reviewing terms one by one. The opportunity is to code the confident cases automatically and route only the ambiguous ones to a human.
In this post, we show you how to combine deterministic orchestration with bounded agentic reasoning using AWS Step Functions and the Amazon Bedrock AgentCore harness integration, a managed capability of Amazon Bedrock AgentCore. We walk through a medical coding workflow that escalates cheapest-first: a safety block-list, then exact matching, then an AI agent only for the terms that need judgment. We also explain the design decisions that keep the model bounded. The complete sample, including infrastructure code and a seeded test dataset, is available in the aws-samples GitHub repository.
Note that MedDRA is a registered trademark of the International Federation of Pharmaceutical Manufacturers and Associations (IFPMA), managed by the MedDRA Maintenance and Support Services Organization (MSSO). WHODrug is maintained by the Uppsala Monitoring Centre (UMC). Both are proprietary, licensed dictionaries. Deploying against them requires a valid subscription. The sample dataset contains generated study identifiers and verbatims with a de minimis illustrative set of dictionary entries, and no patient data. The AWS services used in this sample, including Amazon Aurora and Amazon Bedrock, are HIPAA-eligible when used under a signed AWS Business Associate Addendum (BAA). Before processing protected health information, see the HIPAA Eligible Services Reference and the AWS HIPAA Compliance overview.
Solution overview
The design follows one principle: spend the cheapest, most deterministic mechanism that can code a term with confidence and escalate only when it can’t. A large language model (LLM) agent is powerful but expensive and non-deterministic, so it isn’t the default. Even then, the workflow, not the model, owns every routing decision. The agent might choose which medical code and how confident it is. It doesn’t decide what happens next. A compliance reviewer can read the state machine definition and understand the execution path independent of any agent’s runtime reasoning.
The solution uses AWS Step Functions to orchestrate the coding decision, AWS Lambda for the deterministic steps, and the Amazon Bedrock AgentCore harness for the agentic step. An Amazon Bedrock AgentCore Gateway, a capability of Amazon Bedrock AgentCore, exposes the agent’s tools. Amazon Aurora PostgreSQL-Compatible Edition (Serverless v2) with the pgvector extension serves as a single store for the reference data and its embeddings, so there is no separate vector store to synchronize.
Figure 1 shows the architecture.
Figure 1: Medical coding workflow solution architecture
Each Step Functions execution codes one verbatim: a deterministic direct lookup, an agentic semantic search that runs only on a miss, score-threshold routing on the agent’s confidence, and a write-back onto the same study record. The state machine is deliberately a standard workflow. Executions are exactly-once, and the queryable history is a per-term audit trail. For a regulated coding process, answering “why was this term coded this way, by what, and when” is a requirement. Error paths in this sample end with the term left open, so every record gets a terminal status. In production, send errors that survive the retries to a Fail state instead. The execution then shows as failed in Amazon CloudWatch, and you can redrive it after fixing the cause.
The write-back step stores every symptom entry’s full medical family tree, not just an isolated code. For example, a symptom like ‘muscle spasms’ links all the way up to its overarching category, ‘musculoskeletal and connective tissue disorders’. Safety teams can then roll up signals at any level, not just the lowest one. Additionally, the agent logs a one-sentence explanation for its decision. This makes sure that if similar symptoms (like cramps) end up categorized differently across studies, teams can trace the exact study context that caused the variance. They don’t need to guess based on a generic confidence score.
Figure 2 shows the coding decision flow in Unified Modeling Language (UML) activity notation.
Figure 2: Medical coding decision flow with the three terminal outcomes in UML activity notation
Note: This sample is for demonstration purposes only and is not intended for clinical use. Before using this pattern in production, validate it against a known-correct dataset and measure coding precision and the incorrect autocode rate.
Walkthrough
The walkthrough follows one verbatim through the state machine, focusing on the architectural decision each stage encodes.
Prerequisites
Before you begin, make sure that you have the following:
An AWS account with permissions to create Step Functions state machines, Lambda functions, an Aurora PostgreSQL cluster, Amazon Bedrock AgentCore resources, and AWS Identity and Access Management (IAM) roles.
Access to Amazon Bedrock AgentCore in a supported AWS Region. The harness uses Claude Sonnet 5 (us.anthropic.claude-sonnet-5), and the dictionary search uses Amazon Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0). Both models are enabled by default, but Anthropic models require a one-time usage form before first use, which you can submit from the Amazon Bedrock playground. For model availability by Region, see Supported models by AWS Region in Amazon Bedrock.
Node.js 24+, the AWS Cloud Development Kit (AWS CDK) CLI, and a bootstrapped target account.
Estimated cost
The largest cost is the Aurora Serverless v2 cluster. It keeps a minimum capacity of 0.5 Aurora Capacity Units (ACUs), so it’s billed for every hour it runs, even when idle. For current rates, see Amazon Aurora pricing. To stop the charges, delete the stack as described in Clean up.
Step 1. Resolve the deterministic cases first
The first task state invokes a Lambda function that makes two checks, in a deliberate order. It first checks a terms-not-to-autocode block-list. These are real dictionary terms, like “HAEMORRHAGE,” that are too broad to auto-assign because the code changes depending on where the bleeding occurred. A hit ends the workflow with the term left open for a human. Because this runs before any model call, it’s a hard control: a safety rule placed ahead of the model cannot be prompted away, and costs nothing to enforce.
The same function then runs a case-insensitive exact match against the dictionary and a curated synonym list (“Hypothyroid” maps to “Hypothyroidism”). A match is accepted only if it resolves to exactly one distinct code. On the sample workload, five of twelve cases finish here: one safety block and four auto codes at score 1.00, each for the cost of an indexed query. More than a third of the workload never reaches the agent at all.
Step 2. Declare the coding agent in the workflow definition
We use an agent for this step because deciding when to pull in study context is judgment, not a fixed rule. In our test run, the agent called the study-info tool only for ambiguous terms and skipped it for a clear typo and for gibberish input. That kind of judgment is hard to encode as a threshold, and the agent can take on new tools later without a workflow redesign. This is also why the agent sits last in line, after two cheaper, deterministic checks.
Only on a miss does Step Functions invoke the agent, using the AgentCore harness optimized integration. The harness resource declares only the model. The system prompt, tools, and allowed tool names are supplied per invocation from the task state. The agent’s full contract is therefore readable in the workflow definition, and there is no container to build or operate. The following abbreviated excerpt shows the shape. See the sample for the full definition. The InvokeHarness Task state has a hard 15-minute execution cap regardless of TimeoutSeconds, so a term needing many tool round-trips could hit it. The sample’s two-tool, single-turn design stays well under that limit.
The agent has exactly two tools, exposed as Model Context Protocol (MCP) tools through an AgentCore Gateway. search_dictionary embeds the verbatim with Amazon Titan Text Embeddings V2 and runs a pgvector similarity search. It searches the verbatim exactly as given, because the misspelling is the signal. get_study_info returns the study’s free-text protocol description. It exists because similarity search can return several clinically plausible candidates it can’t separate on text alone. A human coder breaks that tie using knowledge of the study. This tool gives the agent the same context and nothing else: it returns prose only, not codes.
The clearest illustration is the same verbatim coded in two different studies. “cramps” returns an identical candidate list in both. Only the study description differs. In an oncology study that monitors muscle spasms as adverse events of special interest, the agent coded it to “muscle spasms.” In a reflux study where abdominal cramping is expected and musculoskeletal events aren’t an endpoint, it coded the same verbatim to “abdominal pain.” In this run, the oncology result scored 0.92 and was autocoded, and the gastrointestinal (GI) result scored 0.85 and went to a coder for review. In both cases the agent’s job was to bring the right candidate to the top with a reason a coder can check. That’s how a coder who knows the protocol would decide.
Step 3. Route on coding confidence, not retrieval similarity
The most important design decision is what the threshold routes on. Routing on retrieval similarity, the distance between the two embeddings, is measurably wrong. Table 1 illustrates this with an example.
Verbatim
Selected term
Retrieval similarity
Agent confidence
migrane
Migraine
0.37
0.97
flutters in my chest sometimes
Cardiac flutter
0.65
0.75
Table 1. Retrieval similarity and agent confidence for sample terms (one run, Claude Sonnet 5)
Table 1 shows the problem. The typo “migrane” looks very different from “Migraine” as text, so its similarity score is low, but the right code is obvious. “Flutters in my chest sometimes” shares more words with “Cardiac flutter,” so its similarity score is higher, but the right code is less certain. Similarity ranks these two cases the wrong way round, and no threshold can fix that. So instead, the agent reports how confident it is in its own choice, using three bands set in the system prompt. It reports 0.95 to 1.00 when the match is clear, 0.70 to 0.94 when the verbatim is vague or the agent needed study context to choose between candidates, and below 0.70 when nothing fits. For example, “cramps” in the GI study could mean muscle cramps or stomach cramps. The agent checked the study description to choose, and reported 0.85, in the middle band. The bands describe the model’s confidence. The 0.90 autocode gate is a separate, fixed rule in the workflow, applied by the Choice state that comes next.
Note that the routing number is authored by the model. The block-list runs before any model call, and the thresholds are auditable configuration in the state machine definition. The write-back checks the code exists, and anything below the autocode gate reaches a human. Note also that the agent is instructed never to abstain. For gibberish it still returns the top candidate at low confidence, and the gate discards it. The agent scores the candidate, and the workflow decides what happens to it.
Step 4. Write back with a fabrication guard
The terminal task persists the outcome onto the same study record: status, code, hierarchy, score, and a status_changed_by marker that distinguishes machine coding from human action. Before writing, it checks that the chosen code exists in the target dictionary and version. The prompt tells the agent not to invent codes, and this check enforces it. It matters because the agent reads free-text study metadata, which is the kind of input that invites copying a plausible number out of a protocol description. A code that fails the check leaves the term safely open, with the reason recorded. This check has one limit. It can’t tell whether the code was one of the search results, because the workflow only gets the agent’s final response, not what the tools returned. To verify that, look at the agent’s tool calls in AgentCore Observability, a capability of Amazon Bedrock AgentCore.
Deploy and test the solution
The sample deploys with three commands and seeds twelve verbatims that exercise every branch:
npm install && npm run build
npx cdk deploy
npm run seed
Each verbatim runs as one Step Functions execution (the repository README shows the exact CLI calls). The following table shows a live run:
Verbatim
Route
Score
Outcome
HAEMORRHAGE
block-list
n/a
open (safety block)
Dislocated shoulder
exact match
1.00
autocoded
Hypothyroid
synonym match
1.00
autocoded
ibuprofen
exact match (WHODrug)
1.00
autocoded
FULVESTRANT
exact match, derivation-only
1.00
autocoded
migrane
agent
0.97
autocoded (Migraine)
nexiuum
agent
0.95
autocoded (NEXIUM)
cramps (oncology study)
agent + study context
0.92
autocoded (Muscle spasms)
cramps (GI study)
agent + study context
0.85
review (Abdominal pain)
flutters in my chest sometimes
agent
0.75
review (Cardiac flutter)
heart races when I stand up
agent
0.80
review (Tachycardia)
asdfghjkl
agent
< 0.70
open
Table 2. Score and outcome for sample terms (same run as Table 1)
Derivation-only (FULVESTRANT row): a WHODrug ingredient-level exact match, where the write-back updates only the generic-name derivation and ATC hierarchy fields. No new dictionary term is assigned.
The agent skipped the study-info tool for the unambiguous typo and the gibberish and called it only for the ambiguous cases. Every execution’s graph is inspectable in the Step Functions console, and AgentCore Observability records the agent’s tool calls and token usage in Amazon CloudWatch.
Note: Across repeated runs of identical inputs in this sample, the selected code was stable every time, but the confidence score moved within a band. Cases that land near the 0.90 autocode gate can therefore autocode on one run and route to review on another. In our testing, the oncology cramps case scored between 0.75 and 0.92 over six runs. It was autocoded twice and sent to review four times, and Muscle spasms was selected every time. Assert your tests on the selected code, not the exact score, and set your own autocode threshold with margin below 0.90 rather than directly on it.
Scaling beyond one verbatim: This walkthrough codes one verbatim per execution so each decision stays straightforward to inspect. To code a full study, wrap the workflow in a Step Functions Distributed Map state that reads verbatims from an Amazon Simple Storage Service (Amazon S3) manifest, and set MaxConcurrency based on your account’s Amazon Bedrock quota. To size MaxConcurrency, divide your model’s tokens-per-minute quota by the tokens one agent run burns (model calls * (input tokens + 10 * output tokens) for Sonnet 5). Then multiply by the runtime duration in minutes, divide by the share of verbatims that reach the agent, and apply a safety margin. In our testing, 12 concurrent executions against a cluster that had scaled to zero failed 8 of 12 while still reporting SUCCEEDED, which is why the sample keeps a 0.5 ACU minimum and retries the resume errors. Most verbatims resolve in the two deterministic steps, so only a small share ever reaches the agent. For the quotas that apply to your model, see Quotas for Amazon Bedrock.
Clean up
To avoid future charges, delete the deployed resources with npx cdk destroy. All sample data is reseedable, so nothing needs to be retained.
Recognizing the pattern
This pattern fits when the execution path must be reviewable independently of model reasoning, most of the workload is deterministic, only a judgment-heavy minority needs a model, and “escalate to a human” is a first-class outcome rather than an error. It is not limited to clinical trials: the same pattern applies to medical summaries, case narratives, registry intake, study data, and other adjudication workflows such as laboratory test coding, claims triage, and document classification.
You can extend the AgentCore harness with AgentCore Memory to carry context across invocations, Skills to provide task-specific instructions on demand, and AgentCore Gateway to connect MCP servers, knowledge bases, and other tools. Models, tools, and system prompts can also be selected per invocation from the Step Functions task state. Treat a model change like any other release. Rerun your evaluation set, compare the selected codes and confidence scores with the current model’s, and recalibrate the thresholds before you switch. Through all of this, the workflow remains responsible for sequencing and the audit trail.
This isn’t a pattern for open-ended conversations. If the agent must decide what happens next, a workflow engine that owns sequencing is the wrong boundary. Retrieval quality also constrains the agent: if a term isn’t surfaced, it can’t be chosen. In the sample, “flutters in my chest sometimes” would normally be coded to Palpitations, but vector search ranks Palpitations 19th, so the agent does not see it and selects Cardiac flutter instead. The confidence gate still sent it to a coder, but no prompt or study context can recover a term that retrieval never returns. For production dictionaries, use hybrid retrieval, such as lexical search or curated synonym expansion alongside vectors, and calibrate thresholds against your own verbatim distribution.
Conclusion
In this post, we showed how deterministic orchestration and agentic reasoning become complementary when the boundary between them is explicit. AWS Step Functions owns sequencing, thresholds, and the audit trail. The agent owns bounded judgment inside one step and reports a confidence the workflow routes on. The result is a pipeline that codes the common cases for the cost of a query, focuses skilled coders on the ambiguous minority, and routes terms below the confidence gate to a human. The execution history shows exactly which mechanism made each decision. The declarative harness integration keeps the agent inside that boundary as configuration a reviewer can read, not code inside a container.
For an airline operations team, a single flight cancellation sets off a chain reaction. Hundreds of passengers need new itineraries within minutes, and no two cases are alike. They have different loyalty tiers, sit on different fare rules, and have downstream connections that may not wait. Passengers have varying cabin and seat preferences and might fall under different regulatory entitlements depending on where they booked and where they are flying.
Most airlines handle this with a layered system: rule-based automation covers the simple, one-hop rebooks, and everything else flows to a manual queue staffed by service agents. That works when disruptions are isolated. When they are not, the queue overwhelms, waiting times spike, and passengers booked alternatives themselves that create downstream knock-on disruptions.
This is exactly where AI agents become compelling. An agent can reason across seat availability, fare rules, loyalty entitlements, and connection timing the way an experienced desk agent would, but at machine speed and across hundreds of cases in parallel. Multi-agent collaboration typically lets a supervisor agent route work to collaborator sub-agents, with the model itself deciding which sub-agent runs and in what order. But an unconstrained agent might optimize for the passenger’s preference while ignoring a codeshare restriction, rebook onto a flight that meets minimum connection time on paper but not at that specific airport, or calculate compensation under the wrong regulatory regime because it misread the ticket’s point of sale.
Orchestrating specialized Amazon Bedrock AgentCore agents with AWS Step Functions gives you the reasoning power of generative AI with the guardrails of deterministic validation. Step Functions adds native fan-out across thousands of passengers, a callback pattern that pauses a case for human review at zero compute cost, and a durable execution history that serves as your audit trail. The principle is that agents propose, and deterministic code validates. The pattern is demonstrated here for airline rebooking, but it applies anywhere automated decisions can have real financial or regulatory consequences.
Solution overview
The design is a Step Functions state machine where deterministic steps that map to the business processes wrap each agent’s non-deterministic behavior. The following diagram shows the end-to-end flow. At a high level, the workflow proceeds through these stages:
The workflow starts when a flight-cancellation event arrives, for example through an Amazon EventBridge integration.
An enrichment step pulls additional data such as the passenger manifest, current bookings, loyalty status, and stored preferences.
The workflow fans out to run agents in parallel for each affected passenger.
Two agents then run for each passenger: a find-alternatives agent proposes the top three rebooking options, and a compensation agent determines entitlement based on route, delay duration, and cause.
A deterministic validation step runs after each agent, confirming flights are actually bookable and entitlement rules are followed before either result is used.
The workflow checks whether the case can be auto-confirmed, or needs human review.
Bookings are confirmed, compensation issues, and confirmations are sent. Unresolved cases go to human agents.
The key principle: no agent Task state writes to the reservation system or issues a payment. Only deterministic Task states do that, and only after a deterministic validation step has passed.
Integrating AgentCore harness with Step Functions
AgentCore harness is a managed agent loop. You specify a model, system prompt, and tools, and the harness runs the reasoning cycle (model calls, tool execution, memory management, and response generation) end-to-end in a single API call. It handles the intra-agent orchestration so that Step Functions can focus on inter-agent orchestration: fan-out, sequencing, validation gates, and exception routing. Step Functions provides a native optimized integration for AgentCore harness, which calls InvokeHarness against a target HarnessArn. The optimized integration gives you an extended per-Task timeout of 15 minutes (900 seconds), so agents have enough time to reason through complex proposals. The trade-off is that the agent call is request-response only. There is no .sync and no .waitForTaskToken on the agent step, and only the final assistant message is returned to the state machine.
The following Amazon States Language snippet shows the optimized harness invocation inside a Distributed Map. For the full definition, see the sample on Serverless Land.
Note: the service name is spelled bedrockagentcore (no hyphen) in the Step Functions resource string, but bedrock-agentcore (with a hyphen) in the AgentCore ARN.
MaxConcurrency is set to 1000 to bound fan-out and protect downstream booking and inventory systems. If you omit it or set it to 0, you get the default behavior, which runs up to 10,000 parallel child executions. The agent Task flows directly into a deterministic validation Task.
How it differs from managed multi-agent collaboration
Multi-agent collaboration typically means that a supervisor agent decides which sub-agent runs and which tools it calls. Step Functions moves those decisions out of the agent layer entirely.
This design puts orchestration, fan-out, validation, routing, retries, and the audit trail into Step Functions instead. Routing is a deterministic state you define and can test in isolation, not a model classification you hope will be consistent. You get a per-state execution history (every transition recorded with input and output), whereas agent-layer traces require opt-in and provide reasoning rationale rather than a durable, always-on event log.
Design walkthrough of the reference app
The following image shows the Step Functions state machine implemented by the sample application.
Figure 1: The Step Functions state machine for the airline rebooking workflow
Stage 1, Trigger. An Amazon EventBridge rule starts the workflow on a flight-cancellation event.
Stage 2, Enrich. A deterministic Task pulls the passenger manifest, bookings, loyalty status, and preferences into the execution state.
Stage 3, Map fan-out. A Distributed Map iterates affected passengers in parallel. The choice of Map type matters at scale. An inline Map runs up to 40 concurrent iterations, which is the documented threshold for choosing Distributed mode. A Distributed Map runs up to 10,000 parallel child executions by default, the right tool when a hub event affects thousands of passengers.
Stage 4, Agent 1 find alternatives. An AgentCore Task proposes the top three options, reasoning over the passenger’s preferences and constraints.
Stage 5, Deterministic validation of the rebooking proposal. An AWS Lambda Task confirms each proposed flight is bookable by checking live availability, fare rules, and route validity, and it rejects hallucinated options. An agent might confidently propose a flight that does not exist. This stage is where that proposal is caught before it can become a ticket.
Stage 6a, Agent 2 draft compensation. A second AgentCore Task drafts personalized, customer-facing notification text only. It does not compute entitlement and it does not move money.
Stage 6b, Deterministic entitlement check. A Lambda Task computes and validates the entitlement against rule tables before any compensation issues. Consumer-protection frameworks such as EU Regulation 261/2004 (EU261) and US Department of Transportation refund rules are referenced here illustratively, to show why deterministic, auditable computation matters. The specific bands, triggers, and amounts are configuration you own and validate against current legal guidance, not something an agent should infer.
Stage 7, Choice routing and human-in-the-loop. A Choice state auto-confirms rebookings for some passengers and routes the rest to a human. For the cases that need review, the workflow waits on a separate .waitForTaskToken Task, backed by Lambda, Amazon Simple Notification Service (Amazon SNS), or Amazon Simple Queue Service (Amazon SQS), with a 4-hour timeout. The wait happens on this separate callback Task, never on the agent step.
{
"Comment": "Illustrative - route and wait on a human, not on the agent",
"RouteDecision": {
"Type": "Choice",
"Choices": [
{
"Variable": "$.passenger.autoConfirmEligible",
"BooleanEquals": true,
"Next": "ExecuteBooking"
}
],
"Default": "AwaitHumanApproval"
},
"AwaitHumanApproval": {
"Type": "Task",
"Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken",
"Parameters": {
"QueueUrl": "https://sqs.us-east-1.amazonaws.com/123456789012/approvals",
"MessageBody": {
"taskToken.$": "$$.Task.Token",
"passengerId.$": "$.passenger.id",
"options.$": "$.proposal.validatedOptions"
}
},
"TimeoutSeconds": 14400,
"Next": "ExecuteBooking"
}
}
Stage 8, Execute. Deterministic Task states confirm the booking, issue compensation, and send confirmation. Each execution Task derives an idempotency token from the passenger ID combined with the decision ID (the child execution name, or a hash of the validated option set) and passes it to the booking and payment APIs, so a retry or redrive is a no-op instead of a duplicate booking or a second payment.
Stage 9, Aggregate and exception routing. The workflow summarizes outcomes and routes any unresolved cases to human agents.
The validation step itself is ordinary deterministic code. A simplified rebooking validator in Python looks like the following.
# Illustrative - reject any option the agent proposed that is not bookable
def handler(event, context):
passenger = event["passenger"]
proposed = event["proposal"]["options"]
validated = []
for option in proposed:
flight = lookup_flight(option["flightId"])
if flight is None:
continue # hallucinated or stale flight, reject
if flight["seatsAvailable"] < 1:
continue # no inventory, reject
if not fare_rules_allow(passenger["fareClass"], flight):
continue # fare rule violation, reject
if not route_is_valid(passenger["origin"], passenger["destination"], flight):
continue # invalid route, reject
validated.append(option)
return {
"passengerId": passenger["id"],
"validatedOptions": validated,
"autoConfirmEligible": passenger["loyaltyTier"] == "top" and len(validated) > 0,
}
Best practices and guardrails
Reject hallucinations through validations. No agent proposal is applied without a deterministic validation step passing first. This minimizes the impact of hallucinations, prompt injections, or bugs on your workflow.
Keep a complete audit trail. Step Functions execution history records every state transition, input, and output, and pairing that with durable persistence gives you a per-decision record. You can show exactly which proposal was made, which validation passed or failed, and who approved the exception.
Surface only true exceptions to humans. Humans handle only what validation or the agent cannot resolve. Auto-confirmation handles the clear cases, and people spend their attention on the genuinely ambiguous ones.
Hold executions open cheaply. The .waitForTaskToken callback holds the execution open with no compute charges while the execution is paused. For example, you can cost-efficiently park thousands of pending approvals overnight. Refer to the AWS Step Functions pricing page for current details.
Make execution idempotent. Guard reservation execution and compensation issuance against retries and double-sends, as shown in Stage 8. Derive the idempotency token from the passenger ID and decision ID, and pass it to your booking and payment APIs so that a replay is a no-op.
Respect cost and timeouts. Keep each per-agent Task timeout within the 15-minute quota, bound your Map concurrency to protect downstream systems, and track the token usage returned in the agent response so you can attribute and forecast cost.
Handle errors deliberately. Apply Retry and Catch on the agent Tasks for conditions such as BedrockAgentCore.ThrottlingException and BedrockAgentCore.ResourceNotFoundException, and on the Lambda validation Tasks for their own failure modes. A Catch on an agent Task can route a stuck passenger straight to the human queue rather than failing the whole child execution.
Confirm availability and Region support. Check the current availability status and supported AWS Regions for AgentCore and the Step Functions integration at the AWS Capabilities by Region on Builder Center.
Conclusion
A flight-cancellation event is a challenging test of automated decision-making, because the output can have immediate financial impact. The way to use AI agents safely in that setting is to let them do what they are good at, proposing options and drafting language, while never letting a proposal become an action until deterministic code has approved it. In this design, orchestration, fan-out, validation, routing, and retries are implemented in Step Functions rather than inside an agent’s reasoning. Agents do not make changes directly, and their output is only applied after deterministic validation. You get a per-decision record for review, and you hold exceptions open on a callback that adds no compute or storage cost while it waits.
You want to build an AWS Step Functions workflow, and you have an AI coding agent open in your terminal or IDE. But the agent doesn’t know about Amazon States Language (ASL), service integrations, or how to deploy state machines. Before you can start, you need to find the right Model Context Protocol (MCP) server package, figure out the configuration format for your specific agent, and set up credentials.
AWS Step Functions has added a “Copy agent prompt” button to the AWS Step Functions console that removes this setup entirely. You choose the button, paste the prompt into your agent, and the agent configures itself with Serverless skills and an MCP server. You can start building workflows with natural language immediately. The feature works with Claude Code, Kiro CLI, Cursor, GitHub Copilot, Codex, Devin Desktop, OpenCode, and any other MCP-compatible agent.
How it works
The button appears in three places in the Step Functions console:
The home page, under “How it works”.
The Create State Machine modal (at the top, before you begin building).
The Local Development section on the home page.
Here’s an example from the Create State Machine flow:
Open the Step Functions console and choose Create state machine.
At the top of the modal, you see the banner: “Set up your agent to build with Step Functions. Copy and paste this prompt into your AI agent to set up Step Functions skills and MCP server.”
Choose Copy agent prompt. The console copies a fetch instruction to your clipboard.
Paste the prompt into your AI agent’s chat or terminal.
The agent reads the setup guide and self-configures.
The copied prompt is a fetch instruction that points to a setup guide hosted on AWS documentation. You paste it into your agent, and the agent installs two things:
AWS Serverless skill (from the Agent Toolkit for AWS) provides your agent with deep context on Step Functions. It includes how to write ASL, structure workflows with retries and error handling, choose between Standard and Express workflow types, implement patterns like saga orchestration and parallel fan-out, and deploy using AWS Serverless Application Model (AWS SAM) or AWS Cloud Development Kit (AWS CDK).
AWS Serverless MCP Server gives your agent direct access to AWS. Through the Model Context Protocol, your agent can create and update state machines, start and describe executions, inspect workflow history, and manage resources in your account.
Supported agents
The setup guide auto-detects your agent and provides the correct configuration format:
Claude Code: Installs through the plugin marketplace and registers the MCP server with claude mcp add.
Kiro CLI: Writes to ~/.kiro/settings/mcp.json.
Codex: Registers with codex mcp add.
Cursor: Writes to .cursor/mcp.json.
GitHub Copilot: Writes to .vscode/mcp.json.
Devin Desktop: Writes to .devin/mcp_config.json.
OpenCode: Writes to ~/.config/opencode/opencode.jsonc.
If you use a different MCP-compatible agent, the guide provides a generic JSON configuration block you can add to your agent’s config file.
What you can build
Once your agent is configured, you can describe workflows in natural language, and the agent produces valid, deployable state machines. Here are a few examples:
Order processing with compensation: “Build a workflow that validates a payment, reserves inventory and sends a confirmation email. If payment fails, release the inventory reservation.”
Parallel fan-out: “Create an Express workflow that calls three AWS Lambda functions in parallel, waits for all to complete, and merges the results into a single response.”
Human approval gate: “Add a step that pauses the workflow and waits for a manager to approve before proceeding with the deployment.”
Error handling: “Add retry with exponential backoff and a maximum of three attempts to the payment processing step. If all retries fail, route to a fallback notification step.”
Because the agent has the MCP server connected, it can also deploy the workflow directly to your account, start test executions, and inspect the results without leaving the agent interface.
Advantages
Always current: The Agent Toolkit for AWS content stays up to date as Step Functions adds new features, integrations, and patterns. When you run the prompt, your agent gets the latest skills and configurations automatically.
No context switching: You stay in your agent’s interface for the entire workflow: design, build, deploy, test, and iterate. No switching between the console, documentation, and your editor.
Works with your existing credentials: The MCP server uses your local AWS profile. No new AWS Identity and Access Management (IAM) roles or permissions are required beyond what you already use for Step Functions development.
Agent-agnostic: Whether you use Claude Code, Kiro, Cursor, Copilot, or another tool, the same button and prompt works. You don’t need to find agent-specific setup instructions.
Choose Copy agent prompt from the banner (on the home page under “How it works,” in the Local Development section, or in the Create State Machine modal).
Paste the prompt into your AI coding agent.
Start describing the workflow you want to build.
This feature is available in all commercial AWS Regions at no additional cost. To learn more about the setup process, see the agent setup guide. For more on the Agent Toolkit for AWS, see the GitHub repository. For AWS MCP Servers, see the documentation.
We’d like to hear how you use this feature. Tell us about it in the comments.
Organizations in regulated industries like financial services, insurance, healthcare, and government ingest large volumes of data containing personally identifiable information (PII). Your applications, claims processing systems, partner data feeds, and internal workflows produce files that may include names, addresses, Social Security numbers, and domain-specific identifiers such as policy numbers, member IDs, and medical record numbers.
Identifying and classifying this sensitive data is a compliance imperative. Regulations such as GDPR, HIPAA, CCPA, and PCI DSS require you to know where PII resides, who can access it, and how it is protected. Manual classification does not scale. While Amazon Macie provides managed sensitive-data discovery, there could be formats that standard detection engines do not recognize by default. You can extend the detection capability with custom identifiers unique to your business.
In this post, you learn how to build and deploy an event-driven pipeline that automatically detects PII the moment a file arrives in Amazon Simple Storage Service (Amazon S3). It then extends beyond Macie built-in managed data identifiers by incorporating your own custom identifiers, which can detect organization-specific sensitive data types alongside standard PII.
Solution overview
The solution uses Amazon EventBridge to trigger an AWS Step Functions workflow, which orchestrates Macie classification jobs, including custom data identifiers, and produces compliance reports with zero manual intervention. As each file lands in Amazon S3, the pipeline automatically invokes Macie to scan it using built-in and custom data identifiers, detecting standard and organization-specific PII without manual intervention. It then generates audit-ready reports in both CSV and JSON formats with full timestamps, giving your compliance team the documentation they need. For high-severity findings, the pipeline publishes real-time notifications through Amazon Simple Notification Service (Amazon SNS) so your security team can respond immediately. To maintain clear data lineage throughout the process, the architecture implements a three-bucket pattern. This helps isolate objects by their processing state and ensures that unscanned data never mixes with validated data.
Architecture
This pipeline uses an event-driven architecture built on the following AWS services:
Amazon S3 – Storage layer organizing data across three buckets that represent each processing state: raw, staged, and scanned.
Amazon EventBridge – Detects new object uploads and triggers the workflow automatically, without polling.
AWS Step Functions – Orchestrates the entire scan lifecycle, coordinating each step from job creation through report generation.
Amazon Macie – Detects PII, scanning objects for sensitive data using both built-in and custom data identifiers.
AWS Lambda – Supplies the compute logic between orchestration steps from initiating Macie classification jobs, polling job status, parsing findings, generating reports, and moving objects between buckets.
Amazon SNS – Delivers real-time alerts to notify your team when high-severity findings are detected.
The following figure provides an overview of the solution architecture using the listed services.
Figure 1 — Architecture overview
Several design decisions shape this architecture. The pipeline creates a dedicated Macie classification job for each object rather than batching, which enables real-time PII detection at the point of ingestion. The three-bucket pattern noted earlier isolates unprocessed data from validated data, with the staging bucket configured to auto-expire objects. Step Functions orchestrates the workflow, using built-in retry logic and wait states to handle asynchronous job execution in Macie. Custom data identifiers extend detection capabilities of Macie to cover domain-specific patterns unique to your organization. Finally, Amazon EventBridge triggers the pipeline instead of native S3 event notifications, providing advanced filtering, support for multiple targets, and cross-account routing.
Walkthrough
In this section, you deploy and configure the automated PII detection pipeline.
Prerequisites
A sandbox or non-prod AWS account with administrative access.
Open the confirmation email and choose Confirm subscription.
✓ Expected result: Subscription shows Confirmed in Amazon SNS console as demonstrated in the following screenshot
Figure 3 — Amazon SNS confirmed subscription
Step 3: Configure custom data identifiers
The stack automatically provisions custom data identifiers from the CustomIdentifierPatterns parameter you provided during deployment. Verify they exist in the Macie console.
✓ Expected result: Step Functions execution starts within seconds.
The following image shows the test files copied to the S3 bucket:
Figure 5 — Test file uploaded to raw bucket
Monitor the pipeline progression
On your Step Functions console > choose state machine > view running operation.
Note: The underlying Macie jobs may take some time to complete. Scan duration varies from job to job. The amount and type of data being scanned, its compression, and the number of data identifiers used all dictate how long a job takes to run. For the example illustrated in this post, jobs took roughly 15–20 minutes to complete. As a best practice, consider enabling S3 AWS CloudTrail data-event logging to audit the PII pipeline.
✓ Expected result: Graph inspector shows active state in blue, completed in green.
Figure 6 — Step Functions pipeline
Figure 7 — Step Functions Graph inspector
The Step Functions state machine orchestrates the end-to-end scanning workflow through five sequential states. First, TriggerScan copies newly uploaded objects to a staging bucket and creates a Macie classification job targeting that staged data. Once the job is submitted, the workflow enters WaitForMacie, which pauses execution for 60 seconds to allow the classification job time to process. Next, CheckStatus polls the Macie DescribeClassificationJob API to determine whether the job has completed. When the job finishes, GetFindings retrieves the classification results, generates detailed findings reports, and publishes a notification to an Amazon SNS topic for downstream consumers. Finally, MoveFiles relocates the original object from the raw bucket to a permanent scanned-data location and cleans up the staging copy.
Verify the reports
aws s3 ls s3://macie-pii-pipeline-<ACCOUNT_ID>-scanned/reports/
✓ Expected result: Two timestamped files (CSV + JSON) per scan as shown in the following image
Figure 8 — Reports in scanned bucket
Next, Amazon SNS triggers a notification email to the intended recipients. The following screenshot shows a sample notification.
Figure 9 — Amazon SNS notification email
Step 5: Review the data flow
raw bucket → stage bucket → scanned bucket
Raw bucket – The file is removed after processing (versioning enabled).
Stage bucket – Auto-expires after seven days.
Scanned bucket – Contains the processed files and the reports.
The following images show the three sample buckets created as part of this walkthrough.
Figure 10 — Three-bucket state after execution
Hardening measures to deploy in production
For multi-tenant environments, deploy a separate stack per tenant using a distinct BucketNamePrefix to maintain strict data isolation.
Configure cross-account Amazon EventBridge rules for data sources spanning multiple accounts.
Be mindful of Macie quotas for custom data identifiers: up to 10,000 per account, but a maximum of 30 per classification job. Split them across multiple jobs if you need more.
For high-volume workloads, batch multiple objects into a single Macie classification job to reduce API overhead and stay within the CreateClassificationJob throttle of 0.1 requests per second (one job every 10 seconds). See Macie quotas.
Enable Amazon S3 AWS CloudTrail events to maintain a complete audit trail of every object access and movement throughout the pipeline.
Data lake integration – Route JSON findings to Amazon Athena or AWS Glue for trend analysis. Build Amazon QuickSight dashboards for PII density visualization.
Remediation workflows – Extend Step Functions with a remediation branch to quarantine files or revoke access when high-severity PII is found.
Multi-account environments – Use AWS Organizations and Amazon EventBridge cross-account rules to centralize scanning in a dedicated security account.
Enhanced security posture – Publish findings to AWS Security Hub for unified security posture management.
Batch for high-volume – Modify Trigger Scan to batch objects into a single Macie job for cost optimization at scale.
Cleaning up
To avoid ongoing charges, delete all resources created by this solution:
Empty all S3 buckets created by the stack. CloudFormation cannot delete non-empty buckets.
In this post, we demonstrated building an event-driven pipeline that automatically detects PII in files uploaded to Amazon S3. By combining Macie’s managed classification with custom data identifiers, orchestrated by AWS Step Functions and triggered in real time through Amazon EventBridge, the solution detects both standard and domain-specific sensitive data without manual intervention. The pipeline responds to S3 bucket uploads within seconds, runs each object through the full scan lifecycle, generates timestamped CSV and JSON reports for compliance, and delivers real-time Amazon SNS notifications when high-severity findings are detected.
Getting started
To deploy this solution in your own environment, clone the GitHub repository and follow the step-by-step deployment instructions in the README. This is a non-prod sample code without the listed hardening measures. You can extend the pipeline by adding custom data identifiers for your organization’s unique data formats, configuring cross-account Amazon EventBridge rules for multi-account environments, or batching objects per Macie job to optimize for high-volume workloads.
In this 33rd quarterly recap post, discover the most impactful AWS serverless launches, features, and resources from Q2 2026 that you might have missed. Stay current with the latest serverless innovations that can improve your applications.
In case you missed our last ICYMI, read about what happened in Q1 2026.
AWS Lambda MicroVMs
AWS Lambda MicroVMs is a new serverless compute primitive for running user or AI-generated code in isolated, stateful execution environments. Built on the same Firecracker virtualization that powers over 15 trillion monthly Lambda invocations, MicroVMs give you VM-level isolation with near-instant launch and resume. Each MicroVM runs in its own Linux environment with no shared kernel or resources between sessions. This isolation makes it a useful solution for AI coding assistant sandboxes, interactive code or multi-tenant development environments, CI/CD build environments, data analytics platforms, vulnerability scanners, and game servers that run user-supplied scripts.
Standard Lambda functions are best for event-driven, request-response workloads which have a 15-minute timeout. MicroVMs are purpose-built for single end user or session workloads and can preserve state for up to 8 hours. You get full lifecycle controls including launch, suspend, resume, and terminate. You can suspend them during the 8 hours if you don’t need them active. MicroVMs retain memory and disk state for the length of the session, even while suspended. They can auto resume when you need to use them again.
Amazon S3 Files makes your S3 buckets accessible as high-performance file systems. S3 files is a fully featured, POSIX-compatible file system to access to your data with approximately 1ms latency.
For serverless workloads, the Lambda integration with S3 Files lets your functions mount an S3 bucket as a local file system. Your function reads and writes files at a local mount path like /mnt/data, and the file system handles synchronization with S3 automatically. You can avoid downloading objects to /tmp from S3 within your function and work directly with files. Applications that assume a file system can now run on Lambda without rewriting their I/O layer. Use cases include sharing data between functions, ML model loading, document processing, media transcoding, or any pipeline that treats data as files rather than objects.
AWS Lambda durable functions
The Lambda durable functions SDK for Java is now generally available, joining Python and TypeScript. This allows Java developers to build multi-step workflows with automatic checkpointing and recovery without adding external orchestration. Durable functions is also now available in 16 additional AWS Regions. Learn how to build fault-tolerant multi-agent AI workflows to coordinate multiple AI agents that call tools, make decisions, and hand off work. There is automatic recovery if any agent fails mid-task. Voice analytics with Amazon Bedrock shows building a pipeline that processes call recordings through transcription, sentiment analysis, and summarization with durable checkpoints between each stage. For best practices, AI patterns, and futures, view the live stream.
AWS Lambda Managed Instances
Lambda Managed Instances now allows you to build memory-intensive apps with up to 32 GB (3x more than standard Lambda). This allows use cases like in-memory caching, large dataset analytics, and ML inference that previously required considering other services.
Scheduled scaling lets you pre-warm capacity for predictable traffic patterns with Amazon EventBridge Scheduler. This helps reduce cold start latency during known demand spikes. Tag propagation automatically applies your function tags to the underlying Amazon EC2 instances, Amazon Elastic Block Store volumes, and network interfaces. This helps finance teams with cost allocation visibility without manual tag management.
Other Lambda updates
Response streaming is now available in all commercial AWS Regions, bringing full regional parity for progressively streaming data back to clients. This is useful for LLM-powered applications where users expect to see tokens as they generate rather than waiting for a complete response.
If you have a fleet of functions on older runtimes, you can now upgrade runtimes at scale using AWS Transform custom. This uses AI to analyze your function code, identify breaking changes for the target runtime version, and generate the code modifications needed. This can help teams save manual migration effort across many functions. The Serverless Office Hours live stream has more information.
AWS Serverless Application Model (AWS SAM) CLI now supports BuildKit for building container images from Dockerfiles. This allows faster multi-stage builds with better caching, cross-architecture image builds, and Docker secrets to keep credentials out of final image layers.
Containers with Mama J
Serverless with Mama J
Mama J is back in the second video of a series where Eric Johnson explains what he does all day at work to his mother. Previously, they talked serverless and Lambda. This time it’s containers, what they are, why they exist, and how AWS manages them at scale. Eric goes through the “it works on my machine” problem, how Docker builds images, container orchestration and how containers differ from Lambda.
AWS Step Functions has an Amazon Bedrock AgentCore-powered agentic reasoning step. You can embed AI agent reasoning directly inside a workflow as a native step type. This bridges structured orchestration with autonomous agent behavior. Your workflow handles the deterministic parts such as branching, retries, timeouts, parallel execution, while the agentic step handles the parts that require flexible reasoning.
The Amazon BedrockAgentCore harness reached general availability. Two API calls give you a running agent in seconds which runs in its own isolated environment with a filesystem and shell. It can read files, run commands, and write code safely.
AgentCore Payments (preview) allows agents to autonomously access and pay for APIs and MCP servers, opening up agent-to-agent commerce. AgentCore Memory has metadata for long-term memory so agents retain and recall context across sessions. Web Search on AgentCore grounds agents in current, cited web knowledge. The Runtime now supports bring-your-own file systems from S3 Files and Amazon Elastic File System, and Node.js for direct code deployment.
Strands Agents SDK
The open source Strands Agents SDK shipped three capabilities. Context management that cuts token costs in half by intelligently pruning what goes into the model context window, Strands Shell for sandboxed agent code execution, and Strands Evals 1.0 with chaos testing and adversarial red teaming. This can reduce costs to help make production agent workloads cheaper without sacrificing quality. A Serverless Office Hours live stream covered the new features in depth.
The Agent Toolkit for AWS became generally available with three plugins (aws-core, aws-agents, aws-data-analytics), over 30 curated skills, and the AWS MCP Server. View this video for an introduction. This gives AI coding agents such as Kiro, Claude Code, and Cursor expert AWS knowledge which helps to reduce errors and lower token costs. For more information on the serverless tools available when using AI, see this Serverless Land resources page.
Serverless Office Hours ran a live stream series finding out how experts use AI to build serverless applications. Hear from:
Kiro launched Kiro Pro Max and an iOS mobile app for approving and monitoring agentic coding sessions from your phone. Amazon Q Developer IDE plugins are transitioning to Kiro. The Kiro power for AWS DevOps Agent connects your IDE directly to production intelligence. You can investigate incidents and generate fixes without context switching.
Join our live stream every Tuesday at 11 AM PT for live discussions, Q&A sessions, and deep dives into serverless technologies. View episodes on-demand at serverlessland.com/office-hours.
The Serverless landing page has overall information about building serverless applications. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.
You can also follow the Developer Advocacy team to get the latest news, follow conversations, and interact with the team.
Specification-driven composition addresses a common scalability bottleneck in data pipelines. Data pipelines often start as simple scripts, but as they grow, you duplicate transformation logic and small changes cascade across multiple workflows. Copying and modifying data transformation logic across scripts leads to workflows that become difficult to manage at scale. Tracking what each pipeline does becomes harder because workflow intent is embedded in code. This lack of visibility complicates governance, especially in regulated environments such as healthcare, finance, and life sciences.
Many implementations combine orchestration, transformation logic, and validation rules in the same scripts. Supporting new datasets requires modifying and redeploying code, while validation often happens only during processing. As a result, issues surface late in the lifecycle which increases the operational risk. Specification-driven composition separates workflow intent from implementation so you can build flexible data workflows.
You can separate workflow intent from processing logic with specification-driven composition. This approach reduces duplication, shortens the time required to onboard new datasets, and improves consistency across workflows. Instead of embedding logic in scripts, the system describes workflow intent in a structured specification, validates the specification before processing, and dynamically assembles a processing pipeline.
This approach moves pipeline configuration outside application code and composes pipelines from reusable processing components. To separate concerns, it organizes a workflow into three layers, as shown in Figure 1. The intent layer defines workflow behavior using specifications. The composition layer validates specifications and assembles pipelines. The processing layer runs the pipeline of transformation steps.
Specification-driven composition provides several practical benefits when you manage multiple pipelines. First, it improves governance because specifications provide a clear, traceable description of workflow behavior that you can review and validate before invocation. In regulated industries, this traceability shortens audit preparation time and reduces the review burden for new dataset submissions.
Second, the pattern supports reusable transformations. You implement transformation logic once and reuse it across multiple workflows. This reduces duplication and improves consistency. In practice, teams adopting this pattern report being able to onboard new datasets in days rather than weeks because most required capabilities already exist in the registry.
Third, specification-driven composition enables flexible pipeline design. You define new specifications to create pipelines supporting new datasets and use cases without modifying application code and registering new system release.
Fourth, the pattern separates business intent from execution artifacts. The specification expresses what the workflow should do in domain terms, while the generated state machine remains a system artifact. This separation matters in regulated environments (for example, GxP) where business users author intent but are not allowed to author or modify execution code directly.
Finally, the declarative representation of workflows lets AI tools assist with capability discovery, specification authoring, and pipeline analysis, while runtime behavior stays predictable as it relies on validated capabilities.
Core components
You work with four key components to define, validate, and run workflows.
Specification
A specification is a structured document, typically JSON or YAML, that describes datasets, mappings, and transformations. It defines what the workflow should do without including processing logic. Because specifications are explicit and versioned, they provide a clear record of workflow intent.
Composer
The composer converts specifications into runnable steps, so workflows run consistently without embedding transformation logic in application code. It checks that referenced capabilities exist, retrieves metadata, and builds a workflow that can run on the processing layer. The composer does not perform transformations. It only assembles the workflow. In practice, the composer compiles business intent expressed in the specification into a code artifact such as an Amazon States Language (ASL) definition. This abstraction lets domain users author specifications without producing runnable code, which is important in environments with strict separation of duties.
Capability registry
The registry stores metadata about reusable transformation functions. This includes identifiers, input/output formats, invocation details, and permission boundaries. The composer uses the registry to validate specifications and locate capabilities. Treat this registry as a governed artifact rather than a manually edited lookup table. Capability definitions live in version control, and your CI/CD pipeline validates metadata and runs tests before you publish new versions. Specification authors include explicit capability version references in the specifications to support reproducible workflow runs. In regulated systems, you must validate new capabilities and obtain approval through a separate workflow before you register them.
Capability pipeline
Once assembled, the pipeline runs a sequence of transformation steps. Each step performs a specific operation such as formatting, validation, or enrichment. Because these steps are reusable, you can apply them across many workflows.
Technical implementation
Let’s walk through a technical implementation of this pattern using serverless AWS services. In this example, workflow specifications are uploaded to an S3 bucket. An AWS Lambda composer retrieves and validates the specification, looking up capability metadata in Amazon OpenSearch Service. The composer then assembles the workflow in AWS Step Functions, which orchestrates the workflow and invokes AWS Lambda capability processors. Each processor emits traces to Amazon CloudWatch Logs.
Figure 2 shows the architecture.
Figure 2. AWS implementation of Specification-Driven Composition
Interpreting the workflow specification
The workflow specification defines datasets and transformation logic using a structured format (in this example, JSON). A specification is a declarative document that describes what the pipeline should produce rather than how to produce it. Your composer reads the specification, validates it against a schema, and uses it to construct the pipeline.
The following example maps source fields to target fields using reusable capabilities (Figure 3). It takes data from a source dataset (‘raw_orders’), maps specific fields (‘order_date’, ‘amount’) and applies reusable transformation capabilities such as ‘format_date’ and ‘normalize_currency’. Each mapping explicitly links a source field to a target field and references a capability that performs the transformation. Source dataset(s) are listed under the ‘source’ section of the JSON document, target datasets under the ‘target’ section, and mappings ‘mappings’. You can define your own specification structure and build custom validation logic in your composer to make sure specifications are valid.
You can model preprocessing steps such as column standardization, numeric casting, or unit conversion as first-class capabilities and reference them earlier in the specification (for example, in a dedicated ‘preprocessing’ section) before downstream mappings run. This way, preparation logic uses the same metadata, versioning, validation, and observability model as the rest of the workflow, which simplifies lineage and review.
In this example, an S3 event notification invokes the composer Lambda function when you upload a specification. In practice, the same composer can be invoked through several mechanisms depending on the use case: S3 events for new specifications, an Amazon EventBridge schedule for recurring runs, an API or UI action for on-demand invocation, a direct Step Functions StartExecution call, or an upstream pipeline. This flexibility lets you re-run an approved specification against refreshed data without re-authoring or re-approving the workflow.
The composer parses the specification and validates the referenced capabilities by querying Amazon OpenSearch Service to retrieve capability metadata such as Amazon Resource Names (ARNs). OpenSearch Service is used here because the composer does more than direct-key lookups. It supports capability discovery through full-text and semantic search over capability metadata such as capability descriptions, input/output schemas, and tags. This lets data authors and AI tools find reusable capabilities by intent rather than by exact identifier. After validation, the composer assembles and starts an AWS Step Functions state machine which invokes each capability in sequence. Each capability runs independently, making the pipeline modular and reusable.
Securing sensitive data flows
This pattern suits regulated workloads, so handle security as part of the design. Use SSE-KMS with a customer managed key on the specification and data S3 buckets and enable encryption at rest on the Amazon OpenSearch Service domain. Enforce HTTPS (TLS) access with an S3 bucket policy (aws:SecureTransport) and enable node-to-node encryption on Amazon OpenSearch Service. AWS Step Functions and Lambda calls use TLS by default. Each capability processor receives only the fields its mapping references, and you can apply IAM policy at the source bucket to restrict access. Processors emit traces to Amazon CloudWatch Logs.
For data classification, you can tag sensitive fields in the specification (example: "sensitivity": "PHI") and declare the sensitivity of each capability in the registry: a direct-move capability preserves the source classification, a date-of-birth-to-age capability clears it, and an enrichment that introduces sensitive data sets it. The composer combines the source tag with the capability’s behavior to derive the target field’s sensitivity, validating that the combination resolves clearly before assembling the pipeline. It then generates the masking artifact for the output (for example, AWS Lake Formation column grants) so consumers get correct masking without a separate masking specification or manual effort.
Recognizing the pattern
This approach works well when you describe workflows using structured specifications, reuse transformation logic across pipelines, require validation before invocation, and require workflows to be deterministic and auditable. You are likely to recognize the pattern in regulated data pipelines for the submission of tabular datasets to oversight agencies.
Consider clinical trial reporting as a concrete example. Data analysts collect raw data from clinical sites and must transform it into a standard submission format such as the Study Data Tabulation Model (SDTM) before submission to agencies such as the US Food and Drug Administration. With specification-driven composition, data analysts define specifications that map collected data about adverse events, demographics, and vital signs to the standard target variables, and the validated output feeds downstream systems such as patient safety and medical monitoring.
That said, this pattern might add unnecessary complexity for simple, one-time data transformations or pipelines with fewer than three to five workflows. Evaluate this pattern’s impact by tracking the reduction in duplicated transformation logic across pipelines, the time required to onboard new datasets, and the number of workflows you can create without modifying application code.
Conclusion
Script-based data pipelines accumulate hidden costs as they grow, including duplicated logic across files and late-breaking validation failures. Specification-driven composition separates workflow intent from processing so you can manage data workflows more consistently at scale. The result is faster dataset onboarding, stronger governance, and pipelines that are transparent enough to trust in regulated environments.
This pattern is especially valuable for regulated reporting pipelines, multi-source data integration, and reusable ETL frameworks where traceability and flexibility matter. By investing a small set of reusable capabilities and a disciplined specification format, you can reduce the engineering effort for new pipelines by treating them more as configuration tasks which may be delegated to system users.
For a practical first use case, try applying the pattern to a reporting pipeline you maintain today that has three or more variants such as monthly finance reports generated from different source systems. Replace the duplicated scripts with a single composer and a shared capability library, and measure the onboarding time for the next variant. As you expand your capability library, you can apply the same pattern across additional workflows and standardize how transformations are defined and run.
Whether you’re processing financial data, managing e-commerce orders, or training machine learning (ML) models, efficiently coordinating complex processes is essential. Amazon Web Services (AWS) offers two services for workflow orchestration: Amazon Managed Workflows for Apache Airflow (Amazon MWAA) and AWS Step Functions.
This post explores how to select the right workflow orchestration service based on your specific use case requirements. We’ll examine key workflow characteristics, present real-world scenarios, and provide practical guidance to help you make an informed decision for your particular needs.
Understanding workflow orchestration requirements
Before exploring specific services, consider the key dimensions that influence workflow orchestration needs:
Data statefulness: Does your workflow process independent units of work (stateless) or create dependencies where each step modifies data from previous steps (stateful)?
Execution duration: Are your workflows short-lived (seconds to minutes) or long-running (hours to days)?
Scheduling requirements: Do you need built-in time-based execution or rely primarily on event triggers?
Recovery capabilities: How critical is the ability to restart from specific failure points rather than reprocessing entirely?
Integration complexity: What systems, services, and data sources need to be coordinated?
Security and access control: Do you need fine-grained permissions for different workflow components?
Let’s explore how these requirements map to real-world use cases and the appropriate orchestration solutions.
Use case: Enterprise data analytics pipeline
This scenario illustrates how Amazon MWAA handles complex, stateful data pipelines with built-in scheduling and granular recovery.
Business challenge
A global financial services company processes massive volumes of transaction data daily, requiring sophisticated data analytics capabilities. Their requirements include:
Designed to process 5-10 TB of financial transaction data daily
Running complex extract, transform, and load (ETL) jobs with multiple transformation stages
Generating regulatory reports for compliance use cases
Supporting both scheduled batch processing and event-driven workflows
Capable of handling long-running jobs that can take up to 12 hours
Ensuring data consistency and integrity throughout the pipeline
Workflow characteristics
Data statefulness: Highly stateful workflows where each processing step modifies transaction data, creating dependencies throughout the pipeline
Scheduling needs: Mixed time-based and event-driven patterns
Recovery requirements: Critical ability to resume from specific failure points
Integration complexity: Orchestrates multiple AWS services and external systems
Solution: Amazon Managed Workflows for Apache Airflow (Amazon MWAA)
For this enterprise data analytics scenario, Amazon MWAA provides capabilities that align well with these requirements:
Stateful workflow management
MWAA excels at managing complex, stateful data pipelines where data consistency is critical. When processing terabytes of financial data, MWAA’s ability to resume from the last successful checkpoint helps prevent costly reprocessing and maintain data integrity.
The following code example demonstrates how to structure a complex financial ETL pipeline in MWAA:
# Example: Complex ETL pipeline with proper dependency management
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timedelta
dag = DAG(
'financial_etl_pipeline',
schedule_interval='0 2 * * *', # Daily at 2 AM
start_date=datetime(2024, 1, 1),
catchup=False
)
# Define tasks
extract_transactions = PythonOperator(task_id='extract_transactions', ...)
extract_market_data = PythonOperator(task_id='extract_market_data', ...)
transform_data = PythonOperator(task_id='transform_data', ...)
load_warehouse = PythonOperator(task_id='load_warehouse', ...)
generate_reports = PythonOperator(task_id='generate_reports', ...)
# Express complex dependencies clearly
[extract_transactions, extract_market_data] >> transform_data >> [load_warehouse, generate_reports]
This Directed Acyclic Graph (DAG) shows how to define task dependencies for parallel data extraction followed by sequential transformation and loading operations. The >> operator clearly defines the workflow dependencies. Transformation only begins after both extraction tasks complete successfully.
Built-in scheduling capabilities
MWAA includes native scheduling capabilities, making it straightforward to set up recurring workflows without additional services. The schedule_interval parameter in the DAG definition provides flexible scheduling options using cron syntax.
Granular recovery and resume control
During production incidents, operations teams can use the MWAA web interface to restart or bypass specific steps with a few clicks. This capability is important for stateful applications where restarting the entire workflow could compromise data consistency.
The MWAA web interface provides a visual representation of the workflow execution, allowing operators to:
Identify failed tasks – Examine task logs for troubleshooting – Clear the status of specific tasks – Restart execution from specific points
Figure 1: A Directed Acyclic Graph (DAG) in MWAA showing parallel execution ofAmazon Redshift Data APItasks. If any task fails, you can re-run specific tasks rather than restarting from the beginning.
Infrastructure planning: While MWAA requires capacity planning, the automatic scaling capabilities effectively handle variable workloads by setting minimum and maximum worker counts.
Security model: MWAA uses a shared execution role across DAGs, but you can implement additional security through resource-level policies and separate environments for different teams.
Cost predictability: The worker-hour pricing model provides predictable costs for long-running jobs, making budget planning more straightforward.
Use case: Real-time serverless application orchestration
This scenario shows how AWS Step Functions handles event-driven, serverless workflows that need to scale automatically with unpredictable traffic.
Business challenge
An e-commerce platform needs to orchestrate real-time order processing workflows that can handle thousands of concurrent orders during peak shopping periods. Their requirements include:
Designed for processing customer orders in real-time (targeting sub-second response times)
Coordinating payment validation, inventory checks, and fulfillment
For this real-time e-commerce scenario, AWS Step Functions provides capabilities that align well with these requirements:
Serverless architecture and automatic scaling
Step Functions automatically scales to handle traffic spikes without infrastructure management. During peak shopping events like Black Friday, the service handles increased load without manual intervention.
Event-driven workflow execution
Step Functions is designed for order-triggered workflows that need immediate execution. The following JSON definition shows how to structure an e-commerce order processing workflow:
This Step Functions definition demonstrates several key capabilities: – The ValidatePayment state includes built-in retry logic with exponential backoff – The CheckInventory state uses parallel execution to simultaneously check multiple warehouses – Each Lambda function is called via its Amazon Resource Name (ARN), providing direct integration with AWS services
Figure 2: A complex workflow in AWS Step Functions, involving multiple stages of data processing. The parallel execution doesn’t allow resuming from a specific mid-execution step, but the branching structure provides automated error handling and recovery.
Native AWS service integration
Step Functions provides direct integration with Lambda functions, SQS queues, SNS topics, and DynamoDB, eliminating the need for custom connectors or additional infrastructure components.
Cost-effective pay-per-use model
The pay-per-execution pricing model aligns with variable order volumes, keeping costs minimal during slow periods while scaling automatically during busy times.
Human approval workflow support
Step Functions supports human approval steps, making it suitable for high-value order workflows that require manual review or approval processes.
Implementation considerations
Error handling: Built-in retry mechanisms and error handling patterns help provide reliable order processing with configurable retry policies.
Visual monitoring: The Step Functions console provides real-time visibility into order processing status, enabling quick identification of bottlenecks.
Security model: Fine-grained AWS Identity and Access Management (IAM) roles per step so that payment processing functions have different permissions than inventory management functions.
Choosing the right workflow orchestration service
When selecting between Amazon MWAA and AWS Step Functions, consider these workflow characteristics:
Consider Amazon MWAA when your use case involves:
Complex stateful data processing where workflows modify data state and require recovery mechanisms to maintain consistency
Long-running batch jobs executing for hours or days where computational investment is substantial
Built-in scheduling requirements where regular batch processing needs time-based orchestration
Granular recovery needs where resuming from specific failure points is business-critical
Complex task dependencies involving sophisticated relationships between workflow tasks
Existing Apache Airflow expertise where teams have substantial investment in Apache Airflow knowledge
Consider AWS Step Functions when your use case involves:
Event-driven serverless workflows triggered by external events requiring immediate response
Stateless processing where each workflow execution operates independently
Short to medium duration tasks completing within minutes to hours
Heavy AWS service integration involving extensive coordination with Lambda functions and other AWS services
Human approval workflows requiring manual intervention or decision-making
Variable load patterns with unpredictable traffic requiring automatic scaling
Decision framework
To help guide your decision process, consider the following questions:
Figure 3: Decision tree guiding through key considerations for choosing between Amazon MWAA and AWS Step Functions based on workflow characteristics.
Figure 4: Comprehensive comparison between Amazon MWAA and AWS Step Functions, highlighting decision factors for choosing the right workflow orchestration service.
Conclusion
Both Amazon Managed Workflows for Apache Airflow and AWS Step Functions are workflow orchestration services, each designed to address specific use case requirements. By understanding your workflow characteristics and aligning them with the strengths of each service, you can make an informed decision that supports your business needs.
For complex, stateful workflows with long execution times and sophisticated recovery requirements, Amazon MWAA provides robust capabilities. For event-driven, serverless workflows with tight AWS integration and variable load patterns, AWS Step Functions is a strong fit.
Remember that these services are not mutually exclusive. Many organizations use both to address different workflow orchestration needs across their application portfolio. By focusing on your specific use case requirements, you can select the right tool for each job and build resilient, efficient workflow orchestration solutions on AWS.
If you have questions or feedback about choosing between these services, leave a comment.
This week, the AWS IoT Device SDK for Swift reached general availability. As a member of the Swift Server Workgroup (SSWG), this one caught my attention. The SDK brings production-ready MQTT 5 connectivity, Device Shadow, Jobs, and fleet provisioning to Swift developers on macOS, iOS, tvOS, and Linux.
I’m curious to see what you will build with it. Swift on the server has matured over the past few years, and now it reaches IoT devices too. This connects to a broader trend of running Swift at the edge. WendyOS, for example, is an open-source operating system for physical AI that offers first-class Swift support for deploying apps to NVIDIA Jetson and Raspberry Pi hardware. Between server-side Swift, IoT, and edge computing, the language is showing up in places that would have surprised most people a few years ago.
Now, let’s get into this week’s AWS news.
Headlines Amazon RDS for SQL Server supports Bring Your Own Media — Customers who migrate SQL Server applications from on-premises environments can now reuse their existing Microsoft SQL Server licenses, including Software Assurance, through Microsoft’s License Mobility program on Amazon RDS. BYOM is integrated with AWS License Manager for tracking license usage and compliance. Read more.
Amazon Cognito now supports multi-Region replication — You can now synchronize user and machine identity data, including credentials, user pool configurations, and federation setups, to a secondary user pool in a standby Region in near real-time. In the event of a disruption in the primary Region, signed-in users continue accessing their applications without re-authenticating, and registered users can sign in with their existing credentials. Multi-Region replication is available as an add-on for user pools in Essentials or Plus feature tiers across 16 Regions. Read more.
GPT-5.5, GPT-5.4, and Codex from OpenAI are now generally available on Amazon Bedrock — You can now use GPT-5.5 and GPT-5.4 in production workloads on Amazon Bedrock and build with Codex for AI-powered software development, with the same security, governance, and operational controls you already use across AWS. GPT-5.5 is the most capable model from OpenAI, excelling at agentic coding, data analysis, and multi-step autonomous tasks. Codex is available through the Codex App, the Codex CLI, and IDE integrations with Visual Studio Code, JetBrains, and Xcode. Pricing matches OpenAI first-party rates, and usage counts toward existing AWS commitments. Read more.
Last week’s launches Here are some launches and updates from this past week that caught my attention:
Amazon Bedrock adds CloudWatch metrics for OpenAI- and Anthropic-compatible APIs — You can now monitor inference traffic to the bedrock-mantle endpoint with CloudWatch metrics, including inference counts, input and output token totals, and client error counts at account, project, model, and project-and-model granularity.
AWS Step Functions adds AgentCore-powered agentic reasoning step — You can now add AI agent reasoning steps to your Step Functions workflows through an integration with the managed harness in Amazon Bedrock AgentCore. Run multiple agents in parallel or sequence, add human approval, and trace every agent decision.
Amazon EKS and Amazon EKS Distro now support Kubernetes version 1.36 — Kubernetes 1.36 promotes User Namespaces to GA, introduces Mutating Admission Policies, In-Place Pod-Level Resources Vertical Scaling, and Resource Health Status reporting. Available in all Regions where EKS is available.
Amazon Quick now supports VPC connectivity for MCP connections — Enterprise customers can now connect privately hosted Model Context Protocol (MCP) servers to Amazon Quick through VPC, enabling secure access to proprietary applications and internal tools without exposing them to the internet.
Managing identities and access across complex environments has become more critical than ever. AWS Directory Service for Managed Microsoft Active Directory, also known as AWS Managed Microsoft AD, has added new capabilities to manage users and groups. Now, you can perform create, read, update, and delete (CRUD) operations on users and groups directly through AWS Command Line Interface (AWS CLI), APIs, and the AWS Management Console. You can use this powerful capability to automate identity lifecycle management and enhance security in your AWS environment. By using these APIs, collectively known as the Directory Service Data APIs, you can perform operations such as:
Listing users and groups
Retrieving user and group details
Disabling and enabling user accounts
Resetting user passwords
Managing group memberships
These APIs provide new possibilities for automating identity management tasks and integrating Active Directory management into your existing workflows and applications.
The introduction of these APIs brings several key benefits:
Automation of the identity lifecycle: You can now programmatically manage user accounts throughout their lifecycle—from creation to deletion—enabling streamlined onboarding and offboarding processes.
Enhanced security: By integrating these APIs with security services like Amazon GuardDuty, you can create automated responses to potential security threats, such as disabling accounts with inappropriate access.
Improved compliance: You can use automated user management to help enforce consistent policies and help maintain compliance with various regulatory requirements.
Operational efficiency: You can automate routine tasks such as user provisioning, deprovisioning, and group management, reducing manual effort and the potential for human error.
Integration capabilities: By using these APIs, you can seamlessly integrate with existing identity management systems, custom applications, and third-party tools.
Cost optimization: By automating processes and reducing manual intervention, you can potentially help your organization optimize operational costs associated with identity management.
In this post, we explore these new APIs and demonstrate how you can use them to create an automated solution for detecting and responding to unexpected behavior by Active Directory users. We walk through a practical example that combines GuardDuty, AWS Step Functions, Amazon EventBridge, and the new AWS Directory Service APIs to create a robust security automation workflow.
Solution overview
To demonstrate the power of these new APIs, let’s explore a practical solution that automates the detection and response to unexpected behavior by Active Directory users. This solution combines several AWS services to create a robust security automation workflow:
GuardDuty continuously monitors for unexplained behavior of Active Directory users from AWS Managed Microsoft AD. For the example in this post, we’re using Backdoor:Runtime/C&CActivity.B!DNS
An EventBridge rule detects GuardDuty findings related to these users and triggers a Step Functions workflow.
Extract the Active Directory username from the instance using a run command.
Start an automation that will disable the account using the DisableUser API.
Figure 1: Diagram of the Step Functions workflow showing the process of Systems Manager finding the username and starting the automation to disable the account
{
"detail-type": ["AWS API Call via CloudTrail"],
"source": ["aws.ds"],
"detail": {
"eventSource": ["ds.amazonaws.com"],
"eventName": ["DisableUser"]
}
}
This solution delivers automated, near real-time remediation of potential security threats — significantly reducing exposure windows and containing the impact of unauthorized account access.
The following figure shows a high-level architecture diagram of the solution.
Figure 2: Diagram showing the workflow of what happens when potentially damaging activity is detected
Note: The solution must be deployed in the primary AWS Region of your directory.
Prerequisites
To complete the walkthrough in this post, you must have the following prerequisites in place.
GuardDuty
GuardDuty is an automated threat detection service that continuously monitors for unexpected activity and unauthorized behavior to protect your AWS accounts, workloads, and data stored in Amazon Simple Storage Service (Amazon S3).
To activate GuardDuty:
Go to the GuardDuty console.
If you’re activating GuardDuty for the first time, under Try threat detection with GuardDuty, select All Features and then choose Get Started.
If you’ve used GuardDuty before, select Runtime Monitoring and then choose Enable under Runtime Monitoring.
Figure 3: Runtime Monitoring enabled
AWS Managed Microsoft AD
AWS Managed Microsoft AD provides a fully managed service for Microsoft Active Directory (AD) in the AWS Cloud. When you create your directory, AWS deploys two domain controllers that are exclusively yours in separate Availability Zones for high availability. For use cases that require even higher resilience and performance in a specific AWS Region or during specific hours, you can scale AWS Managed Microsoft AD by deploying additional domain controllers to meet your needs. These domain controllers can help load balance, increase overall performance, or provide additional nodes to protect against temporary availability issues. Using AWS Managed Microsoft AD, you can define the correct number of domain controllers for your directory based on your use case.
To deploy a new AWS Managed Microsoft AD:
Go to the Directory Service console.
Choose Set up directory and select AWS Managed Microsoft AD.
Select Standard Edition and enter a directory DNS name and password.
Select a virtual private cloud (VPC). For this example, use the Default VPC.
Choose Create directory.
Create a test Active Directory user
You will use this test user account to sign in to an EC2 instance and initiate a command that simulates unexplained activity that results in this account being disabled.
To create the test user, you can use AWS CloudShell or the AWS CLI from your local machine. Run the following commands, replacing the --directory-id value with your own:
# Create the test user
aws ds-data create-user \
--directory-id "your-directory-id" \
--sam-account-name "TestUser" \
--given-name "Test" \
--surname "User"
Then
# Set a password for the test user
aws ds reset-user-password \
--directory-id "your-directory-id" \
--user-name "TestUser" \
--new-password "YourSecurePassword123!"
To generate alerts on GuardDuty, you need a domain joined Linux EC2 instance. If you don’t have a domain joined EC2 Linux instance, follow these instructions for joining a Linux instance to an Active Directory domain. This instance will be used to simulate suspicious activity that triggers a GuardDuty finding and initiates the automated remediation workflow.
Implement the solution
Let’s walk through the steps to implement this solution in your AWS environment.
For Create Stack, choose with new resources (standard).
For Template source, choose Upload a template file. Choose Choosefile and select the template you downloaded in step 1.
Choose Next.
For Stack name, enter a stack name (such as CRUD-API-MAD).
In the Parameters area, do the following:
For DirectoryID, enter the AWS Active Directory ID.
For NotificationEmail, enter the email address to send the notification to.
On the Configure stack options page, choose Next.
Select I acknowledge that AWS CloudFormation might create IAM resources with custom names, then choose Submit.
After the page is refreshed, the status of your stack should be CREATE_IN_PROGRESS. When the status changes to CREATE_COMPLETE, proceed to the next section.
Test
To simulate a threat, use a GuardDuty test domain that GuardDuty will recognize as a command and control server.
Go to the Amazon EC2 console.
Choose Instances from the navigation pane.
Select the test EC2 instance that you created earlier.
Choose Connect, select the Session Manager tab, and choose Connect.
Authenticate with your test user by entering su followed by the test user with the domain name that you created earlier. For example su [email protected], then enter the password.
Enter the command curl guarddutyc2activityb.com. You will receive an error because the page won’t resolve, but GuardDuty will have detected concerning events.
Go to the GuardDuty console and select Findings from the navigation pane.
Within 3–5 minutes, you should see a high severity finding for Backdoor:Runtime/C&CActivity.B!DNS.
This will then trigger the automation to disable the account.
Figure 4: Account successfully disabled
After the account is disabled, an email notification will be sent notifying an administrator that the account was disabled (it might take up to 5 minutes to receive the notification).
Figure 5: AWS notification message showing the username has been disabled
Note: You must archive the GuardDuty finding before running this test again, because the EventBridge rule only runs once against a GuardDuty finding with the same details. To archive the finding, select the check box next to the Backdoor:Runtime/C&CActivity.B!DNS finding, choose Actions (top right), and select Archive.
Conclusion
The new AWS Directory Service APIs for AWS Managed Microsoft AD provide powerful capabilities for programmatically managing Active Directory users and groups. By using these APIs in conjunction with services such as Amazon GuardDuty and AWS Step Functions, you can create sophisticated automation workflows that enhance your security posture and streamline identity management processes.
The solution we’ve explored in this post demonstrates just one of many possible use cases for these new APIs. As you integrate these capabilities into your own environments, you will probably discover numerous opportunities to improve efficiency, security, and compliance in your identity management practices.
We’re excited to see how you’ll use these new APIs to innovate and improve your identity management workflows. If you have any questions or want to share your own use cases, leave a comment below or reach out to AWS Support.
Remember, the cloud journey is all about continuous improvement and innovation. Keep exploring, keep learning, and keep pushing the boundaries of what’s possible with AWS.
Stay current with the latest serverless innovations that can improve your applications. In this 32nd quarterly recap, discover the most impactful AWS serverless launches, features, and resources from Q1 2026 that you might have missed.
In case you missed our last ICYMI, check out what happened in Q4 2025.
2026 Q1 calendar
Serverless with Mama J
Serverless with Mama J
If you really want to know whether you understand something, try explaining it to your mom!
That’s exactly what Eric Johnson did. His mom, everyone calls her Mama J, wanted to know what serverless actually means and why it matters. So he walked her through it: what servers do, why they’re a headache to manage, and how AWS Lambda lets you skip all that by running code only when it’s needed, scaling automatically, and charging you nothing when nobody’s using it.
AWS is providing a growing set of AI-powered tools to bring serverless expertise directly into your coding assistants. From Model Context Protocol (MCP) servers and Anthropic Claude plugins to Kiro Powers. These tools provide contextual guidance for architecture decisions, implementation patterns, and deployment automation across the full serverless development lifecycle.
For more information on the tools available, see the resources page.
Build fault-tolerant, long-running applications using familiar programming patterns using AWS Lambdadurable functions. You can use Lambda durable functions to write multi-step workflows in your preferred programming language, using built-in methods that automatically handle progress checkpointing and error recovery. This can improve your architecture so that you can focus on your business logic and optimize costs by charging only for active compute time.
Eric Johnson has a new video deep dive showing how to upload videos and scan them with AI. Learn how to coordinate multiple AWS services like Amazon Rekognition and Amazon Transcribe, implement human-in-the-loop approval workflows, and crate a live dashboard for real-time updates.
Lambda now supports the .NET 10 runtime, including support for file-based apps. Developers can take advantage of the latest .NET 10 performance improvements, new language features, and improved startup times for Lambda functions.
You can now see Availability Zone (AZ) metadata in function execution environments. This allows you to determine the AZ ID (e.g., use1-az1) of the AZ your function is running in. This helps build functions that can make AZ-aware routing decisions, such as preferring same-AZ endpoints for downstream services to reduce cross-AZ latency. Operators can also implement AZ-aware resilience patterns like AZ-specific fault injection testing.
Payload size increase
AWS has increased the maximum payload size from 256 KB to 1 MB for a number of services such as asynchronous Lambda invocations, Amazon SQS, and Amazon EventBridge. This gives you more room to build and maintain context-rich event-driven systems and reduce the need for complex workarounds such as data chunking or external large object storage.
This blog post explores a real-world example using rich event context in agentic event-driven architectures
Amazon Bedrock AgentCore is the infrastructure layer for securely deploying and operating AI agents. It works with popular open source frameworks, including Strands Agents, LangGraph and CrewAI, giving you the flexibility to build with your preferred tools without vendor lock-in.
AgentCore Gateway now includes semantic tool search, so you can discover the right tool for a task using natural language queries instead of manually browsing a catalogue. It also adds custom KMS encryption, debugging messages, and resource tagging to give you stronger governance over tool integrations.
Policy in Bedrock AgentCore allows you to define precise boundaries on agent actions and run continuous quality monitoring. This helps you maintain predictable, auditable agent behavior in production without embedding guardrail logic inside each individual agent.
AgentCore Runtime now supports stateful MCP server features, allowing agents to maintain session context across tool calls for richer, more coherent multi-step interactions.
AWS Step Functions introduces an enhanced TestState API that enables API-based testing for validating workflows before deployment. The new API supports testing individual states in isolation or complete workflows end-to-end, making it easier to verify state machine logic without incurring runtime costs.
By integrating TestState API testing into CI/CD pipelines, you can validate workflow logic before deployment, reducing the risk of production issues. Find complete code examples and testing framework in the GitHub repository.
Amazon EventBridge
Amazon EventBridge Scheduler now provides resource count metrics to help you monitor quota usage. These new metrics make it easier to track the number of schedules and schedule groups in your account and proactively manage service quotas.
Amazon DynamoDB
You can replicate Amazon DynamoDB table data across multiple AWS accounts and Regions. This enhances resiliency through account-level isolation, supports tailored security and data-perimeter controls. You can align workloads by business unit or environment and simplify governance requirements.
ECS also now supports Network Load Balancer (NLB) for linear and canary deployment strategies. This helps you perform gradual traffic shifting using NLBs, providing more flexibility in deployment pipelines for latency-sensitive applications.
Join our livestream every Tuesday at 11 AM PT for live discussions, Q&A sessions, and deep dives into serverless technologies. Watch episodes on-demand at serverlessland.com/office-hours.
The Serverless landing page has overall information about building serverless applications. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.
You can also follow the Developer Advocacy team to see the latest news, follow conversations, and interact with the team.
AWS Step Functions recently announced new enhancements to local testing capabilities for Step Functions, introducing API-based testing that developers can use to validate workflows before deploying to AWS. As detailed in our Announcement blog post, the TestState API transforms Step Functions development by enabling individual state testing in isolation or as complete workflows. This supports mocked responses and actual AWS service integrations, and provides advanced capabilities. These capabilities include Map/Parallel states, error simulation with retry mechanisms, context object validation, and detailed inspection metadata for comprehensive local testing of your serverless application.
The TestState API can be accessed through multiple interfaces such as AWS Command Line Interface (AWS CLI), AWS SDK, LocalStack. By default, TestState API in AWS CLI and SDK runs against the remote AWS endpoint, providing validation against the actual Step Functions service infrastructure. We’ve partnered with LocalStack to offer an additional testing endpoint for the TestState API. Developers can use LocalStack for unit testing their workflows by changing the AWS SDK client endpoint configuration to point to LocalStack: http://localhost.localstack.cloud:4566/ instead of AWS endpoint. This approach provides complete network isolation when needed. For a streamlined development experience, you can also use the LocalStack VSCode extension to automatically configure your environment to point to the LocalStack endpoint. This approach is detailed in the AWS blog post.
This blog post demonstrates building test suites to unit test your Step Functions workflows using the AWS SDK for Python using the pytest framework. The complete implementation is available in the GitHub repository.
Building test cases using the TestState API
This example workflow implements a real-world ecommerce order processing system using JSONata for advanced data transformations. It incorporates complex Step Functions patterns including distributed Map states, Parallel execution, and waitForTaskToken callback mechanisms. The process validates orders through AWS Lambda functions, distributes order item processing with configurable failure tolerance, runs parallel payment and inventory updates, handles human approval workflows using task tokens, then persists orders in Amazon DynamoDB with notification delivery. This workflow demonstrates advanced error handling with multiple Catchers and Retriers, exponential backoff for Lambda throttling and DynamoDB limits, and sophisticated state transitions that were previously challenging to test locally. This makes it the recommended choice for demonstrating the use of enhanced TestState API’s local testing features.
The complete workflow is available in the GitHub repository, where you can examine the full state machine definition and see how JSONata expressions handle data transformation throughout the execution flow.
Figure 1: State machine workflow that demonstrates a real-world ecommerce order processing system.
Effective Step Functions testing requires a systematic approach to TestState API integration that provides state validation, error simulation, and assertion capabilities. The testing framework is built using Python’s pytest framework, using fixtures to automatically provide pre-configured runner instances that handle TestState API client initialization and state machine definition loading. This eliminates repetitive setup code and provides consistent test environments. The enhanced TestState API supports both mock integrations and actual integrations with AWS services, providing flexibility in testing strategies. For this demonstration, you use mock integrations to showcase how a complete local testing can be achieved without having any resources deployed to AWS accounts.
This framework is built for demonstration purposes, and you can similarly build your own testing frameworks using other programming languages like Java, Node.js. The testing framework uses method chaining patterns to create readable test cases with comprehensive assertion methods, automatic output chaining between state executions, and error simulation for testing retry mechanisms, backoff intervals, and catch blocks across AWS service error conditions.
The following test implementations demonstrate the testing capabilities that are achievable with the enhanced TestState API in local development environments. The test cases are run against the preceding Statemachine.
Test Case 1: Lambda throttling and retry mechanism testing
Service integrations with Statemachines like AWS Lambda, Amazon DynamoDB may face throttling depending on their usage. A key capability of the enhanced TestState API is its ability to simulate retry mechanisms with control over retry counts and backoff intervals. This test demonstrates the enhanced TestState API’s retry testing capabilities through the stateConfiguration.retrierRetryCountparameter and inspectionData.errorDetailsresponse fields. This response field provides retryBackoffIntervalSeconds for validating exponential backoff calculations, retryIndex for tracking retry attempt sequences, and catchIndex for identifying which error handler processed the exception. These enhanced inspection capabilities enable validation of retry logic, backoff strategies, and error propagation patterns across complex state machine workflows.
Test Case 2: Map state testing with tolerance thresholds
Distributed Map states present unique testing challenges due to their parallel processing nature and failure tolerance capabilities. The enhanced TestState API provides specialized configuration options for testing these complex scenarios.
This test demonstrates the enhanced TestState API’s Map state testing capabilities through the stateConfiguration.mapIterationFailureCountparameter for simulating iteration failures. The API provides comprehensive inspection data including inspectionData.afterItemSelector for validating ItemSelector transformations, inspectionData.afterItemBatcher for batch processing validation, inspectionData.toleratedFailureCount and inspectionData.toleratedFailurePercentage for threshold verification. When the specified failure count exceeds the configured tolerance, the API correctly returns States.ExceedToleratedFailureThreshold, enabling testing of Map state resilience patterns.
Test Case 3: WaitForCallback pattern testing
The waitForCallback integration requires context object construction to simulate realistic execution environments, particularly for human approval workflows.
This test demonstrates the enhanced TestState API’s support for waitForCallback integrations through the `context` parameter for realistic Context object simulation. The API enables comprehensive testing of JSONata expressions that reference $states.context.Task.Token, $states.context.Execution.Id, and other context fields. The inspectionData.afterArgumentsresponse field validates that JSONata expressions correctly processed the context data, while the API automatically handles the complexity of task token embedding in service integration payloads for waitForCallback testing scenarios.
Test Case 4: Happy path testing – complete workflow validation
Happy path testing validates that workflows execute correctly under normal operating conditions. The enhanced TestState API allows you to chain state executions together, automatically passing outputs between states to simulate a complete workflow execution.
This test demonstrates how the TestState API maintains state context between executions, enabling realistic workflow simulation. The get_output() method retrieves the processed output from one state to use as input for the next, mimicking actual Step Functions execution behavior.
Note: The code snippet above shows only the first two states of the complete workflow test for brevity. The full test code with all states (ProcessOrderItems, ParallelProcessing, WaitForApproval, CheckApproval, SaveOrderDetails, and SendNotification) can be viewed in the complete GitHub repository, demonstrating end-to-end workflow validation using the same method chaining pattern.
Integration with modern CI/CD pipelines
In this section, we will explore how to integrate the previous unit tests in a CI CD pipeline to enable local testing.
The sample repository includes a GitHub Actions workflow that demonstrates how TestState API testing integrates into continuous integration and continuous delivery (CI/CD) pipelines. The workflow (.github/workflows/test-and-deploy.yml) provides a two-step process that validates before any AWS resources are deployed using AWS Serverless Application Model (AWS SAM).
The CI/CD pipeline follows the following pattern:
Unit Tests: Executes the complete TestState API test suite using pytest tests/unit_test.py -v
To enable the GitHub Actions workflow to deploy resources to your AWS account, configure these AWS credentials in your GitHub repository settings. For detailed setup instructions, see the AWS blog post.
Following are the required secrets to be configured in GitHub repository settings:
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION
In production environments, you can typically extend this basic pipeline to include additional stages. The enhanced pipeline often begins with deploying to a development account first, followed by integration testing against deployed resources. The final stage involves moving to production with proper approval gates and security scanning compliance checks.
Conclusion
The enhanced TestState API enables testing Step Functions workflows locally without requiring AWS deployments that accelerated development cycles, and reduce testing times. This post demonstrates how to implement testing for state types including Map states with tolerance thresholds, retry mechanisms with exponential backoff, and waitForTaskToken patterns with context object simulation using mock integrations for isolated testing.
By integrating TestState API testing into CI/CD pipelines, you can validate workflow logic before deployment, reducing the risk of production issues. The GitHub Actions workflow example demonstrates an implementation that runs tests and deploys resources in a controlled sequence. The complete code examples and testing framework are available in the GitHub repository to implement similar testing practices for Step Functions workflows.
Stay current with the latest serverless innovations that can transform your applications. In this 31st quarterly recap, discover the most impactful AWS serverless launches, features, and resources from Q4 2025 that you might have missed.
In case you missed our last ICYMI, check out what happened in Q3 2025.
2025 Q4 calendar
Serverless at re:Invent 2025
This post covers the biggest serverless announcements from re:Invent 2025, highlighting key feature updates that can improve your applications, and shares valuable resources to keep you informed.
AWS re:Invent 2025 had more than 60,000 in-person attendees and more than 2 million online viewers for the keynotes. The event featured 3,500 sessions from 3,000 speakers, which included information on 530 AWS service and feature announcements.
Keynote Igniting the serverless movement
The serverless content consisted of two tracks: Containers and Serverless (CNS) and Application Integration (API). These tracks included 150 unique sessions watched in-person by more than 16,000 attendees. There were developer-focused experiences including a Road to re:Invent Hackathon, AWS Builder Loft, and Builders Arena. Serverlesspresso, the coffee shop powered by serverless technology, operated in two locations during the event: the Expo Hall and the certification lounge.
Managing state across multi-step serverless workflows has traditionally required complex external orchestration tools. AWS Lambdadurable functions expand how developers can use Lambda. You can now build reliable multi-step applications and AI workflows directly within Lambda.
AWS Lambda durable functions code
Durable functions automatically checkpoint progress by saving the current state and completed steps at key points during execution. This allows them to suspend execution for up to one year during long-running tasks and recover from failures by resuming from the last checkpoint rather than restarting from the beginning, all without requiring additional infrastructure management.
Developers can now build in Python or TypeScript, wrap calls in steps with automatic retries and checkpointing. You can use waits to suspend execution for minutes, hours, or even up to a year without paying for idle compute. Durable functions use a replay mechanism to maintain state and handle failures gracefully. The replay mechanism works by re-executing your function code from checkpoints when recovering from failures, ensuring state consistency without data loss. This also means you don’t need complex external orchestration tools for many use cases. This can be helpful for AI workflows and multi-step applications where you need reliable state management without managing external infrastructure.
Lambda now offers Lambda Managed Instances, a new compute option that combines Amazon EC2 flexibility with fully managed infrastructure. AWS automatically handles instance provisioning, scaling, and maintenance while allowing access to the full range of EC2 capabilities, including Graviton4, network-optimized instances, and other specialized compute options.
AWS Lambda Managed Instances configuration
Your functions run on dedicated EC2 capacity from your account, in your own Amazon Virtual Private Cloud (Amazon VPC). AWS still manages the operational overhead, including OS patching, load balancing, and auto-scaling. This gives you access to specialized hardware options while maintaining the serverless operational model. You can further improve costs by using EC2 pricing models, including Compute Savings Plans and Reserved Instances for Lambda workloads. Each instance can handle multiple concurrent requests, making this particularly valuable for high-volume, steady-state workloads where predictable pricing and specific hardware requirements matter.
Multi-tenant SaaS applications face challenges like data leakage between tenants and noisy neighbor effects where one tenant’s workload impacts others. They also struggle with implementing custom isolation mechanisms. Tenant isolation mode addresses these by processing function invocations in separate execution environments for each tenant. This manages tenant-level compute environment isolation automatically.
AWS Lambda tenant isolation
Lambda adds Provisioned Mode for Amazon SQS event-source mappings, providing predictable performance and reduced cold starts for high-throughput SQS processing workloads.
Lambda functions now support IPv6 networking, so you don’t need NAT Gateways when accessing the internet or other AWS services from VPC-connected functions.
Lambda internet connectivity through a NAT Gateway (IPv4) and Lambda internet connectivity through an egress-only internet gateway (IPv6).
Lambda Rust support is now generally available, moving from experimental status. This is backed by AWS Support and the Lambda availability SLA.
Lambda has expanded its runtime support by adding Python 3.14, Node.js 24, and Java 25 as both managed runtimes and container base images, providing access to the latest language features and ensuring long-term support.
Amazon ECS
Amazon Elastic Container Service (Amazon ECS) Express Mode streamlines the deployment and management of containerized applications by automating the infrastructure setup that traditionally slows down developers.
Amazon ECS Express Mode deployment
This means you can focus on building applications while deploying with confidence using AWS best practices. Express Mode lets you deploy production-ready containerized web applications and APIs with a single command. This automatically handles domains, networking, load balancing, AWS Identity and Access Management (IAM) roles, and auto-scaling through simplified APIs. When your applications evolve and require advanced features, you can seamlessly configure and access the full capabilities of the resources, including Amazon ECS. Learn more from the launch blog post.
Amazon ECS announced a public preview of a fully managed MCP server, enabling AI-powered experiences for development and operations. The Model Context Protocol (MCP) server provides enterprise-grade capabilities like automatic updates and patching, centralized security through AWS IAM integration, comprehensive audit logging via AWS CloudTrail, and the proven scalability, reliability, and support of AWS.
Amazon Elastic Container Registry (ECR)managed container image signing enhances your security posture and eliminates the operational overhead of setting up signing. Container image signing allows you to verify that images are from trusted sources. ECR automatically signs images as they are pushed using the identity of the entity pushing the image. Signing operations are logged through CloudTrail for full auditability.
Amazon API Gateway
Amazon API Gateway allows you to improve the responsiveness of your REST APIs by progressively streaming response payloads back to the client. With this new capability, you can use streamed responses to enhance user experience when building LLM-driven applications (such as AI agents and chatbots), improve time-to-first-byte (TTFB) performance for web and mobile applications, stream large files, and perform long-running operations while reporting incremental progress using protocols such as server-sent events (SSE).
Amazon API Gateway streaming
API Gateway introduces private integration with Application Load Balancers (ALBs). You can use this to expose your VPC-based applications securely through REST APIs without exposing your ALBs to the public internet.
You can also now configure enhanced TLS security policies on API endpoints and custom domain names, providing you with greater control over the security posture of your APIs.
Amazon EventBridge
Amazon EventBridge introduced an enhanced visual rule builder that helps developers discover and subscribe to events from custom applications and over 200 AWS services. The console-based interface integrates the EventBridge schema registry with a comprehensive event catalog and intuitive drag-and-drop canvas that simplifies building event-driven applications. Developers can browse and search through events with readily available sample payloads and schemas without having to hunt through individual service documentation. The schema-aware visual builder guides developers through creating event filter patterns and rules, reducing syntax errors and accelerating development time.
AWS Step Functions allows for enhanced local testing through the TestState API, providing programmatic access to comprehensive testing capabilities without deploying to AWS. This helps you build automated test suites that validate your workflow definitions locally on your development machines. Test error handling patterns, data transformations, and mock service integrations using your preferred testing frameworks.
There is also a new metrics dashboard, giving you visibility into your workflow operations at both the account and state machine levels.
Other announcements
Savings Plans flexible pricing model extends to AWS managed database services with the launch of Database Savings Plans. This helps reduce database costs by up to 35% when committing to a consistent amount of usage ($/hour) over a 1-year term. Savings automatically apply each hour to eligible usage across supported database services, and additional usage beyond the commitment is billed at on-demand rates.
Amazon DynamoDB now supports multi-attribute composite keys in global secondary indexes. You no longer need to concatenate values into synthetic keys manually, which sometimes results in the need to backfill data before adding new indexes. Instead, you can create primary keys using up to eight existing attributes, making it easier to model diverse access patterns and adapt to new query requirements.
The Strands Agents SDK is an open source framework that takes a model-driven approach to building and running AI agents in just a few lines of code. TypeScript support is now available in preview so you can choose between Python and TypeScript for building Strands Agents.
Amazon S3 Vectors became generally available. S3 Vectors delivers purpose-built, cost-optimized vector storage for AI agents, inference, Retrieval Augmented Generation (RAG), and semantic search at billion-vector scale.
Join our livestream every Tuesday at 11 AM PT for live discussions, Q&A sessions, and deep dives into serverless technologies. Episodes are available on-demand at serverlessland.com/office-hours.
The Serverless landing page has overall information about building serverless applications. The Lambda resources page contains case studies, webinars, whitepapers, customer stories, reference architectures, and even more Getting Started tutorials.
You can also follow the Serverless Developer Advocacy team to see the latest news, follow conversations, and interact with the team.
Organizations often have large volumes of documents containing valuable information that remains locked away and unsearchable. This solution addresses the need for a scalable, automated text extraction and knowledge base pipeline that transforms static document collections into intelligent, searchable repositories for generative AI applications.
Organizations can automate the extraction of both content and structured metadata to build comprehensive knowledge bases that power retrieval-augmented generation (RAG) solutions while significantly reducing manual processing costs and time-to-value. The architecture not only demonstrates the processing of 500 research papers automatically, but also scales to handle enterprise document volumes cost-effectively through the Amazon Bedrock batch inference pricing model.
Overview
Amazon Bedrock batch inference is a feature of Amazon Bedrock that offers a 50% discount on inference requests. Although Amazon Bedrock schedules and runs the batch job (needing a minimum of 100 inference requests) as capacity becomes available, the inference won’t be real-time. For use cases where you can accommodate minutes to hours of latency, Amazon Bedrock batch inference is a good option.
This post demonstrates how to build an automated, serverless pipeline using AWS Step Functions, Amazon Textract, Amazon Bedrock batch inference, and Amazon Bedrock Knowledge Bases to extract text, create metadata, and load it into a knowledge base at scale. The example solution processes 500 research papers in PDF format from Amazon Science, extracts text using Amazon Textract, generated structured metadata with Amazon Bedrock batch inference and the Amazon Nova Pro model, and loads the final output, including Amazon Bedrock Knowledge Base filter, into an Amazon Bedrock Knowledge Base.
Architecture
This solution uses Step Functions with parallel Amazon Textract job processing through child workflows run by Distributed Map. You can use the concurrency controls offered by Distributed Map to process documents as quickly as possible within your Amazon Textract quotas. Increasing processing speed necessitates adjusting your Amazon Textract quota and updating the Distributed Map configuration. Amazon Bedrock batch inference handles concurrency, scaling, and throttling. This means that you can create the job without managing these complexities.
In this example implementation, the solution processes research papers to extract metadata such as:
Code availability and repository locations
Dataset availability and access methods
Research methodology types
Reproducibility indicators
Other relevant research attributes
The high-level parts of this solution include:
Extracting text from PDF documents with Amazon Textract in parallel, through Step Functions Distributed Map.
Analyzing extracted text using Amazon Bedrock batch inference to extract structured metadata.
Loading extract text and metadata into a searchable knowledge base using Amazon Bedrock Knowledge Bases with Amazon OpenSearch Serverless.
Figure 1. Complete architecture diagram
Prerequisites
The following prerequisites are necessary to complete this solution:
After deploying the cdk stacks, upload your data sources (PDF files) into the AWS CDK-created Amazon S3 input bucket. In this example, I uploaded 500 Amazon Science papers. The input bucket name is included in the AWS CDK outputs:
The process begins when you upload a manifest.json file to the input bucket. The manifest file lists the files for processing, which already exist in the input bucket. The filenames listed in manifest.json define what constitutes a single processing job run. To create another run, you would create a different manifest.json and upload it to the same S3 bucket.
The AWS CDK definition for the input bucket includes Amazon EventBridge notifications and creates a rule that triggers the Step Functions workflow whenever a manifest.json file is uploaded.
private createS3Buckets() {
const batchBucket = new s3.Bucket(this, "BatchInputBucket", {
removalPolicy: cdk.RemovalPolicy.DESTROY,
autoDeleteObjects: true,
})
batchBucket.enableEventBridgeNotification()
new cdk.CfnOutput(this, "BatchInputBucketName", {
value: batchBucket.bucketName,
description: "Name of input bucket to send PDF documents that Textract will read.",
})
const manifestFileCreatedRule = new eventBridge.Rule(this, "ManifestFileCreatedRule", {
eventPattern: {
source: ["aws.s3"],
detailType: ["Object Created"],
detail: {
bucket: {
name: [batchBucket.bucketName],
},
object: {
key: ["manifest.json"],
},
},
},
})
return { batchBucket, manifestFileCreatedRule }
}
The first step in the Step Functions workflow is a Distributed Map run that performs the following actions for each PDF in the manifest file:
Writes the Step Functions task token to Amazon DynamoDB, pausing the individual child workflow.
Processes the Amazon SNS message when the Amazon Textract job completes, triggering an AWS Lambda function.
Uses a Lambda function to retrieve the task token from DynamoDB using the Amazon Textract JobId.
Fetches the raw results from Amazon Textract, organizes the text for readability, and writes results to an S3 bucket
A key component of this architecture is the callback pattern that Amazon Textract supports using the NotificationChannel option, as shown in the preceding figure. The AWS CDK definition the Step Functions state that starts the Amazon Textract job is shown in the following.
The Lambda function that handles task tokens extracts the Amazon Textract JobId from the Amazon SNS message, fetches the TaskToken from DynamoDB, and resumes the Step Functions Workflow by sending the TaskToken:
from aws_lambda_powertools.utilities.data_classes import SNSEvent, event_source
@event_source(data_class=SNSEvent)
def handle_textract_task_complete(event, context):
# Multiple records can be delivered in a single event
for record in event.records:
sns_message = json.loads(record.sns.message)
textract_job_id = sns_message["JobId"]
# Get both task token and original file from DynamoDB
ddb_item = _get_item_from_ddb(textract_job_id)
# Send both the job ID and original file name in the response
_send_task_success(
ddb_item["TaskToken"],
{
"TextractJobId": textract_job_id,
"OriginalFile": ddb_item["OriginalFile"],
},
)
# Delete the task token from DynamoDB after use
_delete_item_from_ddb(textract_job_id)
def _send_task_success(task_token: str, output: None | dict = None) -> None:
"""Sends task success to Step Functions with the provided output"""
sfn = boto3.client("stepfunctions")
sfn.send_task_success(taskToken=task_token, output=json.dumps(output or {}))
The Distributed Map runs up to 10 child workflows concurrently, controlled by the maxConcurrency setting. Although Step Functions supports running up to 10,000 child workflow executions, the practical concurrency for this solution is constrained by Amazon Textract quotas. The startDocumentAnalysis API has a default quota of 10 requests per second (RPS), which means you must consider this limit when scaling your document processing workloads and potentially request quota increases for higher throughput requirements.
When all of the Amazon Textract jobs finish, the Distributed Map state creates an Amazon Bedrock batch inference input file, launches the Amazon Bedrock inference job, and waits for it to complete.
A Lambda function collects text results from Amazon S3 and creates an Amazon Bedrock batch inference input file with custom prompts.
The workflow starts the Amazon Bedrock batch inference job by calling createModelInvocationJob and sending the batch inference input file as input.
The workflow pauses and stores the task token in DynamoDB.
An EventBridge rule matches completed Amazon Bedrock batch inference events, and upon job completion and triggers a Lambda function. The Lambda function retrieves the task token and resumes the workflow, as shown in the following figure.
A batch inference input is a single jsonl file with multiple entries such as the following example. The prompt in each inference request instructs the large language model (LLM) to analyze the paper and extract metadata. Read the full prompt template in the GitHub repository.
{
"recordId": "c1b8a3b2086141f963",
"modelInput": {
"messages": [
{
"role": "user",
"content": [
{
"text": "Analyze the following research paper transcript and extract metadata about code and dataset availability. Extract the following metadata from this research paper transcript:\n\n1. **has_code**: Does the paper mention or link to source code? (true/false) ...... Return only valid JSON matching the schema above. Do not include any text outside of the JSON structure."
}
]
}
],
"inferenceConfig": { "maxTokens": 4096 }
}
}
Populating the Amazon Bedrock Knowledge Base
After the batch inference completes, the workflow does the following:
Extracts inference results and creates metadata files based on the Amazon Bedrock inference results (example metadata shown in the following figure).
Starts an Amazon Bedrock Knowledge Base ingestion job.
Monitors the ingestion job status using Step Functions Wait and Choice states.
Sends a completion notification through Amazon SNS.
After the workflow completes successfully, you can test the knowledge base to verify that the documents and metadata have been properly ingested and are searchable. There are two practical methods for testing an Amazon Bedrock Knowledge Base:
Using the Console
Using the AWS SDK to run a query
Testing through the Console
The Console provides an intuitive interface for testing your knowledge base queries with metadata filters:
In the left navigation pane, choose Knowledge Bases under the Build section.
Choose the knowledge base created by the AWS CDK deployment (the name will be output by the AWS CDK stack).
Choose the Test button in the upper right corner.
In the test interface, choose your preferred foundation model (FM) (such as Amazon Nova Pro).
Expand the Configurations column, then navigate to the Filters section.
Configure filters based on the extracted metadata, as shown in the following figure.
Enter a natural language query related to your documents, for example: “Recent research on retrieval augmented generation?”
The console displays the generated response along with source attributions showing which documents were retrieved and used to formulate the answer, filtered by your specified metadata attributes, as shown in the following figure.
Testing via API
For programmatic testing and integration into applications, use the AWS SDK with metadata filtering. The following is a Python example using boto3:
model_arn = "arn:aws:bedrock:us-east-1::foundation-model/amazon.nova-pro-v1:0"
# Query for papers with publicly available code
response = bedrock_agent_runtime.retrieve_and_generate(
input={'text': "What recent research has been done on RAG?"},
retrieveAndGenerateConfiguration={
'type': 'KNOWLEDGE_BASE',
'knowledgeBaseConfiguration': {
'knowledgeBaseId': knowledge_base_id,
'modelArn': model_arn,
'retrievalConfiguration': {
'vectorSearchConfiguration': {
'numberOfResults': 5,
'filter': {"equals": {"key": "has_code", "value": True}},
}
},
},
},
)
# Display results
print(f"Response: {response['output']['text']}\n")
print("Source Documents:")
for citation in response.get('citations', []):
for reference in citation.get('retrievedReferences', []):
metadata = reference.get('metadata', {})
print(f" Document: {reference['location']['s3Location']['uri']}\n")
The following is the test script output:
Response: Recent research on Retrieval-Augmented Generation (RAG) has focused on enhancing the system's ability to dynamically retrieve and utilize relevant information from a Vector Database (VDB) to improve decision-making and performance. Key innovations include:
1. **Dynamic Retrieval and Utilization**: The system is designed to query the VDB for contextually relevant past experiences, which significantly improves decision quality and accelerates performance by leveraging a growing repository of relevant experiences.
2. **Teacher-Student Instructional Tuning**: A novel mechanism where a Teacher agent refines a Student agent's core policy through direct interaction. The Teacher generates a modified SYSTEM prompt based on the Student's actions, creating a meta-learning loop that enhances the Student's reasoning policy over time.
Conclusion
This solution demonstrates how to combine multiple AWS AI and serverless services to build a scalable document processing pipeline. Organizations can use AWS Step Functions for orchestration, Amazon Textract for document processing, Amazon Bedrock batch inference for intelligent content analysis, and Amazon Bedrock Knowledge Bases for searchable storage. In turn, they can automate the extraction of insights from large document collections while optimizing costs.
Following this solution, you can build a solid foundation for production-scale document processing pipelines that maintain the flexibility to adapt to your specific requirements while making sure of reliability, scalability, and operational excellence. Follow this link to learn more about serverless architectures.
Next week, don’t miss AWS re:Invent, Dec. 1-5, 2025, for the latest AWS news, expert insights, and global cloud community connections! Our News Blog team is finalizing posts to introduce the most exciting launches from our service teams. If you’re joining us in person in Las Vegas, review the agenda, session catalog, and attendee guides before arriving. Can’t attend in person? Watch our Keynotes and Innovation Talks via livestream.
AWS CloudFormation StackSets offers deployment ordering for auto-deployment mode. You can define the sequence in which your stack instances automatically deploy across accounts and Regions.
AWS NAT Gateway supports Regional availability to create a single NAT Gateway that automatically expands and contracts across availability zones (AZs).
Today, I’m excited to announce enhanced local testing capabilities for AWS Step Functions through the TestState API, our testing API.
These enhancements are available through the API, so you can build automated test suites that validate your workflow definitions locally on your development machines, test error handling patterns, data transformations, and mock service integrations using your preferred testing frameworks. This launch introduces an API-based approach for local unit testing, providing programmatic access to comprehensive testing capabilities without deploying to Amazon Web Services (AWS).
There are three key capabilities introduced in this enhanced TestState API:
Mocking support – Mock state outputs and errors without invoking downstream services, enabling true unit testing of state machine logic. TestState validates mocked responses against AWS API models with three validation modes: STRICT (this is the default and validates all required fields), PRESENT (validates field types and names), and NONE (no validation), providing high-fidelity testing.
Support for all state types – All state types, including advanced states such as Map states (inline and distributed), Parallel states, activity-based Task states, .sync service integration patterns, and .waitForTaskToken service integration patterns, can now be tested. This means you can use TestState API across your entire workflow definition and write unit tests to verify control flow logic, including state transitions, error handling, and data transformations.
Testing individual states – Test specific states within a full state machine definition using the new stateName parameter. You can provide the complete state machine definition one time and test each state individually by name. You can control execution context to test specific retry attempts, Map iteration positions, and error scenarios.
Getting started with enhanced TestState Let me walk you through these new capabilities in enhanced TestState.
Scenario 1: Mock successful results
The first capability is mocking support, which you can use to test your workflow logic without invoking actual AWS services or even external HTTP requests. You can either mock service responses for fast unit testing or test with actual AWS services for integration testing. When using mocked responses, you don’t need AWS Identity and Access Management (IAM) permissions.
Here’s how to mock a successful AWS Lambda function response:
This command tests a Lambda invocation state without actually calling the function. TestState validates your mock response against the Lambda service API model so your test data matches what the real service would return.
The response shows the successful execution with detailed inspection data (when using DEBUG inspection level):
When you specify a mock response, TestState validates it against the AWS service’s API model so your mocked data conforms to the expected schema, maintaining high-fidelity testing without requiring actual AWS service calls.
Scenario 2: Mock error conditions You can also mock error conditions to test your error handling logic:
This simulates a Lambda service exception so you can verify how your state machine handles failures without triggering actual errors in your AWS environment.
The response shows the failed execution with error details:
The mock result represents the complete output from processing multiple items. In this case, the mocked array must match the expected Map state output format.
The response shows successful processing of the array input:
The mock result must be an array with one element per branch. By using TestState, your mock data structure matches what a real Parallel state execution would produce.
The response shows the parallel execution results:
Scenario 5: Test individual states within complete workflows You can test specific states within a full state machine definition using the stateName parameter. Here’s an example testing a single state, though you would typically provide your complete workflow definition and specify which state to test:
This tests a Lambda invocation state with specific input data, showing how TestState processes the input and transforms it through the state execution.
The response shows detailed input processing and validation:
These enhancements bring the familiar local development experience to Step Functions workflows, helping me to get instant feedback on changes before deploying to my AWS account. I can write automated test suites to validate all Step Functions features with the same reliability as cloud execution, providing confidence that my workflows will work as expected when deployed.
Things to know Here are key points to note:
Availability – Enhanced TestState capabilities are available in all AWS Regions where Step Functions is supported.
Pricing – TestState API calls are included with AWS Step Functions at no additional charge.
Framework compatibility – TestState works with any testing framework that can make HTTP requests, including Jest, pytest, JUnit, and others. You can write test suites that validate your workflows automatically in your continuous integration and continuous delivery (CI/CD) pipeline before deployment.
Feature support – Enhanced TestState supports all Step Functions features including Distributed Map, Parallel states, error handling, and JSONata expressions.
Documentation – For detailed options for different configurations, refer to the TestState documentation and API reference for the updated request and response model.
Get started today with enhanced local testing by integrating TestState into your development workflow.
In this post, we explore using AWS Step Function state machine with asynchronous AWS services, look at some scenarios where the processing time can be unpredictable, explain when traditional solutions such as polling (periodically check) fall short, and demonstrate how to implement a generalized callback pattern to handle asynchronous operations into a more manageable synchronous flow. We cover the related architecture, technical implementation, and best practices, and we provide a real-world examples that uses the AWS Cloud Development Kit (AWS CDK). Services used in this generalized callback pattern include Amazon DynamoDB, Amazon EventBridge and AWS Step Functions.
Understanding the issue this solution addresses
Asynchronous operations are designed to handle long-running operations without blocking resources, a design followed by many AWS services. However, these services create challenges in Step Functions workflows by returning immediate 200 OK responses rather than confirming task completion. This breaks the Step Functions execution model, which expects each step to be complete before advancing. Developers often attempt to address this issue through polling loops to repeatedly check the status of operations, an approach that works for containerized applications and Amazon Elastic Compute Cloud (Amazon EC2). For these services, compute resources are already provisioned, but compute resources become problematic in serverless architectures when AWS Lambda functions have a 15-minute execution limit, making them unsuitable for long-running polls.
Step Functions supports Run a Job (.sync) to call a service and have Step Functions wait for a job to complete, but this works only for selected optimized integrations. However, this functionality is limited to specific AWS services such as AWS Glue. Amazon Translate, Macie, and other services are not optimized integrations. If your operation is not listed as working with .sync, it can benefit from the generalized callback pattern covered in this post.
For these non-optimized integrations, an option is to use polling (periodically check). However, polling can lead to additional latency in response because polling times are unlikely to align with job completion. This is shown in the following figure.
Figure 1: A job processing and delay timeline diagram
The Step Functions generalized callback pattern can solve this latency issue by pausing execution for up to one year while waiting for task completion (this does not incur additional cost). When such an asynchronous operation finishes, a callback mechanism resumes the workflow where it left off. This generalized callback pattern transforms asynchronous operations into synchronous ones, and it maintains cost efficiency and operational agility.
Scenarios
To help us see where this generalized callback pattern could be applied, let’s look at a few scenarios. Each of these scenarios makes use of AWS Step Functions state machines to run the applications’ workflows.
Scenario 1: Document translation with personally identifiable information compliance
Organizations must manage personally identifiable information (PII) when translating documents because PII can be duplicated across language outputs. For example, when translating a document containing “Jane Doe,” that name appears in both the original and translated versions, creating multiple instances of sensitive data that need compliance measures. Amazon Translate batch translation has a default concurrency of 10, meaning that translations could take more than 10 minutes or be queued for longer periods. Additionally, the Amazon Translate batch translation operation is asynchronous, holding the translation request in a queue until completed. The generalized callback pattern in this post makes sure that Step Functions state machine workflows resume appropriately to apply consistent PII handling across all outputs. In this scenario the design makes use of tagging Amazon Simple Storage Service (Amazon S3) files as containing PII or not, which in turn associates S3 lifecycle policies for specific retention periods to those S3 objects.
Figure 2: A text translation workflow diagram
Scenario 2: Using concurrent execution to pause the state machine until processes have completed
Continuing from scenario 1, Macie and Amazon Translate can run in parallel (each approximately 10 minutes) rather than sequentially (approximately 20 minutes) for a better user experience. Similarly to Amazon Translate batch translation operations being asynchronous, the Macie create classification operation is also asynchronous. Step Functions state machines enable concurrent execution of both service requests. The generalized callback pattern enables the state machine to pause each parallel workflow and resume only when the asynchronous services have completed their jobs. Without this pattern, both services would immediately return 200 OK responses, causing the workflow to continue prematurely before translations or classification results are available. If the classification results are not available later in the workflow, then the appropriate PII tags will not be applied and therefore the appropriate lifecycle retention policy will also not be applied, resulting in not adhering to PII handling practices.
Figure 3: A parallel classification and translation workflow diagram
Scenario 3: Intelligent document processing
Organizations that use Bedrock Data Automation for intelligent document processing must take into consideration Regional concurrency limits. BDA has Regional concurrency limits “Max number of concurrent jobs” of 25 jobs in the us-east-1 and us-west-2 Regions. Also, BDA has a concurrency limit of only five jobs in other supported Regions, so large document batches could be queued for extended periods resulting in long processing wait times for the user. This service functionality is handled asynchronously as the duration of the request could be many minutes. The generalized callback pattern makes sure that workflows resume appropriately as soon as a job finishes rather than waiting an arbitrary time to check if the job has been completed. For example, the generalized callback pattern for BDA can be used to enhance the solution outlined in the blog post, Scalable intelligent document processing using Amazon Bedrock Data Automation.
Figure 4: A data automation workflow diagram
Solution architecture
The following architecture diagram shows the generalized callback pattern (the blue section on the right side) integrated with your existing application (the grey section on the left side).
Figure 5: The Step Functions generalized callback architecture
Key components of this post’s solution architecture
This generalized callback pattern architecture consists of four essential components working together. Each component plays a specific role while maintaining cost efficiency and operational reliability. The following components form the foundation of this pattern:
Step Functions task: Implements the “Wait for Callback” task state generating unique task tokens for workflow resumption.
EventBridge rule: Monitors asynchronous service completion events and is customizable for different service patterns. AWS services make use of an event bus to route service event notifications to other services, such as job completions.
DynamoDB: Provides persistent storage correlating job IDs with task tokens for quick lookup.
Step Functions state machine: Manages the resume process and makes sure of proper cleanup of stored tokens.
Solution process
This generalized callback pattern operates through a coordinated sequence of four key steps. Each step builds upon the previous one. The following process demonstrates how the pattern manages workflow execution. The diagram above shows more detailed steps following these key steps.
Start the asynchronous operation for which you want to wait for completion. The asynchronous service responds with success (200 OK) and the state machine continues. Initiating an Amazon Translate batch translation operation is one example of such an asynchronous operation.
Trigger the generalized callback pattern with the “Wait for Callback” capability. Pair the task token with the jobId in DynamoDB using the unique jobId as the primary key. Example:
{
id = translationJobId,
token = stepFunctionTaskToken
}
Monitor for completion: When the asynchronous service completes the requested job, such as translation of documents, an event is created in EventBridge that contains the jobId and status. Example:
{
jobId = translationJobId,
status = complete
}
Resume workflow: The EventBridge rule triggers the workflow to resume, which looks up the task token using the jobId, resumes the paused Step Functions execution, and cleans up the database entry.
Not every service creates events for every action, so validate that your service operation generates the expected events. For example, Macie does not create events when no findings are discovered. In these cases, implement more event generation mechanisms through Amazon CloudWatch Logs subscriptions that trigger Lambda functions to create custom events.
Technical implementation of the solution
For rapid deployment of this post’s solution, AWS CDK users can use this sample CDK pattern with all key components. Alternatively, you can implement the individual components yourself by using the following steps, with each component customizable to your requirements.
Use a Step Functions task that supports “WaitForCallback” to store task token in DynamoDB
Use a Step Functions task that supports ”WaitForCallback” to store the task token in DynamoDB alongside the job ID from the asynchronous service.
AWS services generate a unique ID for that service which refers to that job/request/action. DynamoDB holds the mappings between job IDs and task tokens, supporting multiple state machines paused in parallel with concurrent execution. To prevent clashes when different asynchronous services generate overlapping IDs (for example, if Service A and Service B both generate ID “12345”), use separate DynamoDB tables for each service to maintain ID uniqueness. The sample AWS CDK pattern demonstrates this approach by providing dedicated DynamoDB tables and Step Functions state machines for each service integration. This ID-token structure allows for quick lookups for workflow resumption and cleanup.
The following ASL accomplishes this by using a DynamoDB PutItem task:
In this example, the Item object stores three values: the job ID ($.JobId), the task token ($$.Task.Token), and a TTL value ($.ttl). The ttl field configures Time to Live for automatic cleanup based on your service’s expected completion time. Since this stores only three small string values, data usage per entry is minimal. The primary consideration is the number of concurrent operations, as each active asynchronous job requires one DynamoDB entry until completion or TTL expiration.
The DynamoDB table uses “id” as the primary key and includes a “token” attribute. These fields are essential for the “WaitForCallback” pattern: the “id” (job ID) allows your asynchronous service to look up the correct entry, while the “token” (Step Functions task token) is what your service sends back to Step Functions to resume the paused workflow. The following JSON shows an example of these values:
When your asynchronous service completes its work, it retrieves the task token using the job ID, then calls Step Functions with that token to resume execution from where it paused.
The task token acts as a unique identifier for resuming execution at the exact pause point. To prevent overriding an existing record when a duplicate id is used, you can specify a “ConditionExpression”. This ASL shows just the ConditionExpression.
“ConditionExpression”: “attribute_not_exists(id)”
Create an EventBridge rule to monitor event patterns from your asynchronous service
EventBridge integration forms the heart of the event-driven resumption mechanism. You can create EventBridge rules to monitor specific event patterns from asynchronous AWS services. Most AWS services automatically publish completion events to default EventBridge at no cost, and you can use the EventBridge rule wizard to identify correct event patterns. For services that do not publish events—such as Macie that creates no events when no findings are discovered—implement shims by using Amazon CloudWatch Logs to trigger Lambda functions that generate custom events. This JSON shows the EventBridge Rule pattern definition.
At this point, you know the operation has completed, so you can safely resume the workflow. Using the job ID, call the DynamoDB GetItem operation to receive the task token. This ASL shows the task definition to get the task token for a given job ID retrieved from the event notification.
Use the task token to resume the workflow and then delete the DynamoDB entry for cleanup. This ASL shows the task definition to use the task token to resume the state machine at the point where it was paused at.
This completes the technical implementation of our solution. With all components in place—the WaitForCallback task, EventBridge rules, workflow resumption logic, and DynamoDB storage—you now have a fully functional generalized callback pattern implementation that eliminates polling and efficiently manages asynchronous operations.
Now that we’ve established how to implement the generalized callback pattern technically, let’s explore the best practices and important considerations that will help you optimize and secure your implementation.
Best practices and considerations
When implementing the generalized callback pattern in AWS Step Functions, it’s essential to understand and apply best practices that optimize costs, enhance security, and ensure efficient operation. This section outlines key considerations and recommendations for implementing the pattern effectively, focusing on cost optimization strategies and security measures that help maintain a robust and secure serverless workflow. By following these guidelines, you can maximise the benefits of the generalized callback pattern while minimising potential risks and unnecessary expenses.
Optimize costs by using this post’s generalized callback pattern
Managing costs for long-running asynchronous operations can present challenges. Traditional polling accumulates unnecessary expenses through repeated state transitions and execution time, but this post’s generalized callback pattern is an event-driven approach that significantly reduces operational costs.
Eliminate polling costs and minimize execution time
The generalized callback pattern reduces costs by eliminating polling transitions and pausing execution during wait periods. For standard workflows billed at $0.000025 per state transition, using just two transitions instead of continuous polling achieves approximately an 87% cost reduction. A 15-minute translation job polling every minute would need 15 transitions as opposed to two with the generalized callback pattern. For express workflows billed at $0.000001 per request and $0.00001667 per GB-second, the pattern delivers significant savings through reduced request count and minimal execution time. Traditional polling keeps workflows active during the entire operation, accumulating execution time charges. By contrast, the generalized callback pattern eliminates execution time charges during the wait period. In the translation job example mentioned previously in this paragraph, this could reduce the execution time from more than 15 minutes to just the seconds needed to start jobs and complete processes.
Increase resource efficiency
The callback pattern increases resource efficiency by removing constant polling, resulting in substantial reduction in CloudWatch logging and associated monitoring costs. This creates a more cost-effective solution with a reduced AWS resource footprint.
Further cost-optimize the callback pattern
Enhance cost efficiency through DynamoDB optimizations. Choose on-demand mode for unpredictable workloads or provisioned mode with auto scaling for consistent patterns, configure auto scaling settings based on usage, and implement TTL to automatically remove expired items without consuming write capacity.
Security considerations for the callback pattern
The callback pattern involves storing task tokens, processing events, and managing workflow resumption across multiple AWS services. Implementing proper access controls is essential to protect the integrity of your workflows and prevent unauthorized access or manipulation of the pattern’s components.
This section outlines the security considerations for the callback pattern, focusing on access controls for data storage and event processing.
Data storage security
Enable DynamoDB encryption at rest by using AWS owned or user managed AWS Key Management Service (AWS KMS) keys. Implement identity-based policies by defining the Step Functions AWS Identity and Access Management (IAM) role actions (such as PutItem, GetItem, and DeleteItem) and resource-based policies that specify which IAM principals can access the table. Together, these help ensure that only authorized state machines access token storage and operations are limited to minimum permissions. Also, configure TTL to automatically remove expired tokens so that these tokens do not accidentally get reused, which can result in errors with resuming the relevant AWS Step Function workflows.
Event processing security
Scope EventBridge rules precisely to match only specific necessary events. For Amazon Translate job completion, rules should explicitly match only translation job completion events, thus preventing unauthorized triggers. IAM roles should follow least-privilege principles so that only specific actions can cause workflows to resume.
Conclusion
The callback pattern presented in this post provides a solution for managing long-running asynchronous operations in serverless architectures. You can use the Step Functions “Wait for Callback” task state with EventBridge and DynamoDB to transform asynchronous services into synchronous workflows without the overhead of polling. This pattern reduces costs, improves efficiency through event-driven architecture, and maintains security through proper access controls. You can use the provided CDK implementation to implement this pattern and adapt it to your specific needs while following recommended security and cost optimization practices.
About the authors
Maria John is a Senior Solutions Architect at Amazon Web Services, helping customers build solutions on AWS.
Philip Whiteside is a Senior Solutions Architect at Amazon Web Services. Philip is passionate about overcoming barriers by utilizing technology.
Developers seek to process and enrich semi-structured big data datasets with durably orchestrated network-based workflows. For example, during quarterly earnings season, finance organizations run thousands of market simulations simultaneously to provide timely insights for scenario planning or risk management—these workloads require coordination between raw datasets and on-premise servers to provide the latest market information.
AWS Step Functions is a visual workflow service capable of orchestrating over 14,000 API actions from over 220 AWS services to build distributed applications. Now, Step Functions Distributed Map streamlines big data dataset transformation by processing Amazon Athena data manifest and Parquet files directly. Using its Distributed Map feature, you can process large scale datasets by running concurrent iterations across data entries in parallel. In Distributed mode, the Map state processes the items in the dataset in iterations called child workflow executions. You can specify the number of child workflow executions that can run in parallel. Each child workflow execution has its own, separate execution history from that of the parent workflow. By default, Step Functions runs 10,000 parallel child workflow executions in parallel.
Distributed Map can process AWS Athena data manifest and Parquet files directly, eliminating the need for custom pre-processing. You also now have visibility into your Distributed Map usage with new Amazon CloudWatch metrics: Approximate Open Map Runs Count, Open Map Run Limit, and Approximate Map Runs Backlog Size.
In this post, you’ll learn how to use AWS Step Functions Distributed Map to process Athena data manifest and Parquet files through a step-by-step demonstration.
This post is part of a series of post about AWS Step Functions Distributed Map:
Orchestrating big data processing with AWS Step Functions Distributed Map
Use case: IoT sensor data processing
You’ll build a sample application that demonstrates processing IoT sensor data in Parquet format using Step Functions Distributed Map. These Parquet data files and a manifest file containing the list of the data files are exported from Athena. The data temperature, humidity, and lbattery level from different devices. The following table shows sample of sensor data:
Example IoT sensor data
Your objective is to use the Athena data manifest file, get the list of Parquet files, and iterate over the data in the files to detect anomalies and also stream the processed data through Amazon Kinesis Data Firehose to an Amazon S3 bucket for further analytics using Athena queries. Following is the criteria to detect anomaly:
Low battery conditions: less than 20%
Humidity anomalies: more than 95% or less than 5%
Temperature spikes: more than 35°C or less than -10°C
The following diagram represents the AWS Step Functions state machine:
Parquet files processing workflow
The Distributed Map runs an Athena query which generates Parquet data files and an Athena manifest file (csv). The manifest file contains the list of Parquet data files.
Distributed Map processes these Parquet data files in parallel using child workflow executions. You can control the number of child workflow executions that can run in parallel using MaxConcurrency parameter. See Step Functions service quotas to learn more about concurrency limits.
Each child workflow execution invokes an AWS Lambda function to process the respective Parquet file. The Lambda function processes individual sensor readings and detects anomalies according to the preceeding logic and returns a processed sensor data summary response.
The child workflow sends the summary response record to Amazon Kinesis firehose stream which stores the results in a specified Amazon S3 results bucket.
The following Athena Start QueryExecution state runs an UNLOAD query to generate data files in Parquet format and a manifest file in CSV. The output will be stored in the S3 bucket specified in the UNLOAD query and the manifest file will be stored in the S3 bucket configured for the Athena workgroup.
Additional supported InputType options are CSV and JSONL. All objects referenced in a single manifest file must have the same InputType format. You specify the Amazon S3 bucket location of Athena manifest CSV file under Arguments.
The context object contains information in a JSON structure about your state machine and execution. Your workflows can reference the context object in a JSONata expression with $states.context.
Within a Map state, the Context object includes the following data:
For each Map state iteration, Index contains the index number for the array item that is being currently processed, Key is available only when iterating over JSON objects, Value contains the array item being processed, and Source contains one of the following:
For state input, the value will be : STATE_DATA
For Amazon S3 LIST_OBJECTS_V2 with Transformation=NONE, the value will show the S3 URI for the bucket. For example: S3://amzn-s3-demo-bucket.
For all the other input types, the value will be the Amazon S3 URI. For example: S3://amzn-s3-demo-bucket/object-key.
Using this newly introduced Source field in the context object, you can connect the child executions with the source object.
Run the following steps to deploy the Step Functions state machine.
Clone the GitHub repository in a new folder and navigate to the project root folder.
git clone https://github.com/aws-samples/sample-stepfunctions-athena-manifest-parquet-file-processor.git
cd sample-stepfunctions-athena-manifest-parquet-file-processor
Run the following command to install required Python dependencies for the Lambda function.
Stack name: The CloudFormation stack name (for example, sfn-parquet-file-processor)
AWS Region: A supported AWS Region (for example, us-east-1)
Keep rest of the components to default values.
Note the outputs from the AWS SAM deploy. You will use them in the subsequent steps.
Run the following command to generate sample data in csv format and upload it to an S3 bucket. Replace <IoTDataBucketName> with the value from sam deploy ouptut.
Before you can run queries, you must set up an Athena database and table for your data.
From Amazon Athena console, navigate to workgoups, select the workgroup named “primary”. Select Edit from Actions. In the query result configuration section, select the options as follows:
Management of query results – select customer managed
Location of query results – enter s3://<IoTDataBucketName>. Replace <IoTDataBucketName> with the value from sam deploy output.
Choose Save to save the changes to the workgroup
Select Query editor tab and run the following commands to create database and tables
CREATE DATABASE `iotsensordata`;
Create an Athena table in database iotsensordata that references the S3 bucket containing the raw sensor data. In this case it will be <IoTDataBucketName>. Replace <IoTDataBucketName> with the value from sam deploy output.
CREATE EXTERNAL TABLE IF NOT EXISTS `iotsensordata`.`iotsensordata`
(`deviceid` string,
`timestamp` string,
`temperature` double,
`humidity` double,
`batterylevel` double,
`latitude` double,
`longitude` double
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
WITH SERDEPROPERTIES ('field.delim' = ',')
STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.TextInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION 's3://<IoTDataBucketName>/daily-data/'
TBLPROPERTIES (
'classification' = 'csv',
'skip.header.line.count' = '1'
);
Create an Athena table in database iotsensordata that references the S3 bucket having the analytics results streamed from Kinesis Data Firehose. Replace <IoTAnalyticsResultsBucket> with value from sam deploy output. And replace <year> with the current year (e.g 2025).
Now that you have data ready and Athena set up for queries, start your state machine to retrieve and process the data.
Run the following command to start execution of the Step Functions. Replace the <StateMachineArn> and <IoTDataBucketName> with the value from sam deploy output..
The Step Functions state machine has the Athena StartQueryExecution state which has an UNLOAD query that generates the sensor data files in a parquet format and a manifest file in CSV format. The manifest will have 5 rows referencing the 5 parquet files. The state machine will process these 5 parquet files in one map run.
Run the following command to get the details of the execution. Replace the executionArn from the previous command.
After you see the status SUCCEEDED, run the following command from Athena query editor to check the processed output from Kinesis Data Firehose that was streamed to S3 bucket referenced by the Athena table created in step 4 of the preceding section.
SELECT * FROM iotsensordata.iotsensordataanalytics WHERE anomalycount = 1;
If any of the sensor data exceeds the thresholds, the healthstatus attribute will be set to “anomalies_detected”. The workflow produced a summary table of metadata which you can now query for reporting.
Review workflow performance
Using the following observability metrics, you can review key performance behavior of your data processing workflow. The AWS/States namespace includes the following new metrics for all Step Functions Map Runs.
OpenMapRunLimit: This is the maximum number of open Map Runs allowed in the AWS account. The default value is 1,000 runs and is a hard limit. For more information, see Quotas related to accounts.
ApproximateOpenMapRunCount: This metric tracks the approximate number of Map Runs currently in progress within an account. Configuring an alarm on this metric using the Maximum statistic with a threshold of 900 or higher can help you take proactive action before reaching the OpenMapRunLimit of 1,000. This metric enables operational teams to implement preventive measures, such as staggering new executions or optimizing workflow concurrency, to maintain system stability and prevent backlog accumulation.
ApproximateMapRunBacklogSize: This metric shows up when the ApproximateOpenMapRunCount has reached 1,000 and there are backlogged Map Runs waiting to be executed. Backlogged Map Runs wait at the MapRunStarted event until the total number of open Map Runs is less than the quota.
The following graph shows an example of these new metrics. Use the maximum statistic to visualize these metrics. ApproximateMapRunBacklogSize metrics appear after accounts start getting throttled on the OpenMapRunLimit limit. The OpenMapRun (orange line) is the account hard limit of 1,000 shown as a static line. The ApproximateOpenMapRunCount (violet line) is the current number of active OpenMap runs. The ApproximateMapRunBacklogSize (green line) indicates the map runs waiting in backlog to be processed. When the ApproximateOpenMapRunCount is lower than 1000 (OpenMapRun limit) there are no map runs in backlog. However, when the count reaches the OpenMapRun limit, the backlog of map runs starts to build up. After the active runs complete, the backlog will start to drain out and new runs will begin execution.
Graphed metrics from Amazon CloudWatch
Clean up
To avoid costs, remove all resources created for this post once you’re done. From the Athena query editor, run the following commands:
DROP TABLE `iotsensordata`.`iotsensordata`;
DROP TABLE `iotsensordata`.`iotsensordataanalytics`;
DROP DATABASE `iotsensordata`;
Run the following commands from the AWS CLI after replacing the <placeholder> variable to delete the resources you deployed for this post’s solution:
With this update, Distributed Map now supports additional data inputs, so you can orchestrate large-scale analytics and ETL workflows. You can now process Amazon Athena data manifest and Parquet files directly, eliminating the need for custom pre-processing. You also now have visibility into your Distributed Map usage with the following metrics: Approximate Open Map Runs Count, Open Map Run Limit, and Approximate Map Runs Backlog Size.
New input sources for Distributed Map are available in all commercial AWS Regions where AWS Step Functions is available. For a complete list of AWS Regions where Step Functions is available, see the AWS Region Table. The improved observability of your Distributed Map usage with new metrics is available in all AWS Regions. To get started, you can use the Distributed Map mode today in the AWS Step Functions console. To learn more, visit the Step Functions developer guide.
For more serverless learning resources, visit Serverless Land.
The collective thoughts of the interwebz
Manage Consent
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.