What is the AWS ProcessUnity Global Risk Exchange assessment
Many customers use third-party cyber risk management (TPCRM) services such as ProcessUnity to better manage risks from their evolving third-party environments and to drive operational efficiencies. To help with such efforts, AWS has completed the ProcessUnity Global Risk Exchange assessment of its security posture. ProcessUnity’s assessors perform the review of Global Risk Exchange assessment and validate the results annually.
This assessment approach replaces static spreadsheets with a dynamic system that combines advanced analytics, threat intelligence, and risk models with vendor input. Vendor profiles continuously update as cloud provider risk levels and AWS security posture evolve, keeping risk matrices current in near real time. Additionally, customers can also use Framework View which maps AWS assessment controls to threat profiles, MITRE ATT&CK scenarios, and global compliance standards and frameworks including National Institute of Standards and Technology (NIST) 800-53, International Organization for Standardization (ISO) 27001, Payment Card Industry Data Security Standard (PCI DSS), and Health Insurance Portability and Accountability Act (HIPAA), significantly reducing customers’ third-party due diligence burden.
To learn more about our compliance and security programs, see AWS Compliance Programs. As always, we value your feedback and questions; reach out to the AWS Compliance team through the Contact Us page.
Security has been the foundation of AWS cloud computing since day one. When we built Amazon Elastic Compute Cloud (Amazon EC2), we designed it with strong isolation that keeps customer workloads and data separated from every other tenant on the same hardware. The AWS Nitro System took that further by moving virtualization, storage, and networking onto dedicated hardware, making zero operator access to customer data the default. Customers then asked us for more. They wanted tightly scoped, attestable environments for their most sensitive processing: the code and data that required protection even from their own administrators. That’s why we built Nitro Enclaves. Today, more of those sensitive workloads run in containers on Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Container Service (Amazon ECS), and self-managed Kubernetes and container runtimes on Amazon EC2. Customers are asking for the same level of isolation and confidentiality for container environments.
Container environments today host many independent workloads on a single EC2 instance by using Linux container technology. For a lot of customers, Linux containers have become the prevalent workload delivery vehicle because of their ease of deployment and the ability to have them share resources in one host through a shared kernel. This sharing has some downsides. Operators of a shared cluster have access to all the tasks running in it. A weakness in the shared kernel reachable by one container also exposes every other container on the host. With the current rate of Linux Common Vulnerabilities and Exposures (CVEs), containerized workloads need an architecture that isolates each workload from other workloads and from cluster operators.
Introducing AWS Nitro Sandboxes
This week we introduced AWS Nitro Sandboxes, a new Amazon EC2 building block that gives each container the boundary of a hardware-isolated virtual machine (VM). Each sandbox gets the security properties of the Nitro System (strong isolation, zero operator access to customer data, and attestation) while keeping the familiar container experience. And isolation is designed so that no one outside the Nitro Sandbox can read its contents: Neither the cluster administrator, the platform team, nor AWS can read EKS pod or ECS task memory.
A Nitro Sandbox is a lightweight virtual machine that you launch and control from your EC2 instance. To your instance, it behaves like a child VM. Under the hood, it runs as a separate VM alongside your instance. The Nitro System enforces separation between the sandbox and its parent instance, between sandboxes, and between the sandbox and AWS operators. The parent instance contributes vCPU, memory, and I/O. Nitro keeps the workload isolated and, when you ask, proves what code it is running.
Because a Nitro Sandbox is a first-level guest with no nested hypervisor, and the Nitro Hypervisor enforces its CPU and memory boundary in hardware, compute and memory run at near-native EC2 performance. Instead of needing a separate instance for every workload or customer that needs a hard boundary, you can consolidate many isolated workloads onto instances you already run. You can then improve utilization and lower cost without giving up the boundary that made separate instances attractive in the first place.
Nitro Sandboxes share the same Nitro System isolation technology that we have operated for years as Nitro Enclaves. With Nitro Enclaves, we designed everything around the smallest possible surface: You dedicate full CPU cores and memory to a separate VM that you can only reach over a vsock channel. With Nitro Sandboxes, we built on the same foundation, but adjusted the experience to the needs of container workloads in three ways. First, a sandbox’s vCPUs can share cores with the parent instance through time slicing. Second, the parent can attach standard virtio network and block devices, not only a vsock channel, so the sandbox gets real networking and storage. Third, you can run the sandbox through a standard virtual machine monitor (VMM) such as Firecracker. The result is a general-purpose, hardware-isolated environment that behaves like a normal VM and runs almost anywhere EC2 runs.
We then took Nitro Sandboxes and natively integrated them into Amazon EKS and Amazon ECS, providing a turnkey solution to Nitro-backed, confidential virtual-machine-based container isolation at scale. Nitro Sandboxes are broadly available with the following features:
Instances: Every EC2 instance type that supports Nitro Enclaves, across general purpose, compute optimized, memory optimized, storage optimized, and accelerated families, on AWS Graviton, Intel, and AMD processors, in commercial Regions worldwide, at no additional charge.
Storage: Sandboxes reach storage the same way an EC2 instance does: local instance store for scratch space, Amazon Elastic Block Store (Amazon EBS) for persistent volumes, and Amazon Simple Storage Service (Amazon S3) or Amazon Elastic File System (Amazon EFS) over the network. To keep data confidential from the parent, you can use attestation-gated AWS Key Management Service (AWS KMS) keys, the same model as Nitro Enclaves, now integrated into Nitro Sandboxes, EKS, and ECS.
Networking: Each sandbox runs its own kernel, and therefore its own network stack. The parent connects that stack to your Amazon Virtual Private Cloud (Amazon VPC) the same way it does for containers today. On EKS, pods keep their own VPC IP address through the Amazon VPC CNI. On ECS, each task gets its own elastic network interface in awsvpc mode. Network policies and security groups are enforced on the parent instance, outside the sandbox, so a compromised workload cannot turn them off.
With Nitro Sandboxes, you can give every workload that needs a hard boundary its own VM, without giving it its own instance. AI agents that write and run their own code can run each session in a separate sandbox. Code that a prompt injection gets the agent to run stays inside one VM, which is deleted when the session ends. Software as a service (SaaS) providers can run many tenants on shared EKS nodes or ECS Managed Instances, each in its own sandbox, instead of giving each tenant a dedicated instance. Continuous integration and continuous delivery (CI/CD) pipelines can run privileged builds and untrusted pull request code inside a per-job VM rather than on dedicated build machines. And with confidential sandboxes, model providers can deploy closed-weight models in a customer’s account, knowing the weights are decrypted only inside an attested sandbox. Regulated industries can give auditors cryptographic proof of exactly what ran.
Isolation by design
Containers on a shared node all trust one Linux kernel. A single kernel issue can lead to data exposure across pods or tasks. With Nitro Sandboxes, each pod, task, or agent runs in its own VM with its own kernel. A kernel issue in one sandbox doesn’t affect its neighbors.
The Nitro Hypervisor establishes strong, virtual-machine-based security isolation through its leading security posture and built-in confidential computing which we continuously extend. The hypervisor removes the parent’s access to the sandbox’s memory and CPU register state. Only I/O buffers that the sandbox explicitly shares stay visible to the parent.
Nitro Sandboxes can therefore treat the parent instance as untrusted and attest to what runs inside. They use the Nitro-based Nitro Secure Module (NSM) to gain an attestation document in the same style as Nitro Enclaves. A workload can be configured to get its secrets only after it proves it is running the code you approved. The system is designed so that no one outside the sandbox can read its contents: Neither the cluster administrator, the platform team, nor AWS can read EKS pod or ECS task memory. You write AWS KMS key policies that require specific attestation measurements before AWS KMS releases a decryption key. This creates a verifiable link between who can decrypt and what code is actually running.
Isolation modes in EKS and ECS
On top of Nitro Sandboxes, EKS and ECS provide two isolation modes. With these modes, you can choose between faster launch times and cryptographic attestation:
Confidential sandbox: Sandboxes are fully attestable. Image loading and other runtime operations are gated by a Cedar policy, image contents are verified and extracted inside the sandbox, and the ephemeral layer is encrypted. AWS KMS releases your secrets only to workloads that prove they run the code you approved. Use this mode for anything that handles secrets, including when you want protection from your own operators. SaaS providers can show their customers that tenant data is protected through attestation. Regulated industries can give auditors cryptographic evidence in addition to documentation.
Isolated sandbox: Each pod (EKS) or task (ECS) runs in its own hardware-isolated VM with its own kernel, protecting the node and neighboring workloads from anything that happens inside the sandbox. The node and its operators are trusted, just as they are for pods and tasks today, and image validation works the same as it does without sandboxes. For workloads that also need protection from the node operator, use confidential sandbox.
Getting started on Amazon EKS
To get started on EKS, use EKS Auto Mode. Add runtimeClassName to your pod spec, and EKS handles instance selection, Nitro component setup, and scheduling. Use confidential-sandbox for a confidential sandbox or isolated-sandbox if you do not need protection from node operators. For self-managed EKS, you will need a compatible Amazon Machine Image (AMI) to support Nitro Sandboxes.
To learn more about the EKS experience with Nitro Sandboxes, see the Amazon EKS blog post.
Getting started on Amazon ECS
You can use Nitro Sandboxes on ECS with ECS Managed Instances. ECS launches compatible EC2 instances in your account, sets up a fully configured Bottlerocket image, and runs each task in its own Nitro Sandbox. ECS supports the two isolation modes, CONFIDENTIAL_SANDBOX and ISOLATED_SANDBOX.
Each task reports its isolation mode in a read-only trustedExecutionSummary field when you describe it. Per-task security groups, VPC Flow Logs, Application Load Balancer, Network Load Balancer, and Amazon VPC Lattice work as before. To learn more about the ECS experience with Nitro Sandboxes, see the Amazon ECS Developer Guide.
Getting started on Amazon EC2
If you manage your own instances, or build your own platform, you can launch Nitro Sandboxes directly on nearly any instance built on the AWS Nitro System. Amazon Linux and Firecracker support Nitro Sandboxes today. We have started to upstream support for Nitro Sandboxes in Linux and other virtual machine monitors. We also worked with operating system vendor partners to add support early, so you can keep the stack your workload already depends on.
In addition to containers, Nitro Sandboxes serve as a building block for more in-instance virtualization use cases. Nitro Sandboxes support a special “Direct Mode” switch which disables attestation, but allows the parent to retain access to sandbox memory. We have seen early adopters use this mode to execute non-enlightened operating systems, including Windows or Android, in sandboxes. Other customers have used Nitro Sandboxes to create fully attested package build environments, where each built package receives a tamper-proof signature of its build inputs. See the Amazon EC2 User Guide for details.
Pricing and availability
Nitro Sandboxes, along with the EKS and ECS integrations, are available at no additional charge. You pay for the EC2 instances, plus the standard EKS Auto Mode or ECS Managed Instances management fee if you use those capabilities. Nitro Sandboxes run on nearly every EC2 instance, EKS cluster, and ECS cluster, so strong per-container, per-task, and in-instance isolation is available to everyone.
Operating one of the world’s largest data-streaming platforms, Confluent, the company founded by the original creators of Apache Kafka and now part of IBM, continuously modernizes its infrastructure to stay ahead of rapid data growth. Confluent’s data-streaming products, including its Apache Kafka, Connect, and Flink fleets, are available on the AWS Cloud, and its internal logging system runs on Amazon OpenSearch Service, ingesting hundreds of terabytes of logs per day to support operations across that fleet.
At that ingest volume, indexing work dominates the hot tier. The question Confluent’s engineering team set out to answer was not “can we run on cheaper hardware,” but “how much more ingest can we absorb per node, and how much headroom does that free up?” This post shows how a measure-first migration of the OpenSearch hot tier to the purpose-built OpenSearch optimized instance type, oi2, improved indexing performance and ingestion headroom first. oi2 costs more per node, but we needed far fewer of them. The lower cost came from that efficiency, not from a lower instance price.
Why instance choice matters for ingestion
Confluent’s hot tier ran on r6gd, a previous-generation, general-purpose Graviton2 instance with local NVMe added as a bolt-on. The team migrated to the current-generation OpenSearch Optimized Instance family (oi2), purpose-built for OpenSearch. On oi2, local NVMe is the primary storage tier with no separate Amazon Elastic Block Store (Amazon EBS) volume. Data is copied synchronously to Amazon Simple Storage Service (Amazon S3) for durability, and the storage and network path is integrated specifically for indexing and merge I/O. For an ingest-bound logging workload, that storage path is the lever. It’s what lets each node sustain a higher indexing rate at lower latency.
A domain with OpenSearch Optimized Instances uses local storage as the primary tier, with data copied synchronously to Amazon S3 as it arrives. This provides increased indexing throughput with high durability (11 nines). Within the family, OR1, OR2, and OM2 use Amazon EBS volumes (gp3 or io1) locally, whereas oi2 instances use local NVMe disks. These instances benefit indexing-heavy operational analytics workloads such as log analytics, observability, and security analytics.
OpenSearch supports two replication strategies: logical (document) and physical (segment) replication. With logical replication, data is indexed on all the copies independently, leading to redundant computation on the cluster. OpenSearch Optimized Instances use the physical replication model, where data is indexed only on the primary copy. Additional copies are created by copying the segments from the primary (durably persisted to Amazon S3). Because this frees up compute resources on replicas that would otherwise redundantly index data, and network overhead on the primary, the cluster can churn more throughput. When Optimized Instances are configured with replicas, the indexing rate might appear lower than it actually is (for example, showing a rate of 1,000 when the actual indexing rate is 2,000). This is because indexing is only performed on primary shards.
Reference architecture
The following diagram shows the OpenSearch multi-domain topology, split by priority (P0/P1) and volume, behind Cross-Cluster Search.
Figure 1: OpenSearch multi-domain topology, with writes going directly to the P0 and P1 domains and reads served through cross-cluster search
A measure-first approach
Each domain was modernized in two logical phases, so the efficiency of the new instance could be measured in production before capacity was removed:
Phase 1, a type swap at constant node count: an in-place Amazon OpenSearch Service blue/green re-provision at the same node count, to measure real production efficiency at fixed ingest. At equal nodes, oi2 ran at roughly one-third the CPU of r6gd for the same ingest, about 2.7x the headroom per node.
Phase 2, iterative rightsizing: nodes were removed in steps, holding at each step to confirm cluster health, until a guardrail became binding. That guardrail turned out to be JVM heap, not CPU, because heap pressure is driven by query patterns, and removing nodes concentrates the same shards onto fewer of them.
Spending the Phase 1 headroom this way let the fleet consolidate to roughly half the nodes. Each oi2 node carried about 1.9x the load of an r6gd node (168 ÷ 90), while indexing got faster, not slower.
Production results: Ingestion efficiency first
The following figures are production averages at the same ingest (7-day baseline on r6gd compared to a 24-hour window on oi2 at the final node counts). Because the log traffic is identical, the comparison reflects a real change per unit of work. We didn’t benchmark R8gd for this migration. Our evaluation focused on the OpenSearch optimized (oi2) family, which is built specifically for log-analytics ingestion and storage economics.
Figure 2: Total hot-tier fleet size, 168 r6gd nodes compared to 90 oi2 nodes, a 46% reduction
Figure 3: Node count per domain on r6gd compared to oi2, with reductions of 31 to 60 percent
Figure 4: Average indexing latency per domain on r6gd compared to oi2, roughly halved across all four domains
Figure 5: Write queue depth in queued operations per domain on r6gd compared to oi2, cleared to near zero
Domain (anonymized)
Nodes
Avg CPU
Indexing latency
Write queue (ops)
Domain A (high-volume)
30 → 12 (-60%)
49% → 43%
0.19 → 0.08 ms
17.6 → 0.2
Domain B (streaming logs)
36 → 18 (-50%)
60% → 40%
0.18 → 0.08 ms
20.6 → 0.3
Domain C (high-volume)
39 → 27 (-31%)
58% → 39%
0.19 → 0.09 ms
37.9 → 0.2
Domain D (low-volume)
63 → 33 (-48%)
43% → 56%
0.18 → 0.10 ms
8.4 → 1.0
Fleet
168 → 90 (-46%)
—
~halved
<1 everywhere
Indexing latency roughly halved everywhere: approximately 0.18–0.19 ms on r6gd to approximately 0.08–0.10 ms on oi2, reflecting oi2’s faster storage path.
Write backpressure cleared. Every r6gd domain carried a double-digit write queue (8–38 queued ops, the indexing-backpressure signal). On oi2, all four dropped below approximately 1. Faster storage keeps indexing current.
CPU fell even on far fewer nodes for three of four domains, the clearest sign of per-node efficiency (for example, Domain B: 60% → 40% on half the nodes).
Heap, not CPU, is the ceiling. oi2 has the same heap per node as r6gd, so concentrating shards onto fewer nodes raised average heap on every domain. This is what set the floor on rightsizing.
The honest outlier. Domain D was cut the hardest relative to its load. Its CPU rose (43% → 56%) and its write queue, although far better, is the only one not near zero. It’s the domain to watch through peak traffic before calling it settled.
Stability: zero write/primary-write rejections in the final state. All clusters healthy throughout.
In AWS internal benchmarks, oi2 instances deliver up to 9 percent higher indexing throughput than OR2 instances and up to 33 percent more than I8g instances. Each OpenSearch Optimized instance is provisioned with compute, third-generation AWS Nitro SSDs for caching, and remote managed storage on Amazon S3. The oi2 family is available in sizes from large through 24xlarge with up to 22.5 TB of storage.
To maximize indexing throughput from OpenSearch Optimized Instances, AWS recommends:
Use large bulk sizes to improve buffer utilization. The recommended size is 10 MB.
Use multiple clients to improve parallel processing performance.
Consider more primary shards to raise indexing throughput, but note that over-sharding every index drives up per-node shard density and lowers it.
Keep the index refresh interval at 10 seconds or higher (the default for Optimized Instances is 10 seconds).
Fewer nodes, lower bill — even at a higher per-node price
oi2 carries a higher per-node price than r6gd, so a like-for-like swap alone would have raised spend. The saving came entirely from needing fewer nodes: consolidating the hot tier by approximately 46 percent more than offset the premium, for an estimated approximately 18 percent lower hot-tier cost at on-demand pricing. Committing the final footprint to 1-year reserved instances is estimated to take the hot tier to roughly 47–51 percent below the original r6gd on-demand baseline.
Cost figures are approximate, list-derived percentages at on-demand pricing (reserved-instance figures are estimates pending a quoted rate). Storage (UltraWarm and snapshots) and dedicated-master tiers were unchanged and are excluded. The takeaway for teams modernizing OpenSearch: optimize efficiency per unit of work, not instance unit price.
Conclusion
With the measure-first approach in two phases, Confluent raised ingestion efficiency and halved indexing latency on its Amazon OpenSearch Service cluster. Before removing any capacity, let the real sizing signal (heap in this case) guide the optimization work. This approach is what improved the efficiency of the system and lowered the overall cost. The same measure-first approach applies to ingest-bound OpenSearch workloads evaluating purpose-built storage-optimized instances. To learn more, see the Amazon OpenSearch Service documentation on instance types and sizing domains.
Greg Kroah-Hartman has released the 7.3.4, 6.18.63, and 6.12.117 stable kernels. The 7.3.4 kernel has more
than 1,250 patches, 6.18.63 contains more than 860, and 6.12.117 includes more
than 1,530 patches. Users of these kernels are, of course, encouraged to upgrade.
AWS re:Invent returns to Las Vegas for cloud and AI’s biggest week from November 30–December 4, 2026. This year we have over 2,200 sessions, 70 percent of which are interactive, led by experts from Amazon Web Services (AWS), our customers, and our partners.
With that much to choose from, we’ve pulled together the sessions built around Digital Sovereignty at AWS. Whether you’re navigating evolving data residency requirements, choosing the right home for your most sensitive workloads, or scaling AI in environments where your data can’t leave the jurisdiction, there’s something for you at re:Invent.
In this guide, you’ll find breakout sessions, chalk talks, workshops, and a lightning talk covering a range of digital sovereignty topics, grouped by theme so you can easily find what matters most to you. Reserve your seat through your attendee portal, or walk up based on availability. If a session is full, check the catalog for another time, since several of these sessions run more than once.
Pro tip: Sessions are spread across several venues, so allow extra time to get between them. All times are Pacific.
Theme 1: Control, choice and operational autonomy with the AWS European Sovereign Cloud
In 2026, we launched the AWS European Sovereign Cloud, giving customers another option to help meet their most stringent digital sovereignty requirements. As part of its design, the AWS European Sovereign Cloud operates with no critical dependencies on non-EU infrastructure, and is backed by strong technical controls, sovereign assurances, and legal protections. In these sessions, get the full picture of the AWS approach to digital sovereignty, including verifiable controls, operational autonomy, and greater choice with infrastructure options like the AWS European Sovereign Cloud.
SEC206 | Breakout session | Digital sovereignty: Verifiable controls and greater choice Speakers: Mustafa Isik, Chief Technologist Sovereignty, AWS; Stephane Israel, Managing Director European Sovereign Cloud and Digital Sovereignty, AWS When and where: Thursday, December 3, 1:00 PM | Wynn/Encore Evolving regulations and geopolitical shifts are prompting organizations to rethink how they protect data and maintain control in the cloud, without slowing innovation and AI adoption. This session explores how AWS helps customers meet their digital sovereignty requirements with the full power of the cloud through a range of offerings, including the AWS European Sovereign Cloud, backed by verifiable and auditable controls. Learn what AI sovereignty means in practice, get answers to the most common questions, and hear how AWS and partners are helping organizations navigate complexity in a fast-changing landscape.
SEC234 | Lightning talk | EU operational autonomy in action: AWS European Sovereign Cloud Speakers: Zoltan Albrecht, Director, AWS European Sovereign Cloud Managed Operations, AWS When and where: Thursday, December 3, 8:30 AM | Wynn/Encore The AWS European Sovereign Cloud is built on fully isolated infrastructure within the EU, operated exclusively by EU-resident staff, with no access from outside Europe. In this lightning talk, learn from the AWS European Sovereign Cloud Managed Operations team what independent operations means in practice and how it works day to day.
Theme 2: AI under digital sovereignty and data residency requirements
Moving AI from pilots to production means making clear decisions about where your data and AI workloads run to meet your digital sovereignty needs. These sessions show how to bring AI infrastructure to your data, from fully managed AWS AI Factories in your data center, to the right hybrid deployment for each workload, to agentic AI applications that extend to hybrid and edge environments such as AWS Outposts and AWS Local Zones.
AIM206 | Breakout session | AWS AI Factories: Bring production scale AI to your data center Speakers: Pranav Chachra, Senior Manager, Product Management, AWS; Gareth Tucker, Global Specialist – AWS AI Factories, AWS; Daniel Xu, Member of Technical Staff, Anthropic When and where: Thursday, December 3, 10:30 AM | Venetian Data sovereignty, data gravity, and latency requirements prevent many organizations from running all AI workloads in the public cloud. AWS AI Factories bring AWS and NVIDIA AI infrastructure directly into your data center, fully operated, maintained, and continuously updated by AWS. Run leading foundation models on premises for training and inference without building and managing AI infrastructure yourself or negotiating separate agreements with model providers. In this session, learn how organizations are accelerating AI adoption, reducing time to value, and maintaining full control of their data while benefiting from the AWS experience in their own facilities.
HMC303 | Breakout session | Choosing the right hybrid deployment for your AI workloads Speakers: Shivam Saggar, Sr. Product Manager (Tech), AWS; Chris McEvilly, Senior Solutions Architect, AWS; Shelby Seward, DMTS, Sr. Director, T-Mobile; Hai Lu, Senior Engineering Manager, Databricks, Inc. When and where: Monday, November 30, 9:00 AM | MGM Grand Organizations are running AI workloads for model training, fine-tuning, reinforcement learning and inference, across an expanding set of environments, including AWS Regions, AWS Local Zones, AWS Outposts, AWS Dedicated Local Zones, and AWS AI Factories. The right deployment option depends on latency, data sovereignty, compute density, and scale. See how customers deploy accelerated AI across the hybrid spectrum: NVIDIA GPU-based and AWS Trainium-powered instances in Local Zones, purpose-built AWS AI Factories for large-scale training and model serving, and edge deployments for real-time inference. Hear how customers navigate these decisions, including Databricks tapping GPU capacity on AWS Local Zones and T-Mobile running latency-sensitive distributed inference, and learn which environment fits each AI workload without compromising performance and compliance.
HMC402 | Workshop | Implementing agentic AI solutions on-premises and at the edge Speakers: Kyle Palasti, Edge Specialist Solutions Architect, National Security, AWS; Fernando Galves, Senior Applied AI Architect, AWS When and where: Wednesday, December 2, 3:30 PM | Wynn/Encore Repeat: Thursday, December 3, 3:00 PM | Caesars Palace Learn how to extend Amazon Bedrock AgentCore to hybrid and edge services to build distributed agentic apps using Model Context Protocol (MCP) and agent-to-agent (A2A) communication with on-premises data. Get hands-on with hybrid agentic AI using Amazon Bedrock and Strands Agents while exploring AWS hybrid and edge solutions for data residency compliance.
HMC403 | Workshop | Self-managed agentic AI at the hybrid edge Speakers: Fernando Galves, Senior Applied AI Architect, AWS; Chris McEvilly, Senior Solutions Architect, AWS When and where: Tuesday, December 1, 12:30 PM | MGM Grand Repeat: Thursday, December 3, 8:30 AM | MGM Grand As organizations use agentic AI to automate complex infrastructure operations, many have data residency, low latency, and security requirements that depend on running AI closer to their data. This workshop explores how customers deploy self-managed agentic AI at the hybrid edge to meet these requirements while enabling autonomous operations. Learn how to architect, provision, configure, and optimize AI agents that run locally, and how to extend the same architecture to AWS hybrid and edge infrastructure such as AWS Local Zones and AWS Outposts.
Theme 3: More control, more choice: Finding the right infrastructure for your digital sovereignty needs
Digital sovereignty requirements vary, and AWS offers control and choice across a range of deployment options. Whether you’re comparing AWS Dedicated Local Zones, AWS Outposts, the AWS European Sovereign Cloud, and AWS Regions, or planning a migration, you’ll find the infrastructure options to meet your unique digital sovereignty needs. These sessions help you learn more and choose with confidence.
HMC314 | Chalk talk | Architecting for digital sovereignty with AWS Hybrid Cloud solutions Speakers: Sherry Lin, Principal Product Manager, AWS; Lakshmi VP, Solutions Architect – WWPS, AWS When and where: Wednesday, December 2, 8:30 AM | MGM Grand This session uses real-world case studies to understand how AWS hybrid cloud architectures such as AWS Local Zones, AWS Dedicated Local Zones, and AWS Outposts help customers meet their digital sovereignty needs. It examines data residency boundaries, network separation, operational controls, and security guardrail options available in hybrid cloud solutions. It then shows how to configure governance controls, including Service Control Policies (SCPs), to help customers comply with regulatory requirements. Finally, it covers how to detect and prevent unauthorized data flows between hybrid infrastructure and the parent Region.
HMC202 | Breakout session | AWS wherever you need it: From the cloud to the edge Speakers: Spencer Dillard, Director, Software Dev, EC2, AWS; Ian Cottingham, Director, Software Development, AWS; Rama Raghavan, AVP Cloud Architecture, AT&T Inc. When and where: Wednesday, December 2, 3:00 PM | Caesars Palace While most workloads can run in the cloud, some remain on-premises or at the edge due to low latency, local data processing, or data residency needs. AWS Local Zones, AWS Outposts, AWS Dedicated Local Zones, and AWS AI Factories extend AWS services, APIs, and tools to metro areas, on-premises data centers, and the far edge. Explore how these infrastructure offerings power a broad range of hybrid cloud and edge computing use cases such as financial trading, manufacturing, and AI. Learn how AWS helps you run regulated workloads with edge requirements, power generative AI with edge data, and migrate workloads with on-premises dependencies, featuring AT&T’s migration and modernization at scale.
HMC304 | Breakout session | Real-world migrations: Lessons learned migrating to AWS Outposts Speakers: Frankie Negro, Senior Hybrid Edge Solutions Architect, AWS; Scott Rose, Senior Manager Solution Architecture – Telco, AWS; Marcelo Andrade, Cloud Engineer, SERPRO When and where: Monday, November 30, 10:00 AM | MGM Grand Organizations with data residency and digital sovereignty requirements face a critical question: how do you move workloads from on-premises infrastructure to AWS Outposts without disrupting compliance posture or operations? This Level 300 session distills lessons from migrating customers to Outposts across industries, organization sizes, and migration strategies, featuring SERPRO, Brazil’s federal data processing company, and AT&T, migrating telecom workloads at scale. The session walks through the processes that delivered results, the quantifiable outcomes measured, and the critical mistakes to avoid. Learn how to select the right migration approach and navigate common blockers. The session also covers transitioning from first-generation to second-generation Outposts hardware.
Theme 4: Disaster recovery, resilience, and cross-partition connectivity
Digital sovereignty goes beyond where your data lives. It also shapes how you operate, recover, and connect your workloads. These sessions help you learn about the operational side of digital sovereignty: recovering from failures under regulatory requirements, designing for resilience alongside data residency, and strategies for connecting across global AWS Regions, GovCloud, the AWS European Sovereign Cloud, and China Regions.
ARC405 | Breakout session | Your regulator set your RTO. Not your architecture. Speakers: Hossam Ebrahim, Solutions Architect, AWS; Paul Armstrong, Lead Architect UKI, AWS When and where: Tuesday, December 1, 4:00 PM | Wynn/Encore Organizations with pre-approved failover regions recover in hours. Those without recover in weeks. Same impairment. Same services. One governance decision separates them, and it’s made years before the event. This session presents architectural patterns for multi-Region recovery under sovereignty constraints, drawn from real operational experience with regulated workloads under regional pressure. Three categories: decisions that set your resilience ceiling before any event, dependencies that create circular failure during recovery, and assets nobody backs up until they become the critical path. You leave with the decisions to evaluate before your next DR test.
ARC318 | Chalk talk | Architecting for resilience and data residency Speakers: Andy Orlosky, AWS Solutions Architect, AWS; Mike Haken, Senior Principal Solutions Architect, AWS; Jamie Pasterick, Senior Solutions Architect, AWS When and where: Monday, November 30, 2:30 PM | Wynn/Encore Repeat: Wednesday, December 2, 1:30 PM | Wynn/Encore Your data can’t leave the jurisdiction, but your workload still has to survive a Region degradation. This chalk talk explores architecture patterns that address both resilience and data residency together. We break the challenge into practical layers: where data lives, how workloads recover, what isolation looks like, and how to maintain business continuity across geographic boundaries. We map each layer to AWS capabilities including Dedicated Local Zones, Outposts, and policy controls, then work through real scenarios together. Bring your architecture challenges and we will whiteboard design patterns and tradeoffs with you.
NET327 | Breakout session | Connectivity strategies across AWS Partitions Speakers: Kunal Sharma, Sr. Solutions Architect, AWS; Rohit Aswani, Principal Solutions Architect – Networking Specialist, AWS When and where: Wednesday, December 2, 4:00 PM | MGM Grand Organizations increasingly operate across AWS commercial, GovCloud, Sovereign Cloud, and China Regions to satisfy regulatory, data residency, and national security requirements. But connecting them introduces networking challenges that go well beyond standard multi-Region design. In this session, learn cross-partition connectivity patterns using AWS Site-to-Site VPN, AWS Direct Connect, and application-layer proxies alongside DNS resolution strategies, compliance boundary enforcement, and operational trade-offs. We walk through architectural blueprints for securely linking partitions while preserving the isolation guarantees they exist to enforce, and share lessons learned from customers navigating ITAR, GDPR, and national sovereignty mandates at scale.
Theme 5: Digital sovereignty across industries
Priorities for digital sovereignty vary by industry, and are shaped by different regulations, risks, and requirements. Learn how organizations in government, energy, and telecommunications are meeting their unique needs with the control, choice, and scale of AWS.
IND361 | Workshop | Build hybrid data planes for government GenAI pipelines Speakers: Mohamed Salah, Sr. Solutions Architect, AWS; Usama Salama, Sr. Specialist SA AI/ML, AWS When and where: Monday, November 30, 3:30 PM | Wynn/Encore Strict national security mandates block government agencies from adopting cloud GenAI. To resolve this, architects deploy a decoupled edge-to-cloud tokenization pattern. In this workshop, use AWS CDK to build a three-phase data plane that isolates sensitive citizen data in a simulated edge environment. You will deploy local data tokenization, route masked payloads to Amazon Bedrock for stateless inference, and reconcile state on-premises. Leave with an infrastructure-as-code blueprint to unlock frontier AI and satisfy federal localization mandates.
IND359 | Chalk talk | In-Country HPC: How TGS Processes Seismic Data on AWS Brazil Speakers: Kun Jiao, Principal Solutions Architect, AWS; Faisal Asif, Global Head, Cloud Transformation and Innovation, TGS When and where: Monday, November 30, 9:00 AM | Wynn/Encore Repeat: Tuesday, December 1, 12:00 PM | Wynn/Encore TGS — one of the world’s largest multi-client geoscience data providers — faced a significant cost challenge running its business-critical Rackmon EFWI GPU workload at scale. By strategically deploying this workload on AWS infrastructure in Brazil, TGS leveraged Brazil’s ReData Special Tax Regime to dramatically reduce the tax burden on GPU-intensive seismic processing. The result: material cost savings on one of the most computationally demanding workloads in the energy industry — and a reusable in-country processing architecture that can serve future customers with data residency requirements in other geographies.
IND322 | Chalk talk | Physical AI in Telecom: A Framework for Workload Placement Speakers: Young Jung, Sr. Manager of Telco Network SA, AWS; Angelo Sampietro, Principal Solutions Architect, AWS; Umer Chaudhary, Sr. Mgr Network Solutions, AWS When and where: Tuesday, December 1, 1:30 PM | Caesars Forum Repeat: Tuesday, December 1, 3:00 PM | Caesars Forum Some physical AI workloads need sub-millisecond performance at edge cell sites, others are better suited to a full AWS Region, and many fall in between on AWS Local Zones or AWS Outposts. In this chalk talk, we provide a framework for matching each workload to the right option. We trace the physical AI development lifecycle, examining how data sovereignty, latency, data gravity, and token economics interact to shape placement decisions at each phase. Based on real-world use cases, we discuss the options so you leave with a placement framework you can apply immediately.
You can view the full list of digital sovereignty sessions in the re:Invent catalog, including sessions with partners.
Activities in the Expo
Join us in the re:Invent Expo at the Venetian to meet with AWS experts and learn through interactive demos. Stop by these kiosks in the AWS Village:
Digital Sovereignty: Learn more about Digital Sovereignty at AWS and the choice and control we offer across the cloud and AI. Use the AWS Digital Sovereignty Lens to get extended guidance for designing, building, and operating sovereign workloads on AWS, see how AWS helps your organization innovate with AI while meeting your sovereignty needs, and go deep on the AWS European Sovereign Cloud with AWS experts.
Hybrid Cloud & Edge Computing: See how to run familiar AWS services, APIs, and tools you use in AWS Regions on premises and at the edge with AWS Local Zones, AWS Dedicated Local Zones, AWS Outposts, and AWS AI Factories. Bring a workload that needs to stay local, and AWS experts will walk through its architecture with you.
Global Infrastructure & Sustainability: Explore the scale and reach of the world’s most extensive cloud infrastructure including AWS Regions, Availability Zones, AWS Local Zones, AWS Outposts, and AWS AI Factories. Discover the latest infrastructure innovations and how AWS Sustainability helps you scale workloads globally while minimizing your environmental impact.
Data Center Experience: See the technology inside AWS data centers firsthand, from custom chips to full server racks, and explore a 3D model of an AWS AI Factory.
Get ready for re:Invent 2026
Here’s how to prepare for the digital sovereignty sessions:
Register for AWS re:Invent 2026. Don’t have a ticket yet? Use the discount code REINVENT26 for $150 off a full conference pass. The code expires on November 23, 2026, and can’t be combined with any other offers or discounts.
The day is finally here for the launch of AMD’s EPYC 9006 “Venice” platform and the Zen 6 CPU architecture. AMD has been hard at work over the past two years to build a better server CPU, and thanks to its performance, Venice is primed to deliver another knockout blow for AMD
In highly regulated industries such as financial services, healthcare, and government, security isn’t negotiable. Compliance frameworks require defense in depth strategies where incoming requests are inspected against security policy to confirm they are safe before reaching applications.
Organizations typically solve this challenge by implementing centralized ingress proxy fleets: hardened entry points that enforce consistent authentication, authorization, threat detection, and audit logging across hundreds of applications.
For critical applications that use regional services and need to operate multi-Region disaster recovery (DR) strategies, this creates a challenge. These applications must regularly demonstrate failover capability, not only to prove DR works, but to satisfy auditor expectations for tested, repeatable recovery procedures. Yet with centralized ingress controls in place, the teams accountable for recovery time objective (RTO) commitments often cannot independently execute failover. They depend on centralized infrastructure teams who might be handling many similar requests simultaneously. This introduces coordination overhead, extends recovery times, and conflicts with the DevOps principle of “you build it, you run it.” Scale this to dozens of interdependent applications and the problem compounds. Individual team recovery is necessary, but without coordinated portfolio-level failover, the enterprise remains exposed.
The challenge isn’t whether to implement defense in depth, which is non-negotiable, but how to preserve application team autonomy, support self-service failover within the constraints of centralized enterprise ingress controls, and ultimately orchestrate coordinated recovery across the portfolio.
In this post, we show how application teams in regulated enterprises can execute self-service regional failover using Amazon Application Recovery Controller (ARC) Region Switch, even when operating behind centrally managed ingress proxy fleets. The pattern uses AWS Transit Gateway peering, Amazon Route 53 Private Hosted Zones, and (optionally) Amazon VPC Lattice to maintain security controls while removing cross-team coordination during DR events. We then demonstrate how a central parent plan can run each application team’s child plan, shared through AWS Resource Access Manager (AWS RAM), to support coordinated portfolio-level failover. A single central action runs dependent plans across the enterprise in dependency order, delivering deterministic, fully automated recovery orchestration that replaces the manual coordination of today’s DR playbooks.
Why ARC Region Switch?
Launched in August 2025, Amazon Application Recovery Controller Region Switch is purpose-built for disaster recovery orchestration. Custom scripts and home-grown automation must be built, tested, and maintained independently, often across multiple AWS Regions with no guarantee of synchronization. Region Switch is a fully managed, highly available failover orchestrator that removes the undifferentiated heavy lifting of DR operations.
Effective disaster recovery orchestration for enterprise applications has to overcome three common challenges. The following table describes each one and how Region Switch addresses it.
DR challenge
How ARC Region Switch addresses it
Configuration drift. A recovery plan written months ago might no longer match the environment it protects. Permissions change, resources are renamed, and standby capacity shrinks, and teams often only find out mid-failover.
Plan evaluation. Region Switch automatically evaluates your plan’s configuration, AWS Identity and Access Management (IAM) permissions, and resource readiness multiple times per hour, surfacing issues before they block a recovery. Warnings are visible on the console and available through Amazon EventBridge or the API.
Slow detection and decision-making. In a real incident, time is lost noticing the problem, assembling the right people, and agreeing to fail over. All of it adds to your RTO.
Automatic triggering. Plans can be initiated automatically from Amazon CloudWatch alarms, removing the time to detect and time to decide from your effective RTO. Teams that want a human checkpoint can add a manual approval step.
Recovery logic that drifts across Regions. Multi-step recovery is often scripted separately for each Region. The scripts diverge, are hard to review, and fail silently when they’re finally needed.
Codified recovery workflows. Plans are fully codifiable with AWS CloudFormation, the CLI, and the API, so they’re version-controlled and deployed through your existing continuous integration and continuous delivery (CI/CD) pipelines alongside the application. The visual Workflow Builder helps teams design and modify recovery steps, with the ability to export the configuration back into your infrastructure as code (IaC) pipeline.
A Region Switch plan isn’t bound to a single Region. Its execution engine operates independently in each Region, keeping the failover orchestrator outside the blast radius of the very impairment it’s responding to.
The solution: Self-service regional failover with ARC Region Switch
Application teams create a plan in the ARC Region Switch Workflow Builder that includes task-specific execution blocks: automated steps that handle different aspects of the failover activity. A typical plan might include execution blocks to promote database read replicas, scale compute resources (Amazon Elastic Compute Cloud (Amazon EC2) Auto Scaling groups, EKS pods, or ECS tasks), manipulate Route 53 health checks to redirect traffic, and invoke custom AWS Lambda functions for application-specific logic.
When the team runs a Region Switch plan, whether for a DR test or during an actual incident, ARC runs this workflow from the recovery Region automatically, orchestrating the entire failover sequence without requiring manual intervention from application teams.
Figure 1: ARC Region Switch runs a plan’s execution blocks from the recovery Region to orchestrate failover
To understand how ARC Region Switch supports this self-service capability within the constraints of centralized ingress controls, we need to examine the underlying infrastructure. The architecture relies on a shared responsibility model where infrastructure and application teams each control distinct layers of the stack, creating clear boundaries that preserve security while supporting operational independence.
Building the foundation: Centralized networking with decentralized control
The architecture rests on two foundational elements: centrally managed AWS Transit Gateways and an application-team-owned Amazon Route 53 Private Hosted Zone (PHZ).
Figure 2: Centrally managed Transit Gateways with an application-team-owned Route 53 Private Hosted Zone
A central infrastructure team owns and manages the peered Transit Gateway infrastructure in both AWS Regions, providing VPC attachments to each application team account so traffic can flow from their secure ingress proxy solution. The application team owns and manages a Route 53 PHZ, associated with all four VPCs: both the infrastructure team’s centralized ingress proxy fleet VPCs and the application team’s own VPCs in each Region. This PHZ contains a failover recordset that the ingress proxy fleet uses to resolve the active application endpoint.
This establishes a clear technical boundary. The infrastructure team provides the secure networking foundation and VPC attachments. The application team controls the Domain Name System (DNS) records that determine where traffic flows. Neither team needs to coordinate changes with the other during failover events.
Understanding the multi-Region architecture
To understand how these components work together, let’s examine the complete architecture. The following diagram illustrates the multi-Region design, showing how AWS Global Accelerator, cross-Region AWS Transit Gateway peering, Amazon VPC Lattice, and Amazon Route 53 Private Hosted Zones combine to support self-service failover through Amazon ARC Region Switch while maintaining centralized security controls.
Figure 3: Multi-Region architecture combining Global Accelerator, Transit Gateway peering, VPC Lattice, and Route 53 to support self-service failover
AWS Global Accelerator provides static anycast IPs for the ingress proxy fleet. The 50/50 traffic weighting reflects the platform’s multi-tenant design. Because the centralized ingress proxy fleet serves multiple application teams simultaneously, it must be able to route traffic to applications running in either Region, or active-active across both.
The goal of AWS Global Accelerator is to route traffic to the closest point of presence. If your application is active in Region 1 but a consumer is closer to Region 2, traffic routes to the Region 2 endpoint first, then traverses the cross-Region Transit Gateway peering connection to reach the active application.
The architectural trade-off
This represents an intentional trade-off: accepting an additional network hop through Transit Gateway peering in exchange for true self-service failover. Without this cross-Region connectivity, application teams would need to coordinate with infrastructure teams during failover, reintroducing the exact coordination dependency this architecture seeks to remove.
Fine-grained failover control with Amazon VPC Lattice service network endpoints
While the architecture we’ve described supports independent regional failover, Amazon VPC Lattice service network endpoints add another layer of flexibility worth highlighting: fine-grained control over which services fail over.
Each service network endpoint acts as a consolidated endpoint that can host multiple VPC Lattice services behind it. When you associate a Lattice service with a service network endpoint, it receives its own unique Regional DNS name. For example, an authentication service and a payments service hosted behind the same endpoint each receive distinct DNS names. These per-service DNS names each map to a Route 53 Private Hosted Zone failover record set, giving each service its own independent failover control.
Figure 4: Each VPC Lattice service behind a service network endpoint maps to its own Route 53 failover record set
The key benefit is fine-grained control: application teams can fail over individual Lattice services one at a time, rather than being forced to flip every service behind the same service network endpoint at once. You can validate your authentication service this week, your payment processing service next week, and your inventory service the week after. Each has its own dedicated failover record set in your PHZ, and each is tested without affecting services still running in the primary Region.
This component-level approach requires the least operational overhead, but it carries a risk of bi-modal behavior, where a workload performs differently, with different response times or processes, under normal conditions versus failure modes. Each independent failover adds another active/standby combination, and each must tolerate the cross-Region latency when a service calls a dependency now in the other Region. Design your components to be resilient to this latency and test the realistic combinations. Component-level failover is best suited to teams who have that confidence, so where you don’t, a coarser-grained approach might be the safer choice.
Operationally, the boundary stays clean. The infrastructure team’s ingress proxy fleet continues resolving through the service network endpoint, while the application team controls which services are active in which Region by managing the health check on each service’s DNS record in their PHZ. As your application grows, you associate new microservices with the existing service network endpoint and add their failover record sets, with no changes required to the infrastructure team’s proxy configuration or VPC attachments.
Note: While this post demonstrates a pattern using Amazon VPC Lattice as the application endpoint target, the same shared responsibility approach works regardless of your application stack. Whether you’re running containers on EKS, serverless functions, or traditional EC2 workloads behind Application Load Balancers or Network Load Balancers, the foundational pattern of centralized networking with decentralized DNS control applies equally.
How an application team fails over independently
When the application team runs their ARC Region Switch plan, traffic shifts between Regions without any involvement from the central infrastructure team.
Each application team can independently test their failover capability at their own cadence, as frequently as they choose. These tests are completely independent: one team’s DR exercise has no impact on any other team’s services. As noted in the preceding section, teams should validate the realistic combinations of failed-over and in-place services rather than only the all-in-one-Region baseline.
The failover sequence
What makes this powerful is that it’s a single orchestrated flow, not a set of manual steps. ARC Region Switch runs each block, verifies it has completed successfully, and only then manages the transition to the next, so the sequence advances deterministically, with no human stitching steps together under pressure.
1. Plan initiation. The application team runs their Region Switch plan through the AWS Command Line Interface (AWS CLI) or API, whether for a scheduled DR test or in response to a real incident. The application team can also set up CloudWatch alarms to trigger plans automatically, then run either fully automatically, with no human intervention required, or with a manual approval step built into the workflow that gives teams control to gate execution at critical points.
Figure 5: Plan initiation through the AWS CLI or API, with optional CloudWatch alarm triggers and a manual approval step
Note: While plans can be built and run through the AWS console (useful during initial testing), the best practice for real DR invocations is to use the Region Switch API or AWS CLI directly. This reduces cross-service dependencies in your critical path and makes sure your recovery execution doesn’t rely on console availability.
2. Database failover. The first execution block initiates the database transition, promoting the secondary Region’s database to become the primary writer. ARC Region Switch supports both switchover and failover for Amazon Aurora Global Database, RDS instances with cross-Region read replicas, and Amazon DocumentDB (with MongoDB compatibility) clusters. For databases beyond these, such as Amazon DynamoDB global tables, execution blocks can invoke custom Lambda functions to handle service-specific failover logic.
3. Application stack scales up. Subsequent execution blocks handle application infrastructure: scaling compute resources in the standby Region (EKS pods, ECS tasks, or EC2 Auto Scaling groups) and invoking Lambda functions for application-specific logic, such as warming application-level caches so the standby Region doesn’t serve its first requests from a cold cache. This makes sure the standby Region is fully prepared to handle production traffic before any DNS changes occur.
4. Route 53 health check manipulation. After the application stack is ready, ARC Region Switch marks the primary Region’s health check as unhealthy through the native Route 53 health check block, or flips state through an ARC routing control block, triggering Route 53’s failover record logic and redirecting traffic to the now-ready standby Region. You can monitor the current status on the Region Switch plan’s monitoring tab.
Completing the failover: With the health check marked unhealthy, Route 53’s failover mechanism activates. After DNS time-to-live (TTL) values expire, the ingress proxy fleet’s queries resolve to the standby Region’s endpoint. The infrastructure team never needs to reconfigure proxies, update routing rules, or modify DNS records. The application team controls traffic flow through the PHZ record’s health check state by using ARC.
Failback: Returning traffic to the original primary Region works exactly the same way: you trigger your failback plan. Because the design is symmetric, the plan scales the original Region, transitions databases back, and redirects traffic through the same orchestrated flow, in the opposite direction. Failback is as automated, tested, and repeatable as the failover itself.
From independent to coordinated recovery across the portfolio
The pattern described in the preceding section gives each application team independent failover capability. In practice, however, applications rarely exist in isolation. A payments service depends on an authentication service, and a customer portal depends on an API gateway. When interdependent applications need to recover, they must do so in dependency order. Without a mechanism to coordinate this sequence, the answer today is humans in a war room making phone calls.
Most enterprises capture these dependencies in IT service management (ITSM) tools and wikis, but documentation isn’t orchestration. The worst moment to discover that your dependency maps are stale or your runbooks don’t work is in the middle of a major incident.
Federated recovery with AWS RAM-shared plans
With AWS RAM, you can share Region Switch plans across AWS accounts. Each application team shares its plan from its spoke account with a central hub account, where it’s added as a child plan of a parent plan. Using the Region Switch plan execution block, the parent plan runs each child plan as a step in its workflow. This creates explicit, executable dependency chains between independently owned recovery plans.
Application Team A and Application Team B both share their plans with the hub. The parent plan runs Team A’s plan first, and only when it completes successfully does it run Team B’s plan, so Team B fails over only after its upstream dependencies are ready to serve requests in the recovery Region.
Figure 6: A central parent plan runs each team’s shared child plan in dependency order
These chains can extend across the portfolio (A, then B, then C, with independent applications running in parallel), modeling the full dependency graph as ordered steps in a single parent plan. Each child plan remains independently owned, versioned, and testable by its respective team. The AWS RAM share governs only the hub’s access to the plan, not the plan’s contents or execution logic.
Portfolio-level recovery
The result is deterministic, dependency-ordered portfolio recovery initiated by a single action. A central function triggers the parent plan, and the child plans run in the order it defines. Each child plan can be tested independently by its owning team, and the full parent plan exercised end-to-end during planned DR events, proving coordinated recovery works before it’s needed under pressure.
This addresses a question that enterprises across regulated industries are increasingly asking: “How do we press a big red button to reach our minimum viable business?” A single monolithic recovery script cannot work, because no central team understands every application’s recovery requirements. Parent and child plans solve this: each team owns their recovery logic and maintains it as part of their software development lifecycle (SDLC), keeping it in step with the application it protects, while the parent plan runs them in the correct sequence.
Post-recovery workflows: What happens after failover
Failover isn’t the end of the story. It’s the beginning of recovery operations. After traffic is successfully serving from the new active Region, teams need to run post-recovery workflows to stabilize the environment, validate health, and prepare for eventual failback. Region Switch supports these through additional execution blocks that run after the primary failover sequence completes.
Validation and health confirmation
Application health checks – Invoke Lambda functions that run synthetic transactions against the newly active stack, verifying API response codes and end-to-end data flow. This measures real recovery time: not when the plan finished, but when requests actually started succeeding again.
Data integrity verification – Validate that database promotion completed cleanly, check replication lag resolution, and confirm write availability.
Dependency validation – Verify that downstream services and third-party integrations are responding correctly from the new Region.
Operational stabilization
Scaling adjustments – Fine-tune Auto Scaling desired counts, EKS Horizontal Pod Autoscaler (HPA) targets, or ECS replica counts based on actual traffic patterns.
Alerting reconfiguration – Update CloudWatch alarm thresholds and notification routing to reflect the new active Region.
Standby preparation – Begin provisioning the former primary Region as the new standby.
Notification and compliance reporting
For regulated enterprises, post-recovery documentation is as important as the failover itself:
Automated notifications – Trigger SNS notifications or Lambda functions that update incident management systems with completion status and duration metrics.
Compliance evidence – Produce audit-ready logs documenting the failover sequence: what ran, when, which resources were affected, and total recovery time achieved.
Stakeholder communication – Automatically distribute status updates confirming successful recovery.
Preparing for failback
Post-recovery workflows also set the stage for an eventual return to the original primary Region:
Replication re-establishment – Confirm that database replication is flowing in the new direction (from new primary to new standby) and monitor replication lag.
Standby readiness verification – Confirm the former primary Region is healthy and scaled to accept traffic, so it’s ready when failback is needed.
Failback plan validation – Check the plan evaluation status of your failback plan. Region Switch evaluates it automatically several times an hour, so any warnings about permissions, resources, or capacity are visible before you need to return traffic.
Because these post-recovery steps are defined as execution blocks within the same Region Switch plan, they benefit from the same automation, visibility, and IaC management as the failover itself, with no separate runbooks or manual checklists required. The entire lifecycle, from initial trigger through failover, validation, stabilization, and failback preparation, is captured in a single, version-controlled, repeatable plan.
Conclusion: Reconciling security and autonomy in enterprise cloud architecture
In regulated industries, the tension between centralized security controls and application team autonomy isn’t solved by choosing one over the other. It’s an architectural challenge that demands both. Defense-in-depth strategies with centralized ingress controls are non-negotiable. But modern cloud operations require that teams accountable for availability can act independently when it matters most.
This architectural pattern using Amazon Application Recovery Controller Region Switch delivers the following:
Application team autonomy – Teams run DR tests, respond to incidents, and meet RTO commitments independently, with no tickets, coordination meetings, or waiting. Application teams control their recovery orchestration while operating within the guardrails of the shared responsibility model.
Preserved security posture – Centralized ingress controls remain unchanged. The proxy fleet continues enforcing authentication, authorization, threat detection, and audit logging. Application teams never need access to proxy configurations. They only need to control their own PHZ records.
Automated coordination – Route 53, VPC Lattice, and downstream infrastructure react to ARC-orchestrated changes without manual intervention. Traffic shifts by changing health check state rather than editing DNS records, so recovery relies on highly available data plane operations instead of control plane APIs, in line with Well-Architected guidance. The shared responsibility boundary is enforced through IAM policies and VPC associations, not manual processes.
Federated portfolio recovery – A central parent plan runs each team’s shared child plan in dependency order, initiated by a single action. Each team retains ownership of their recovery logic while Region Switch coordinates execution across organizational boundaries. Where human oversight is required, a manual approval execution block can gate any step, so automation never removes a team’s control over its own failover.
Because ARC Region Switch is fully managed and highly available, with its data plane operating independently in each Region, application teams can trust that their failover orchestrator is operational precisely when they need it most. And because plans are defined declaratively and can be managed as IaC, teams avoid the maintenance burden of keeping custom scripts synchronized across Regions.
For organizations operating under strict compliance frameworks, this pattern offers a practical blueprint for modernizing disaster recovery without compromising security controls. You don’t need to choose between security and agility. You can architect for both.
Getting started
If your organization operates a similar centralized ingress architecture and wants to support self-service failover:
Start small: Create your first ARC Region Switch plan in the visual Workflow Builder for a single non-critical service.
Define as IaC: Export and manage your plan with AWS CloudFormation or the AWS CLI to integrate with your deployment pipelines.
Set up automatic triggers: Configure CloudWatch alarms to initiate plans automatically, reducing your effective RTO.
Validate incrementally: Use VPC Lattice (or per-service ALB/NLB records) to test failover for individual services before expanding to full-stack DR exercises.
Measure and iterate: Track failover duration, identify bottlenecks, and refine your execution blocks.
Validating a streaming pipeline beyond current peak takes production-scale test data, and you can get it by multiplying pre-recorded streams with Amazon Managed Service for Apache Flink. You typically observe backpressure, growing checkpoint durations, and shard hot-spotting when you test under sustained load. Reproducing them in a pre-production environment takes a data source that matches both the volume and the statistical characteristics of production traffic. Customer compliance teams often decide against replaying real production data directly because they want to avoid exposing sensitive fields. Existing approaches to generating that test load each solve for part of this problem while leaving a gap. As a result, teams either test at a fraction of expected load and discover the limits in production, or maintain a bespoke generator whose output gradually drifts from real production distributions.
With the stream multiplier pattern, a relatively small sample from Amazon Kinesis Data Streams is enough. With appropriate configuration, the pattern can help preserve the statistical properties of real traffic. You control output volume through a single parameter, multiple.factor.
This post covers why at-scale testing is difficult, where existing approaches fall short, how the multiplier works, and how it performs.
Understanding streaming workloads and the need for at-scale testing
Streaming applications process high-throughput data continuously. In production, they ingest several thousands to millions of events per minute and maintain stateful computations across operators. They write results to downstream sinks without interruption.
You typically see their critical failure modes under sustained, high-volume load. Checkpoint stalls, growing checkpoint durations, state backend pressure, application restarts, and state loss are problems that compound over time and under volume. Persistent high utilization of streaming resources does the same. A pipeline that runs cleanly at a fraction of its target throughput provides no evidence that it will behave the same way at full production rate.
The remaining question is where the test data comes from.
Challenges with existing approaches to at-scale testing of streaming applications
One approach to testing at production scale is to replay a full-volume pre-recorded stream. You capture live traffic from a production Kinesis data stream into a non-production stream over a defined window. You then replay it through your pipeline under test at the original rate. The data and field distributions are real, so you can compare the results with production behavior. However, at 50,000 records per second, a two-hour recording window holds 360 million records (50,000 × 7,200 seconds). You would need to capture and retain a recording at every throughput tier you validate. That becomes an operational and cost burden as the target volume grows.
Synthetic data generators take the opposite approach. Tools such as DataFaker or custom Kinesis producers remove the dependency on pre-recorded data. They emit records at any target rate from a schema definition, bounded only by the producer’s compute capacity. The limitation lies in data fidelity: generated records follow configured distributions, not observed ones.
Generators can model individual field distributions and known correlations. They can’t reproduce the natural skew, micro-bursts, and cross-field dependencies of real user behavior.
For pipelines whose correctness depends on realistic data, that gap can mask the exact problems the test is designed to expose.
A third option is to validate directly against live production traffic using canary deployments, shadow pipelines, or percentage-based traffic splitting. The data is real because it is production data, but you are limited to current production throughput. Testing at 5x or 10x that rate means waiting for organic traffic growth or building a separate load injection path into production.
Each of these approaches solves for either fidelity or volume, but not both at the target scale. The stream multiplier pattern addresses that gap.
Solution overview
Consider a ride-hailing company, AnyCompany Rides, that runs a real-time analytics pipeline on Apache Flink. The pipeline ingests trip events from a Kinesis data stream, computes fare estimates, driver matching scores, and surge pricing signals, and writes the results to a downstream Kinesis data stream. It processes approximately 15,000 events per second at peak. The team expects a regional expansion to increase peak traffic by 5x. Before launch, it needs to validate that the pipeline sustains 75,000 events per second without checkpoint degradation or backpressure.
The team has a ten-hour, sanitized recording of production traffic captured at 15,000 events per second. Replaying it at the original rate only validates current load. The team can’t capture a recording at 75,000 events per second because that traffic doesn’t exist yet. A synthetic generator that reproduces the trip types, fare ranges, and geographic patterns of the original stream would take significant modeling effort, with no guarantee of accuracy.
Using the stream multiplier pattern, the team sets a multiplication factor of 5x on the existing recording. For each event, the multiplier emits the original unchanged plus four synthetic variants. Each variant replaces the fields you configure with DataFaker values and preserves business keys such as trip ID. The output stream delivers approximately 75,000 events per second, and fields you don’t configure for replacement keep the distributions of the original recording.
The following diagram illustrates this architecture.
Figure 1: Stream multiplier solution architecture
The solution uses three AWS services and one open source library:
Amazon Kinesis Data Streams holds the pre-recorded data stream on the source side, and carries the amplified stream to the pipeline under test on the sink side.
Amazon Managed Service for Apache Flink runs the multiplier, a RichFlatMapFunction that emits the original record plus N−1 variants.
Amazon CloudWatch collects the metrics that tell you whether the generator itself is healthy: throughput, backpressure, checkpoint duration, and heap.
DataFaker, an open source Java library for generating realistic synthetic values, produces the replacement values for mutated fields, driven by expressions you declare in faker-config.yaml rather than in code. It is declared as a build dependency and packaged into the application JAR, so there’s nothing to install separately.
The pattern is schema-agnostic: it operates on whichever fields you nominate as keys and whichever you nominate as mutable. In this sample, changing those fields is a configuration change. Adopting a different schema also requires updating the event model and the JSON serializers.
Prerequisites to deploy the stream multiplier
An AWS account with permissions to create Managed Service for Apache Flink applications, Kinesis data streams, Amazon Simple Storage Service (Amazon S3) buckets, and Amazon CloudWatch log groups.
A local build environment with Java Development Kit (JDK) 17 and sbt 1.9.9.
A source Kinesis data stream holding the pre-recorded data, and a sink stream sized for source throughput × multiplication factor.
An S3 bucket for the packaged application JAR.
AWS Command Line Interface (AWS CLI) version 2, configured.
An AWS Identity and Access Management (IAM) execution role with read access to the source stream and write access to the sink stream.
We tested the sample against these versions. The build.sbt file declares everything except the build tools, and sbt assembly bundles it into the JAR.
JDK : 17.
sbt : 1.9.9.
Scala : 2.12.20.
Apache Flink : 2.3.0.
Amazon Managed Service for Apache Flink runtime environment : FLINK-2_3.
The Kinesis connector version is paired to the Flink version (6.0.1-2.0 targets Flink 2.x), and Flink 2.x requires Java 17 and uses the Java DataStream APIs, because the Scala DataStream APIs are no longer available.
A note on source data: This pattern doesn’t perform data masking, anonymization, or removal of personally identifiable information (PII). Fields that you don’t list in updatable.fields pass through unchanged. You are responsible for using only open data, or data that you sanitized, as the pre-recorded data stream. The walkthrough uses the public NYC taxi trip schema, which has the same shape as the AnyCompany Rides trip events.
Deploy the solution
The sample-datastream-multiplier repository includes an AWS CloudFormation template that creates the Kinesis data streams, the Managed Service for Apache Flink application, the IAM role, and the CloudWatch log group. It also creates an AWS Key Management Service (AWS KMS) key for log encryption and an AWS Lambda function that seeds the source stream with sample records.
Build the uber-JAR with sbt assembly and upload it to your S3 bucket.
Deploy infrastructure/cfn-template.yaml as a stack named data-stream-multiplier with aws cloudformation deploy, passing S3BucketArn and S3Key, and set MultipleFactor, KeyFields, and UpdatableFields as stack parameters.
Start the application with aws kinesisanalyticsv2 start-application.
Begin at a factor of 2x or 3x, confirm the output is correct, then raise the factor while monitoring CPU and backpressure.
The template encrypts both Kinesis data streams at rest with the AWS managed key (alias/aws/kinesis), and the CloudWatch log group with a customer managed AWS KMS key. To use your own key, pass EncryptionKmsKeyArn. If you reuse an existing source stream (CreateInputStream=false), you’re responsible for confirming that it has server-side encryption enabled. Amazon S3 encrypts the application JAR with SSE-S3 by default, and the AWS SDK uses TLS in transit.
Two ways to produce the stream
The multiplier reads from one of two sources, which you select with the source.type runtime property, and both feed the same multiplication operator. Amplification mode (source.type=kinesis) is the focus of this post. It replays a pre-recorded data stream, so fields you don’t replace keep the distributions of real traffic. Throughput is the source rate multiplied by multiple.factor. Pure synthetic mode (source.type=datagen) uses the Apache Flink DataGen connector to generate records in the application from faker-config.yaml, with no real data involved. It is useful for validating the pipeline end to end before you have a pre-recorded stream.
How it works
In amplification mode, the multiplier processes each incoming event in five steps:
Read and deserialize the event from the source Kinesis data stream.
Emit the original event unchanged.
Compute a deterministic hash from the configured key fields combined with a per-iteration random seed.
Generate synthetic values for the fields listed in updatable.fields, using that hash as the DataFaker seed and the expressions in faker-config.yaml. The key fields stay unchanged, so variants keep the same keys as the original.
Serialize and write all records, original plus variants, to the sink stream.
Managed Service for Apache Flink application runtime properties drive the behavior, so the same artifact serves both modes regardless of schema:
Property
Purpose
source.type
kinesis for amplification mode, datagen for pure synthetic mode
multiple.factor
Total records emitted per input record: 1 original plus N−1 variants
key.fields
Business identifiers preserved across variants, and hashed to seed generation
updatable.fields
Fields replaced with synthetic values in each variant
random.seeds
One seed per multiplication iteration, so variants differ from each other
rate.limit.per.second
Caps generated throughput in pure synthetic mode
The multiplication operator
The fan-out itself is deliberately small. A trait defines the contract, and flatMap is designed to emit the original followed by N−1 variants:
override def flatMap(value: IN, out: Collector[OUT]): Unit = {
if (isValidEvent(value)) {
emitOriginalEvent(value, out)
for (i <- 1 until multiplier) {
emitMultipliedEvent(value, out, i)
}
}
}
A multiplication factor of 3 produces three records: the original and two variants. The application validates the factor at startup and rejects values outside a range of 1 to 100. A mistyped factor can generate enough output to exhaust the sink’s shard capacity or the application’s heap. Bounding it in code helps prevent a configuration error from overwhelming the test environment.
Field generation with DataFaker
You define which fields become synthetic, and how, outside the code. Each entry pairs a DataFaker expression with a type:
With key.fields=tripId and five updatable.fields, one input trip fans out into variants that share its identity but differ in the nominated fields. Three of the ten variants of a single trip, read back off the amplified stream during a 10x run:
{"tripId":8801,"timestamp":1789627092733,"fromLocationName":"Brooklyn - Williamsburg","toLocationName":"Manhattan - Central Park","distance":18.9,"amount":22.75,"category":"1"}
{"tripId":8801,"timestamp":1789627092733,"fromLocationName":"Bronx - Yankee Stadium","toLocationName":"Upper West Side","distance":0.1,"amount":69.85,"category":"2"}
{"tripId":8801,"timestamp":1789627092733,"fromLocationName":"Tribeca","toLocationName":"Tribeca","distance":4.29,"amount":128.88,"category":"3"}
tripId is identical on every variant because it is a key field, and timestamp because it appears in neither list and passes through untouched. The five updatable.fields differ in each. A consumer that groups by tripId still sees a coherent entity, because the key travels with every variant regardless of which shard carries it.
Figure 2: Inside the multiplier: the key field and untouched fields are identical on every variant, while the nominated fields are regenerated from a per-variant seed
The choice of updatable.fields controls the trade-off between variant diversity and preservation of production distributions.
A repeatable, deterministic generator
If the generator produces different data on every run, a change in throughput or checkpoint duration between runs tells you nothing. The multiplier is therefore designed to be deterministic: the same pre-recorded data stream and the same configuration generate the same variant values on every run. After a failure and restore, replayed records regenerate the values they had before. The at-least-once Kinesis sink can still write some of them twice. Each variant’s values come from a seed built from the record’s key fields and a per-iteration seed from random.seeds, so variants are reproducible but still differ from one another. Because the generator is reseeded for every record, the values don’t depend on replay order after a restore or on which subtask processes the record. That holds only while every generated value comes from the seeded generator. Values such as wall-clock time or UUID.randomUUID() would break it.
What this looks like in practice
This section shows how the multiplier performed in our tests: first its throughput ceiling and resource use across factors, then how the sink’s shard count changes what you measure.
We ran a factor sweep on Managed Service for Apache Flink in the US West (Oregon) AWS Region (us-west-2). We deliberately over-provisioned the sink so that the multiplier itself was the only limit. The application ran at parallelism 2 with automatic scaling off, on 3 Kinesis Processing Units (KPUs): 2 for processing and 1 for orchestration. Records used the NYC taxi trip schema, averaging about 181 bytes, with key.fields=tripId, five updatable.fields, and one seed per variant. To hold the input rate constant, we used the pure synthetic (datagen) source with rate.limit.per.second and scaled the sink to 240 shards. Per-variant field generation, the compute-intensive path, runs the same code in both modes. Your results will vary with record size, schema, and configuration.
Throughput and capacity
Factor
Input rec/s
Output rec/s
Output per processing KPU
5x
~25,600
~128,000
~64,000
10x
~14,300
~143,000
~71,500
15x
~11,300
~169,000
~84,500
20x
~8,600
~171,000
~85,500
25x
~7,040
~176,000
~88,000
Stability and resource utilization
Factor
CPU % avg/max
Backpressure
Checkpoint duration
Checkpoint size
Heap % avg/max
5x
99 / 100
0
~460 ms
~17 KB
31 / 63
10x
96 / 100
0
~272 ms
~73 KB
34 / 65
15x
99.6 / 100
0
~258 ms
~46 KB
35 / 59
20x
99.7 / 100
0
~226 ms
~57 KB
38 / 66
25x
99.7 / 100
0
~333 ms
~57 KB
36 / 63
Figure 3: Output ceiling rises toward an asymptote as the factor increases, while the input rate the application can consume falls
Output throughput rises with the factor. Between 5x and 25x in our sweep, output climbed from approximately 128,000 to approximately 176,000 records per second while the input rate the application consumed fell from approximately 25,600 to approximately 7,040. Each input record carries a fixed cost to read and deserialize, and each output record carries the per-variant field generation cost. Raising the factor amortizes the fixed cost over more outputs, so output flattens near approximately 176,000 records per second.
CPU is the binding constraint, not memory and not state. CPU sits at 96–100 percent in every row, so in this configuration these plateaus are the multiplier’s ceiling. Heap stays in a narrow 31–38 percent average band with no upward trend. The job turns on Flink object reuse and reseeds one DataFaker instance per subtask instead of creating one per record. Checkpoints stay tiny, 17–73 KB completing in 226–460 ms, because the stateless flatMap has nothing to snapshot beyond source offsets.
For capacity planning, size from output per processing KPU: roughly 64,000 records per second at 5x, rising to approximately 88,000 at high factors.
Per-factor CloudWatch views show output rate in blue and sink throttling in red. The flat red line at zero indicates each plateau is compute-bound rather than sink-limited:
Figure 4: At 5x, output plateaus at ~128,000 records per second with zero sink throttling
Figure 5: CPU pegged near 100% while heap stays comfortable. This is the signature of a compute-bound plateau
Your sink decides your throughput, not the multiplier
That ceiling is only reachable if the sink can absorb it. Holding the factor at 10x and changing nothing but the sink’s shard count shows where the real limit usually sits:
Sink shards
Sink capacity rec/s
Observed output rec/s
Throttling
CPU %
Limited by
20
20,000
~20,505
heavy
~16
sink
60
60,000
~62,000–70,000
heavy
~48
sink
120
120,000
~122,000–123,000
mild
~86
sink (marginal)
180
180,000
~143,000
none
~96
compute (ceiling)
At 20 shards the application ran at 14 percent of its capability with CPU at 16 percent, waiting on the sink. Only at 180 shards does throttling disappear and CPU saturate, so you are measuring the multiplier rather than your Kinesis provisioning.
A shard accepts 1,000 records or 1 MB per second on writes, whichever limit you reach first. For records under about 1 KB, like the approximately 181-byte trip events here, the record limit applies first. The sink needs roughly target output rate ÷ 1,000 shards before any factor comparison is meaningful. For larger records, size by bytes: one shard per MBps of target output. Under-provision it and every factor reports the same throughput: the shard limit.
The async KinesisStreamsSink re-queues and retries throttled records. Across our sweep, transient per-attempt failures stayed at 73 or fewer in a window against tens of millions of records, and we observed no permanent loss. Write latency held at approximately 40–54 ms average with occasional approximately 1.1 s spikes.
Shard load also stays even as the factor rises, because the partition key comes from the record’s hash rather than the multiplication factor. In our 25x run across 240 open shards, every shard carried between 671 and 678 records per second, within about 0.5 percent of the approximately 675 average.
Validating a run
Before you trust the load test, confirm that the multiplier is doing what you expect:
Measure throughput on the Kinesis side, not the application side. Flink chains the source, the multiplier, and the sink into a single operator. As a result, the application-level numRecordsInPerSecond and numRecordsOutPerSecond don’t cleanly separate input from output. Use GetRecords.Records on the source stream and IncomingRecords on the sink stream. Their ratio is the true fan-out and should equal your multiplication factor. In our 10x amplification run, it measured 2,056 records per second in against 20,505 out, a ratio of 9.97.
Key preservation. Sample the sink stream and confirm that variants share key-field values and that fields outside both lists are unchanged.
Troubleshooting
If the amplified stream isn’t what you expect, check these first:
All variants of a record are identical. random.seeds has fewer entries than the multiplication factor requires.
Output rate is below factor × input rate. Check WriteProvisionedThroughputExceeded on the sink and backPressuredTimeMsPerSecond on the operator, and provision at least target output rate ÷ 1,000 sink shards for records under about 1 KB.
The source falls behind. Watch millisBehindLatest on the application or GetRecords.IteratorAgeMilliseconds on the source stream. If the lag climbs during a live run, add source shards or use Enhanced Fan-Out and watch SubscribeToShardEvent.MillisBehindLatest instead.
The application crash-loops after switching source.type. Switching modes changes the job graph, so restoring the previous snapshot fails with a KinesisShardSplitSerializer EOFException. Start the application with SKIP_RESTORE_FROM_SNAPSHOT. The multiplier is stateless, so there is no application state to restore. Property changes, including source.type, take effect only after an application update.
Performance considerations and limitations
The multiplier inherits the timing of the pre-recorded data rather than synthesizing it. Amplification multiplies volume and keeps the sample’s timing shape, so bursts in the pre-recorded data carry through at higher amplitude. Injecting bursts that are not in the sample, spreading variants over an interval rather than emitting them together, and simulating late arrival all need extra logic. The sample uses WatermarkStrategy.noWatermarks() because the multiplier has no event-time operators. Flink also doesn’t write watermarks into Kinesis records, so the watermark setting in the multiplier has no effect on the pipeline under test. Each variant keeps the original timestamp in its payload. The pipeline under test assigns its own watermarks from that field, as it does in production.
Generated fields are independent of each other. Each entry in faker-config.yaml is evaluated on its own, so in the trip example, distance is drawn from 0–20 and amount from 5–150 with no relationship between them. A 0.5-mile trip can be assigned a $150 fare. Ordering is preserved within a partition but not across partitions, so if the pipeline under test depends on global ordering, the amplified stream won’t provide it.
The sample assumes that the source and sink streams are in the same AWS account. For a stream in another account, grant the application’s execution role access with a resource-based policy on that stream. The sample also reads and writes JSON. Because the multiplier works on deserialized objects, supporting Apache Avro or Protocol Buffers means replacing JsonDeserializationSchema and JsonSerializationSchema. The multiplication logic is unchanged.
Cost considerations
You pay for Kinesis shard-hours, PUT payload units, and extended retention, and for Managed Service for Apache Flink KPU-hours and running application storage. Amazon CloudWatch Logs and AWS KMS add smaller charges. The following estimates are approximate and use on-demand prices in US West (Oregon) as of October 2026. Prices vary by AWS Region and change over time, so check Amazon Kinesis Data Streams pricing, Amazon Managed Service for Apache Flink pricing, or the AWS Pricing Calculator for your Region.
Default deployment. The template runs three shards with seven-day retention and 3 KPUs, two for processing and one for orchestration. That comes to about $0.45 per hour, or about $11 per day if you leave it running, plus PUT payload units for the records you write. Automatic scaling is on by default and can add KPUs under load.
The 25x run in this post. Two hours at 25x, with 240 sink shards and about 176,000 records per second, uses about 1.3 billion PUT payload units ($17.75), 480 sink shard-hours ($7.20), and 6 KPU-hours ($0.66), for about $26. Seven-day retention adds about $9.60, for about $35. Lower factors write fewer records and cost less.
To keep costs down, set StreamRetentionHours to 24 unless you plan to replay the amplified stream. Size the sink to your target output rate, keep test windows short, and follow the clean-up steps as soon as you finish.
Clean up
To avoid ongoing charges, stop the application, wait until it reports READY, then delete the stack:
Deleting the stack removes the Kinesis data streams, the application, the IAM role, the producer Lambda function, and the log group. If the stack created the AWS KMS key, AWS CloudFormation schedules it for deletion after a 30-day waiting period. Delete the JAR from your S3 bucket separately.
Conclusion
Validating a streaming pipeline at production scale requires a data source that delivers both realistic data and sustained volume. With the stream multiplier pattern, real distributions come from the pre-recorded capture, and you control volume through a single parameter, multiple.factor, rather than through storage. To get started, clone the sample-datastream-multiplier repository and follow the steps in Deploy the solution, starting at a factor of 2x or 3x. For more on building and running applications on the service, see the Amazon Managed Service for Apache Flink Developer Guide.
Mike Lothian owns a docking station (or dock) that uses a DisplayLink chipset
that had, at least in his opinion, subpar performance on Linux. He
decided to try improving the Linux driver, with LLM assistance, but ran into one
problem that he could not solve—so he asked for help, hoping that
Synaptics’s developers would lend a hand. (DisplayLink was acquired by
Synaptics in 2022.) Rather than help him in improving the
driver, however, the Synaptics legal team responded with patent infringement
claims, a demand
that he cease unauthorized implementation of DisplayLink technology, and threats
of expensive legal penalties. For now, it appears that Linux users of the
company’s docks will have to continue to live without feature parity with other
platforms.
Version
160 of the Firefox web browser has been released. Notable features in this
release include the ability to block sites from asking to handle
“mailto:” links, ability to play Matroska files with FLAC, MP3, or PCM
audio, playback speed controls in the picture-in-picture
video player, and a new Mini
Window feature that is in early development.
It’s a bad one, and it’s been in the code for a decade:
A newly disclosed flaw involving OpenSSH and musl, dubbed “*sshole,” allows attackers to log in to certain Alpine Linux and other musl-based Linux accounts configured to block password authentication simply by entering an oversized password.
If you run data pipelines, you know how much work they can take to set up, sometimes only to break without you realizing. That doesn’t even include data from sources too small to justify a project of their own, such as instrument exports, partner CSV files, and database tables. Each one can take an engineer days to map, code, test, and provision, and the work needs to be repeated when the schema changes.
Agentic Data Ingestion, the solution referenced in this post, automates that work with agentic AI. You point it at a file or a database table, and foundation models (FMs) propose the field mapping and write the transformation code. The agent tests that code on Apache Spark before it deploys the code as an AWS Glue job, and you approve or reject the result at the mapping or code testing step. In our test deployment, a run at sample scale costs about USD $0.10–0.25, and the idle deployment about USD $15–20 a month. For more information, see Cost to run the solution.
To see it in action before you deploy, try the interactive demo.
In this solution, the FMs run on Amazon Bedrock, which is a fully managed service that provides foundation models, and the agents, built with the open-source Strands Agents SDK, run on Amazon Bedrock AgentCore, which hosts and operates AI agents. AWS Step Functions, which provides serverless workflow orchestration, manages each run, and AWS Glue, which provides serverless data integration capabilities, runs the generated jobs. You deploy the whole solution with the AWS Cloud Development Kit (AWS CDK). Before you begin, review the Prerequisites.
Solution overview
The following diagram shows the architecture.
Figure 1: Solution architecture, from sign-in and the data sources through the agents and Step Functions orchestration to the AWS Glue job and its targets
File validation – Check format and size from object metadata.
Schema extraction – Infer field names and types.
Schema comparison – Detect drift against the target schema.
Schema mapping – An FM maps source to target fields with confidence scores. (Approval gate).
Code generation – Generate the PySpark transform and data quality rules.
Code testing – Run the transform on Spark and check its output. (Approval gate, plus a forced review if tests keep failing).
Infrastructure provisioning – Create the target if it doesn’t exist.
AWS Glue job creation – Create the job with a validated configuration.
Quality checks – Record which data quality rules the job will enforce.
Run pipeline – Run the AWS Glue job.
The data quality rules, for AWS Glue Data Quality, a capability of AWS Glue, come from the approved mappings and run inside the job, which quarantines failing rows before it writes. They’re on by default for file uploads into Amazon S3 Tables, a capability of Amazon S3, which stores Apache Iceberg tables, and optional for the other targets.
Figure 2: The 10-step workflow. Steps 1 and 2 depend on the source type, approval gates follow steps 4 and 6, a failing test returns to step 5, and the job writes to one of three target types
Supported sources and targets
Source
Details
Amazon S3 upload
CSV (including TSV and pipe-delimited), JSON, Parquet, Avro, ORC, and Excel (.xlsx, one worksheet per run). Each file starts its own run.
JDBC database
PostgreSQL, Amazon Aurora, MySQL, MariaDB, Oracle, or SQL Server through the AWS Glue Java Database Connectivity (JDBC) connector, on demand or on a schedule. Incremental loads read only rows newer than the last run, keyed on a timestamp or numeric watermark column.
How we made generated code reviewable and repeatable
In our very first attempt, we asked the model to create the whole AWS Glue script, more than 500 lines on each run. Reviewing that each time wasn’t sustainable. It failed for random reasons, and because the model is non-deterministic, the next run could fail somewhere new no matter how much we optimized the prompt. To solve this, we made the job as deterministic as we could. A human-written/reviewed Jinja2 template for each target type now supplies the scaffold, and the FM writes only the transformation block, typically 5–15 lines for these samples, which is checked against an allowlist of permitted constructs before the job is created.
In our next iteration, these jobs failed inconsistently with GlueArgumentError because different parts of the code used different names for the same job arguments. We made the code itself consistent, but not how the arguments were named. The solution parses the parameters that the rendered code reads with getResolvedOptions, resolves a value for each, and refuses to create the job if one is missing.
Figure 3: Template-first code generation. The model writes only the transform block, and the job’s parameters are validated before the job is created
When we first started testing, we translated the PySpark into pandas, which we thought would simplify testing. But this approach couldn’t express everything PySpark can. It failed inconsistently and could never guarantee the script would work, which defeats the purpose of testing. So now the transform runs in an AWS Glue interactive session on the same Spark version as the job, against a 100-row sample of the source file. Its output is checked for record counts, nulls, and schema compliance. Database sources are tested on rows generated from the schema instead, because the test session in the workflow has no access to your databases.
Human approval gates
To review the results before continuing while you build trust in the pipeline, two gates pause a normal run and wait for approval, one after schema mapping and one after the code passes its tests. A mapping skips review only when each field pair is high-confidence and corroborated by a name match, and a reused mapping goes to review by default.
The gates fail closed, so if the workflow can’t tell whether a step needs review, it waits for a person. A run left waiting at a gate for 72 hours fails, and approvals and rejections are recorded with the reviewer’s identity. Two agents can help along the way. When mapping confidence is low, a complex-mapping agent adds sample values and specific questions to the mapping review. When an AWS Glue job fails, a self-healing agent starts one new run, which regenerates the code after a schema or type error or uses a larger worker after an out-of-memory error. It skips the retry when a second run could duplicate data the failed job already wrote, or when a retry can’t fix the error, such as a permissions problem. The run then stays failed, with the agent’s diagnosis on its detail page. For the full rules, see Approval Gates and the agents guide.
Customer perspective: Montai Therapeutics
Montai Therapeutics applies the same pattern to scientific data. Rather than hand-writing a parser for each new assay format, the team embedded an agent that generates the parser into the ingestion application they already had.
“We have implemented agentic parser generation to ingest semi-structured or variably formatted scientific data into our schema-controlled system of record. Embedding the parser function generation agent into our existing ingestion app has brought lead time to onboard and deploy new assays and protocols to near zero (5 to 10 minutes for parser generation and output checking on an agent runtime), where before there could be days to weeks of developer time spent scripting bespoke parsing functions that may only be used a few times.
The biggest business impact has been the proportion of experimental data that now lands in our data lake rather than in people’s inboxes, which will funnel into improving our drug specificity and efficacy predictive models. Time spent by scientists troubleshooting data formatting issues is now relegated to solving the scientific problems at the core of our business strategy.”
— Kevin Klicki, Senior Engineer – Bioinformatics, Montai Therapeutics
Security considerations
Generated code runs in an isolated test session under a read-only role before an AWS Glue job exists. The solution creates dedicated AWS Identity and Access Management (IAM) roles for pipeline jobs, scoped to that pipeline and bounded by a permissions boundary, with separate roles for agent runtimes. Data is encrypted at rest and in transit, and JDBC credentials are referenced only by their Secrets Manager Amazon Resource Name (ARN), so they don’t appear in job arguments.
You sign in through Amazon Cognito, with multi-factor authentication (MFA) required by default. The dashboard is served through Amazon CloudFront behind an AWS WAF web access control list (web ACL) with managed rules and a rate limit. A second web ACL admits only API requests that come through CloudFront. By default, signed-in users have full dashboard access. The solution doesn’t include identity federation or role-based access, which the README’s Hardening for real data section lists as next steps for production.
Cost to run the solution
We measured these figures on our test deployment in the US West (Oregon) AWS Region (us-west-2) with the repository’s sample files, which have seven or eight columns, and priced the usage at public list prices.
Idle, the solution costs about USD $15–20 per month. About USD $14 is the two AWS WAF web ACLs and their rules, up to USD $5 is Amazon CloudWatch monitoring, depending on your free tier, and storage and one secret add about USD 1. Everything else bills per use.
A run at sample scale costs about USD $0.10–0.25:
Component
Grows with
Per run, measured
Basis
Amazon Bedrock, Claude Sonnet 4.6
Number of fields and model calls
USD $0.05–0.14
About 10,500 tokens over three model calls on a clean run, up to 32,000 with regeneration, at USD $3.30 and USD $16.50 per million input and output tokens (us. inference profile)
AWS Glue interactive session (testing)
Test attempts
USD $0.01–0.02 per attempt
Two G.1X workers at USD 0.44 per data processing unit (DPU) hour
Amazon Bedrock AgentCore runtime at USD $0.0895 per vCPU hour and USD $0.00945 per GB hour
The AI cost depends on the number of fields rather than rows, because prompts carry schemas and at most 100 sample rows. A scheduled pipeline repeats most AI steps on each run and skips only the mapping call when the schema is unchanged. We haven’t yet measured a wider source or a lower-cost model such as Amazon Nova Pro. For current prices, see Amazon Bedrock, AWS Glue, Amazon Bedrock AgentCore, and AWS WAF pricing.
Prerequisites
You don’t need prior experience with Amazon Bedrock AgentCore or Strands Agents. You need the following:
An AWS account, with credentials that can create IAM roles and the solution’s resources in your Region and in US East (N. Virginia) (us-east-1). The AWS WAF stack deploys to us-east-1 because CloudFront requires its web ACL there, so service control policies (SCPs) that restrict Regions must allow it.
Node.js 20 or later, Python 3.12 or later, the AWS Command Line Interface (AWS CLI), and Docker. The deployment script installs the AWS CDK CLI for you.
A host that can build linux/arm64 images, because Amazon Bedrock AgentCore runtime supports only ARM64 containers, with about 100 GB of free disk, mostly for the Docker images. Apple silicon or an AWS Graviton instance such as a t4g.large with a 100 GB gp3 volume builds them natively (see the EC2 build host guide). An x86 host needs QEMU emulation, which ./deploy.sh --check detects. The browser-based AWS CloudShell can’t build the project, because it has no Docker daemon.
An authenticator app for MFA.
Note: This solution uses the us.anthropic.claude-sonnet-4-6 cross-Region inference profile, which is available only from US AWS Regions. If you deploy outside the US, update the model ID in the configuration before deploying.
We tested the solution in us-west-2. Other Regions need the same Amazon Bedrock models and inference profiles, AgentCore, AWS Glue 5.0, and, for S3 Tables targets, Amazon S3 Tables.
Deploy the solution
The deployment script checks the prerequisites, builds the container images, bootstraps both Regions, and deploys eight AWS CloudFormation stacks.
Note: Stack names end in -dev. If you deploy with ./deploy.sh --environment <name>, use -<name> in the commands that follow.
To deploy the solution:
Clone the repository:
git clone https://github.com/aws-samples/sample-agentic-data-ingestion.git
cd sample-agentic-data-ingestion
(Optional) Check the prerequisites without changing anything. The script prints All prerequisite checks passed. or lists each problem with its fix.
./deploy.sh --check
Deploy the solution. A first deployment can take up to roughly 20 minutes on a t4g.large, mostly to build container images.
./deploy.sh
Confirm that the script exits without an error and its last line reads All 8 stacks present and healthy. If a stack is MISSING or failed, check its Events tab in the AWS CloudFormation console, fix the error, and run ./deploy.sh again.
Run the remaining commands in a shell where AWS_PROFILE and AWS_REGION are still set. Self-sign-up is disabled, so you create the first user yourself.
To create your first user and sign in:
Create a user with a random temporary password, replacing <your-email-address> with your email address. The temporary password expires after 24 hours.
aws cloudformation describe-stacks --stack-name AgenticDataIngestion-Hosting-dev \
--query "Stacks[0].Outputs[?OutputKey=='CloudFrontUrl'].OutputValue" --output text
Open the dashboard URL and sign in with your email address and the temporary password. Set your own password, then enroll your authenticator app.
Enable the Claude model
The AI steps call the default model, Claude Sonnet 4.6, which needs a one-time setup per account. The first call to the model enables it, and that call needs AWS Marketplace permissions that the solution’s roles deliberately don’t have, so you make it with your own credentials.
To enable the Claude model:
Submit Anthropic’s one-time use case details, either by choosing an Anthropic model in the model catalog of the Amazon Bedrock console or with the PutUseCaseForModelAccess API. For details, see Request access to models.
Confirm that you get a text reply. If you get AccessDeniedException right after submitting the use case details, wait a few minutes and try again.
Without this setup, the deployment still works, but the AI steps fail with AccessDeniedException.
Run a pipeline
Next, you create a pipeline that loads a sample customer file into a new Amazon S3 Tables table, then review the run. With the sample file, a run takes roughly 5–15 minutes plus the time you spend at the gates. The following interactive demo shows the same steps, and Human approval gates describes what each gate checks.
On the dashboard, choose Create Pipeline. On Name & Description, enter a Pipeline Name, and choose Next.
On Source, for Source Type, choose S3 Upload, and choose Next.
On Target Destination, for Target Type, choose S3 Table Bucket, and then choose Create new.
Enter a Bucket name, Namespace, and Table name. The bucket name must be unique in your account and Region (see the naming rules). For Write Mode, keep the default, Append, and choose Next.
On Target Schema, for Schema Source, choose Upload, then use Choose file to select tests/test-data/target_schema.json from your local clone of the repository. Choose Next.
On AI Model & Options, for AI Model, choose Claude Sonnet 4.6. Keep the approval and data quality settings at their defaults, and choose Next.
On Review & Create, check the settings, and choose Create Pipeline.
To run and review the pipeline:
On the pipeline’s overview page, choose Upload File.
Select tests/test-data/customer_data_4.csv from your local clone, and choose Upload. The upload starts a run. The file’s column names differ from the target schema (for example, email_address instead of email), which gives the mapping step something to decide.
When the run pauses at a gate, choose Approvals in the navigation bar.
Choose Review next to the pending approval, check the mapping or the code, optionally add a comment, and choose Approve or Reject. Repeat steps 3 and 4 for each gate the run reaches.
When the run completes, open it from Recent Runs on the pipeline’s overview page, and review the Step Flow, Code, and Data Quality tabs.
Note: The mapping review lists each field pair with its Confidence, and the code review shows the transform with its Test Report.
To read from a database instead, choose JDBC Database as the Source Type. You need a Secrets Manager secret with username and password keys, an Amazon S3 gateway endpoint in the database’s subnet, and a security group that allows traffic from itself on the database port. The JDBC source guide covers the details.
Clean up
Delete your pipelines first, while the dashboard is still deployed.
To clean up the deployed resources:
For each pipeline, choose Archive Pipeline on its overview page. Then choose Archived in the navigation bar, choose Delete next to the pipeline, and confirm with Continue and Permanently Delete. This removes the pipeline’s AWS Glue job, rules, and uploaded files, and for a database source the AWS Glue connections and crawler it created.
From the repository root, list what the teardown script would delete. Without --force, it deletes nothing.
Delete the stacks. If you deployed with --environment, pass the same option. The script also deletes the DynamoDB tables the Data stack retains and the project’s IAM roles and policies across the account, because IAM is global.
Delete what your pipelines created at run time: the S3 Tables resources, which hold the ingested data and bill for storage, and any AWS Glue databases or secrets left with the project’s name prefix. Find the table bucket ARN with aws s3tables list-table-buckets, and the leftover names with the two list commands.
The CDK bootstrap stack, CDKToolkit, stays in both Regions. Delete it only if nothing else in the account deploys with CDK.
Conclusion
The approval gates are where this gets interesting in practice. The mapping review in particular surfaces assumptions about your data that are worth catching before a job runs in production. We’ve seen how the model sometimes picks a reasonable mapping that’s technically wrong for the business context, which is exactly why the human review step exists. We’ve only measured our own files and sources so far. If you run it on a wide source, say, 100 columns from a production instrument export, we’d genuinely like to know whether the mapping and costs held up.
It’s a reference solution, so the next step is adding your own sources, targets, data quality rules, approval policies, and production identity setup.
Connecting enterprise systems to modern analytics and AI platforms often requires building brittle pipelines, duplicating data, or sacrificing the governance and business context that makes the data trustworthy. Today we’re introducing the integration of Workday Data Cloud with AWS. With this integration you get bi-directional, zero-copy access to Workday’s governed people and money data from the AWS analytics and AI services that you already use.
Workday Data Cloud overview
Workday Data Cloud is an open, secure, and headless data platform that makes Workday’s people and money data available in the tools where developers build, with its semantic definitions, permission models, metric logic, and audit controls intact. It provides a curated, unified view of people and money data through the Workday Data Lake, bi-directional zero-copy access to that data through Workday Data Connect using Apache Iceberg, and near real-time SQL access for applications and AI agents through Workday Live Data Query.
AWS analytics overview
AWS provides a comprehensive set of analytics and AI services that work together for data engineering, analytics, and machine learning. Here’s how the key services come together in the context of the Workday Data Cloud integration:
Amazon SageMaker Unified Studio
Amazon SageMaker Unified Studio (SMUS) is a unified development environment for data analytics and AI. It brings together data access, querying, visualization, and notebook-based development in one place. With SMUS, developers can connect to Workday Data Cloud, explore datasets interactively, run SQL queries, and build machine learning models, all without switching between multiple tools or interfaces.
Amazon Athena
Amazon Athena is a serverless, interactive query service to analyze data directly in Amazon Simple Storage Service (Amazon S3) using standard SQL. With the Workday Data Cloud integration, analysts and developers can run ad hoc queries against Workday data stored as Iceberg tables without needing to set up or manage infrastructure.
Amazon Redshift
Amazon Redshift is a cloud data warehouse that delivers fast query performance on structured and semi-structured data using SQL. Organizations already using Amazon Redshift for their analytics workloads can now incorporate Workday people and money data through the Glue Data Catalog federation for cross-domain analytics without data movement.
AWS Glue Data Catalog
AWS Glue Data Catalog serves as the central metadata repository that makes Workday data discoverable across the supported AWS analytics services. Through the integration, Workday’s Iceberg REST Catalog is registered as a federated catalog in AWS Glue, which means any AWS service that reads from the AWS Glue Data Catalog such as Amazon Athena, Amazon Redshift, Amazon EMR, and Amazon SageMaker AI can automatically discover and query Workday datasets without additional configuration.
Glue Data Catalog also works in the other direction, letting Workday query data that lives in AWS. Customers can register the Iceberg REST Catalog endpoint of the AWS Glue Data Catalog as an external catalog in Workday Data Cloud. Workday users can then browse and query Apache Iceberg tables stored in Amazon S3 in near real-time, without copying the data into Workday.
AWS Lake Formation
With AWS Lake Formation, you can set up and manage a secure data lake. In this integration, AWS Lake Formation provides the fine-grained permission layer that governs who can access Workday data within the AWS environment, complementing the governance that Workday already applies at its data layer. Administrators can scope each analyst or application to specific databases, tables, rows, or columns.
Amazon S3
Amazon S3 is the foundational storage layer for data lakes on AWS. In the context of this integration, Amazon S3 stores Apache Iceberg tables that can be queried from Workday, so organizations can query and analyze data in AWS alongside their Workday data.
Workday Data Cloud and AWS integration overview
The integration between Workday Data Cloud and AWS enables bi-directional, zero-copy data access. This means data doesn’t need to be copied or replicated between Workday and AWS. Instead, each platform can query the other’s data in place, respecting existing governance and security controls on both sides.
Accessing data in AWS from Workday
Workday applications and AI agents can access data stored in AWS, enriching HR and finance workflows with external business data. Developers building agents or analytics within the Workday platform can query Apache Iceberg tables stored on Amazon S3 by registering the Iceberg REST Catalog endpoint of the AWS Glue Data Catalog as an external catalog in Workday Data Cloud.
Zero-copy federation
Workday Data Cloud federates to the AWS Glue Data Catalog using OAuth 2.0 authentication. After registration, Workday users can discover and query Iceberg tables backed by AWS directly from within Workday Data Cloud without copying data. AWS Lake Formation defines which databases and tables the Workday connection can read, whereas Workday’s Configurable Security controls which individual Workday principals can access the federated tables. This means an AI agent handling workforce planning in Workday can pull in revenue forecasts from an Iceberg table on AWS. Combining those financial projections with headcount data gives HR planners hiring recommendations to review, accounting for both business growth and budget constraints.
Accessing data in Workday from AWS
AWS analytics and AI services can access Workday’s governed people and money data. There are two primary patterns for AWS to query Workday data, each optimized for different use cases:
Live Data Query
Some use cases require near real-time access to Workday data, such as an AI agent checking current headcount or a dashboard showing today’s financial close status. For these, developers can use Workday Live Data Query. From Amazon SageMaker Unified Studio, Amazon Athena, Amazon EMR, or AWS Glue, users connect to Workday through the Live Data Query connector and execute SQL queries directly against live Workday datasets. The query runs within Workday’s governed environment and results flow back in near real-time. This is ideal for operational AI agents, near real-time dashboards, and any application where data freshness is critical.
Figure 1: Querying live Workday supplier data from an Amazon SageMaker Unified Studio notebook with Workday Live Data Query
Zero-copy federation with Apache Iceberg
For near real-time analytics, machine learning (ML) training, and large-scale data exploration, developers access Workday data through Workday Data Connect’s Apache Iceberg integration. By registering the Workday Iceberg REST Catalog in AWS Glue Data Catalog or AWS Lake Formation, all Workday datasets appear as standard Iceberg tables discoverable by any AWS analytics service. The compute runs on the customer’s AWS environment (using Athena, Amazon Redshift, Amazon EMR, or SageMaker AI) while the data stays in Workday Data Cloud.
Figure 2: Querying a Workday Data Connect Iceberg table with standard SQL in the Amazon Athena query editor
Figure 3: Querying a Workday Data Connect Iceberg table with standard SQL in Amazon Redshift Query Editor v2.
Figure 4: Browsing Workday Iceberg tables with PySpark in a SageMaker Unified Studio notebook using Workday Data Connect
What this means for customers
This integration changes how organizations work with their people and money data for analytics and AI. Here’s what it unlocks:
Organizations no longer need to build and maintain complex extract, transform, and load (ETL) pipelines to get Workday data into their analytics environment, which removes data silos without duplicating data. The data stays where it lives, governed and accurate, while remaining fully accessible for analysis and AI.
When AI agents and models can access Workday data with its full business context, developers spend less time reverse-engineering business logic and rebuilding permission models. That context includes who has access to what, what a metric actually means, and how calculations are defined.
Workday’s permission model, audit controls, and business logic travel with the data, and AWS Lake Formation adds governance on the AWS side. The result is data access that is both open and controlled, which is exactly what regulated industries and enterprise security teams require.
Instead of building pipelines, monitoring data freshness, and reconciling duplicates, data teams can focus on building analytics and machine learning applications that drive business outcomes.
Use case walkthrough: AI-powered workforce analytics
Consider a Chief People Officer who wants to understand the relationship between employee engagement scores, compensation benchmarks, and attrition patterns across the organization. Today, this requires extracting data from Workday, loading it into a separate analytics platform, joining it with market data, and building custom reports. This process takes weeks and produces stale results.
With the Workday Data Cloud and AWS integration, a data team can perform the following steps:
Connect to Workday Data Cloud from AWS Glue using the pre-built Workday connector.
Explore workforce datasets such as compensation, tenure, engagement, and performance directly, with full semantic context (job levels, org hierarchies, and tenure calculations) already defined.
Join Workday people data with market compensation benchmarks and recruiting pipeline data stored as Iceberg tables on Amazon S3, all through standard SQL in Athena or SageMaker AI notebooks.
Build a machine learning model in SageMaker AI that predicts attrition risk based on the combined dataset.
Deploy an AI agent that flags high-risk attrition patterns for HR business partners to review and act on. The agent uses Live Data Query for near real-time signals and batch Iceberg access for historical trend analysis.
The entire workflow runs without any data leaving its governed environment, without any custom pipelines to maintain, and with full auditability at every step.
Getting started
The Workday Data Cloud integration with AWS is generally available.
Amazon SageMaker Unified Studio users can connect to Workday Data Cloud directly through the Workday Live Data Query connector or the Workday Iceberg REST Catalog connector.
Workday Live Data Query connections can also be created in the AWS Glue console, and users can query live Workday data from Amazon Athena, Amazon EMR, and AWS Glue ETL jobs.
AWS Glue and AWS Lake Formation users can register the Workday Iceberg REST Catalog as a federated catalog to make Workday datasets discoverable across AWS analytics services.
Workday users seeking to query AWS data can register their Glue Data Catalog Iceberg REST endpoint as an external catalog in Workday Data Cloud.
For deeper technical walkthroughs and step-by-step configuration guides, refer to the following resources:
This integration represents a new model for enterprise data access, where governed data flows freely between platforms without compromising security, accuracy, or business context. We are excited to see what customers build with Workday Data Cloud and AWS together.
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
![Diagram of one input record becoming three. tripId and timestamp stay the same in every output, while the multiplier regenerates location, distance, amount, and category per variant from the key-field hash plus random.seeds[i].](https://d2908q01vomqb2.cloudfront.net/b6692ea5df920cad691c20319a6fffd7a4a766b8/2026/11/02/BDB-6228-2-1.png)
