Tag Archives: Thought Leadership

Scale distributed AI workloads with Ray and Anyscale on Amazon EC2

Post Syndicated from Sindhura Palakodety original https://aws.amazon.com/blogs/architecture/scale-distributed-ai-workloads-with-ray-and-anyscale-on-amazon-ec2/

If you are building multimodal artificial intelligence and machine learning (AI/ML) pipelines on Amazon Elastic Compute Cloud (Amazon EC2) that process text, video, and audio, you have likely hit at least one of these challenges. Scaling from a single instance to distributed compute without rewriting code, managing costs across training, fine-tuning, and inference, and keeping up with an evolving network of models, frameworks, and hardware are among the biggest challenges AI engineers face today. These challenges compound when each pipeline stage has distinct compute, networking, and storage requirements, and multiple teams own separate parts of the workflow.

This post shows how the Ray-powered Anyscale application on AWS helps you tackle these challenges. You will learn how to: deploy Ray across three AWS compute options based on your operational requirements, optimize costs by matching heterogeneous instance types to each pipeline stage, and scale from prototype to production without code changes.

How Anyscale builds on Ray

Ray Core is the distributed computing foundation underlying the entire framework. It provides primitives for parallelizing Python workloads across clusters, handling task scheduling, fault tolerance, and autoscaling. Ray AI libraries are built on top of Ray Core. As a result, custom workloads that don’t fit a standard pattern can still use the same distributed infrastructure.

Ray Data ingests and transforms datasets at scale, feeding processed data into training, tuning, and inference stages. Ray Train orchestrates distributed multi-node jobs with built-in fault tolerance, supporting foundation model fine-tuning across architectures such as Llama and Mistral with automatic selection of data-parallel, model-parallel, or pipeline-parallel strategies. Ray Tune parallelizes hyperparameter experiments across large clusters. Ray Serve runs production inference endpoints with dynamic batching, automatic scaling, and multi-model deployment support. Ray RLlib supports distributed reinforcement learning workloads, including reinforcement learning from human feedback (RLHF) for large language model (LLM) alignment.

Anyscale on AWS

Many AWS customers already run self-managed Ray clusters on Amazon EC2. Anyscale builds on that foundation as a fully managed service that deploys within your AWS account, integrating with your existing Amazon Virtual Private Cloud (Amazon VPC) networks, AWS Identity and Access Management (IAM) roles, and compliance frameworks. The application handles cluster management, autoscaling, and cost controls so you can focus on workloads rather than infrastructure.

AWS and Anyscale together address three dimensions of production AI/ML: cost-efficient infrastructure spanning the full Amazon EC2 portfolio, a Ray-on-Anyscale stack that scales reliably from experimentation to production, and an integrated framework that reduces friction at every stage of development. With Anyscale, you can map each pipeline stage to the appropriate instance type. Intel and AMD-based instances handle preprocessing, tokenization, and feature extraction. AWS Graviton-based instances cover preprocessing, worker nodes, embedding generation, and lightweight inference where price-performance matters. NVIDIA GPU instances and AWS Trainium handle training and fine-tuning. AWS Inferentia serves high-throughput inference endpoints. Amazon EC2 Spot Instances run automatically for batch training, preprocessing, and hyperparameter sweeps.

Anyscale deployment architecture on AWS

Three deployment options provide flexibility based on operational requirements:

Amazon EC2 with Elastic Fabric Adapter (EFA) provides direct compute control and high-performance inter-node networking.

Amazon Elastic Kubernetes Service (Amazon EKS) with EFA supports containerized workloads.

Amazon SageMaker HyperPod delivers managed large-scale foundation model training with built-in cluster resiliency.

Regardless of the deployment path, Ray libraries handle distinct pipeline stages. Ray Data orchestrates preprocessing across CPU-only instances, AWS Graviton-based instances, GPU instances, and Spot Instances. Ray Train, Ray Serve, and Ray RLlib handle distributed training, inference serving, and reinforcement learning on NVIDIA GPU instances and AWS AI chips.

Reference architecture showing the Anyscale on AWS solution deployed across multiple AWS services

Figure 1: Anyscale on AWS architecture

Anyscale runs a hosted control plane that sends instructions to an operator deployed in your AWS environment. The data plane (Amazon EC2 instances, storage, and networking) stays in your account, and Anyscale manages Ray cluster lifecycle without direct access to your data.

Option A: Amazon EC2

Amazon EC2 supports the full pipeline without constraint. This makes it the recommended default for teams running EC2-based workloads who want to scale into AI/ML without retooling their operations. Ray Data runs preprocessing on cost-efficient instance types including Spot Instances and Graviton processors. Ray Train handles distributed training on GPU and Trainium instances with EFA. Ray Tune runs hyperparameter search across CPU and GPU instances. Ray Serve deploys inference on Inferentia or Graviton. Ray supports reinforcement learning through RLlib on GPU instances, and RLHF workflows run through frameworks such as OpenRLHF.

Anyscale manages instance lifecycle, provisioning and terminating nodes based on demand. Amazon Simple Storage Service (Amazon S3) handles datasets, checkpoints, and model artifacts. Amazon FSx for Lustre provides parallel file system throughput for training jobs with repeated large-dataset reads. EFA is available on supported instance types for multi-node training.

Option B: Amazon EKS

Amazon EKS runs the same full pipeline within a Kubernetes environment. This makes it the right choice if your organization already manages Kubernetes and wants Ray to share the same infrastructure and tooling. The Anyscale operator deploys Ray nodes as pods, and Ray libraries run identically to the Amazon EC2 path at the application layer.

Karpenter handles node autoscaling, provisioning GPU nodes for training, Graviton nodes for preprocessing and inference, and Spot Instances for batch workloads. Ray Serve benefits from Kubernetes-integrated zero-downtime upgrades for production inference deployments. The tradeoff: Amazon EKS adds container orchestration complexity and requires custom configuration for the Anyscale operator.

Option C: Amazon SageMaker HyperPod

Amazon SageMaker HyperPod is purpose-built for resilience-critical stages where multi-week runs on large GPU or Trainium clusters make node failures expensive. It continuously monitors cluster health, automatically replaces unhealthy nodes, and resumes from the last checkpoint without manual intervention. HyperPod supports both Slurm-orchestrated and Amazon EKS-based clusters, and now supports the full model development lifecycle including inference deployment. Amazon SageMaker Studio integration adds experiment tracking and model management.

Native integrations shared by EC2, EKS, and HyperPod

Networking and storage

EFA provides low-latency, high-bandwidth inter-node communication for distributed training on all three deployment paths. Amazon S3 is the primary store for datasets, checkpoints, and artifacts. Ray Data pipelines read directly from Amazon S3-based data lakes. Amazon FSx for Lustre integrates with S3 for parallel file system throughput. Amazon Elastic File System (Amazon EFS) covers shared file access for workspaces and notebooks.

Security and compliance

Amazon Virtual Private Cloud (Amazon VPC) network isolation, IAM role-based access control, and AWS Key Management Service (AWS KMS) encryption at rest and in transit apply regardless of deployment path. Data stays within your AWS account and encryption keys remain under your control. This architecture supports compliance with frameworks including HIPAA and SOC 2 without changes to Ray workload code.

Observability

Anyscale monitoring provides workload-specific dashboards covering training progress, pipeline throughput, and request latency. Grafana dashboards are accessible from the Anyscale console. Amazon CloudWatch handles AWS metrics, logs, and alarms. Prometheus supports custom metrics collection for teams with existing monitoring pipelines.

Each deployment path follows the same workflow: Ray Data feeds processed data to Anyscale Runtime, Ray Train orchestrates training, Ray Serve handles inference through Amazon API Gateway, and Ray RLlib supports reinforcement learning.

Cost optimization and provisioning

Right-sizing compute per stage

AWS Graviton-based instances deliver strong price-performance for CPU-bound Ray workloads across preprocessing, embedding generation, and lightweight inference. PyTorch inference on Graviton3 C7g instances shows up to 50% cost savings versus x86 equivalents (Tomaz Berisa, n.d.). Graviton4 adds 30% better compute per core and 75% more memory bandwidth than Graviton3 (AWS, n.d.). Graviton also runs small language models (1–7B parameters) cost-efficiently for generative AI inference.

Intel-based instances provide strong x86 price-performance for tokenization, batch transforms, and workloads requiring broad framework compatibility. AMD-based instances deliver strong performance for CPU-intensive preprocessing, feature engineering, and inference workloads requiring maximum throughput per core. NVIDIA GPU instances provide high throughput for distributed training, fine-tuning, and large-model inference.

AWS Trainium2 delivers 30–40% better price-performance than comparable GPU instances for training at scale (AWS, n.d.). AWS Inferentia2 provides up to 4x higher throughput and 10x lower latency than its predecessor for production inference. Ray auto-detects accelerator types, so Anyscale routes each workload stage to the appropriate hardware without manual configuration.

Spot Instances with Ray fault tolerance

Anyscale keeps head nodes on On-Demand and runs workers on Spot with automatic fallback. Ray Data retries failed tasks on replacement nodes. Ray Train resumes from checkpoints, achieving up to 60% savings on distributed training. Ray Serve proactively migrates replicas on the 2-minute preemption warning.

Capacity Blocks and On-Demand Capacity Reservations

For scheduled training runs requiring capacity availability, Amazon EC2 Capacity Blocks reserve 1–64 accelerated instances for up to six months, bookable eight weeks in advance. Each block supports up to 512 GPUs or 1,024 Trainium chips, placed in Amazon EC2 UltraClusters with petabit-scale networking. Anyscale machine pools reference capacity reservations directly with priority-ordered fallback: Capacity Block, then Spot, then On-Demand. On-Demand Capacity Reservations (ODCRs) provide open-ended capacity availability.

Combined provisioning strategy

Anyscale runs two-layer autoscaling: Ray Serve scales replicas on request metrics while the cluster autoscaler provisions and terminates nodes. The recommended purchasing stack starts with AWS Compute Savings Plans, saving up to 66% across Graviton, GPU, Trainium, and Inferentia. Layer on Spot for fault-tolerant batch jobs, Capacity Blocks for scheduled training, and On-Demand for burst.

Getting started: From prototype to production

Your path follows four phases, each building on the last without code changes:

Phase 1: Validate locally (1–2 days). Start with Ray on a single Amazon EC2 instance or your laptop. Build your data pipeline and training loop, and validate model quality before touching distributed infrastructure.

Phase 2: Scale data (1–2 weeks). Deploy Ray Data with AWS Graviton and Amazon EC2 Spot Instances to process your full dataset at scale, with Spot Instances reducing preprocessing costs by up to 90% (AWS, n.d.).

Phase 3: Scale training (2–4 weeks). Choose your deployment option: Amazon EC2 for maximum flexibility, Amazon EKS for Kubernetes-native environments, or Amazon SageMaker HyperPod for resilience-critical runs. Scale training across multiple GPUs or AWS Trainium instances with EFA.

Phase 4: Deploy inference (1–2 weeks). Deploy Ray Serve for model serving with autoscaling configured to your traffic patterns.

The same code runs across all four phases. Moving from a single instance to a multi-node cluster requires only configuration changes managed by Anyscale.

Conclusion

Ray and Anyscale on AWS provide a practical path from prototype to production across the full AI/ML pipeline. Three deployment options address different operational requirements: Amazon EC2 for maximum flexibility, Amazon EKS for Kubernetes-native environments, and Amazon SageMaker HyperPod for resilience-critical training. With heterogeneous clusters mixing Amazon EC2 Spot Instances, AWS Graviton, NVIDIA GPUs, AWS Trainium, and AWS Inferentia, you can match compute to each pipeline stage. The same Ray code runs from a single instance to a multi-node cluster without modification, and Anyscale handles cluster management, fault tolerance, autoscaling, and resource orchestration.

Your next steps depend on where you are today:

  • Already running Ray on AWS – deploy Anyscale through the AWS Marketplace for managed autoscaling, cost controls, and governance.
  • New to distributed ML – start with the Anyscale documentation on a single Amazon EC2 instance, then scale out.
  • Evaluating Anyscale – explore the Ray quickstart for architecture guides and tutorials.

For more on the AWS services covered in this post, see the documentation for Amazon EC2, Amazon EKS, Amazon SageMaker HyperPod, AWS Trainium, AWS Inferentia, and AWS Graviton.


About the authors

Sindhura Palakodety

Sindhura Palakodety

Sindhura is a Senior Solutions Architect at Amazon Web Services (AWS) and Single-Threaded Leader (STL) for ISV Generative AI, where she is dedicated to empowering customers in developing enterprise-scale, Well-Architected solutions. She specializes in generative AI and data analytics domains, enabling organizations to leverage innovative technologies for transformative business outcomes.

Kinnar Sen

Kinnar Sen

Kinnar Sen is a Principal Specialist Solutions Architect at AWS for the Amazon EC2 portfolio. He has more than 20 years of industry experience in research, consultancy, engineering, and architecture, including 15 years in cloud computing. He works with AWS customers worldwide to design, migrate, and scale production workloads on EC2. Kinnar leads EC2 technical strategy for generative AI, agentic, and data analytics workloads, with a focus on price performance. He sets long-term technical direction across specialist and field teams, works with EC2 product and engineering leaders to shape what AWS builds next, and mentors the next generation of builders.

Arnab Sinha

Arnab Sinha

Arnab is a Senior Solutions Architect at Amazon Web Services (AWS), where he helps independent software vendors (ISVs) and startups build world-class solutions on AWS. He has deep expertise in cloud infrastructure, machine learning (ML), and artificial intelligence (AI), and he advises organizations on how to architect, optimize, and scale their workloads on AWS to drive business outcomes.

A coordination framework for formal proof development with Kiro

Post Syndicated from Haoyu Li original https://aws.amazon.com/blogs/devops/a-coordination-framework-for-formal-proof-development-with-kiro/

Large language models (LLMs) and coding agents are increasingly capable of proving difficult mathematical theorems and completing formally verified Lean programs. Yet, existing agentic provers rely on the backend models’ capabilities while lacking the ability to explore when they get stuck: when a proof resembles patterns the model already knows, it might succeed quickly. When it doesn’t, the model can spend thousands of tokens repeating failed tactics, lose track of intermediate results, or abandon the proof altogether.

We develop a prover harness, MAESTRO, in Kiro CLI that gives LLMs a systematic framework for exploring, decomposing, and making structured progress on difficult problems. The prover constructs a shared proof skeleton and organizes the work across specialized agents that explore independent proof obligations concurrently. A persistent coordinator evaluates their feedback, repairs broken proof structure, and decomposes difficult goals into smaller lemmas when direct attempts fail. This turns proof generation into an iterative process of planning, proving, diagnosing, and revising.

We test on a series of challenging benchmark datasets, including ArxivLean (research-level mathematical theorem proving) and FormalQualBench (graduate-level mathematical theorem proving). Across these, Kiro CLI with GPT and Claude models consistently solves more hard problems than Codex and Claude Code with the corresponding models. We apply Kiro CLI to prove the six problems from the 2026 International Mathematical Olympiad (IMO, held July 15–16, 2026). Kiro CLI solves all six problems in 1 hour 46 minutes. Previously reported total times were 8 hours 18 minutes for Humanfia and 24 hours 56 minutes for AxiomProver. We also find that Kiro CLI can discover and test novel proof strategies, suggesting potential applications in mathematical research.

Kiro CLI strengthens base LLMs’ proving capabilities

We evaluate Kiro CLI across several established benchmarks and compare it against plain agent harnesses, Codex and Claude Code, using the same underlying base models. These benchmarks are:

  • PutnamBench, formalized undergraduate Putnam competition problems.
  • FormalQualBench, graduate qualifying-examination problems.
  • ArxivLean, theorem statements drawn from recent research papers.
  • Neural Theorem Proving for Verification Conditions (NTP4VC), verification conditions extracted from real-world software-verification projects.

Together, they span diverse levels and domains of formal reasoning. Across this spectrum, Kiro CLI consistently achieves higher solve rates, with the gap becoming particularly pronounced on the harder research-level problems. This suggests that the gains are not simply because of stronger base models. The specialized coordination machinery in MAESTRO makes a substantial difference in how effectively those models can solve difficult mathematics.

Benchmark Number of tasks GPT 5.6 Sol Claude Opus 5
Kiro CLI Codex Improvement Kiro CLI Claude Code Improvement
PutnamBench 672 690 656 +14 671 645 +26
NTP4VC 600 599 588 +11 600 589 +11
ArxivLean – June 48 33 24 +9 37 17 +20
FormalQualBench 23 14 11 +3 16 9 +7

We evaluate Kiro CLI across five reasoning-effort levels for GPT 5.6 Sol and Claude Opus 5 on a dataset comprising six challenging IMO 2026 problems, formalized by AxiomMath. With GPT 5.6 Sol, Kiro CLI solves five of the six problems at low effort and all six at medium effort or higher. By comparison, Codex solves none at low effort, three at medium, five at high, and all six at the two highest levels. With Claude Opus 5, Kiro CLI solves all six problems at every effort level. Claude Code solves three at low and medium effort and five at each higher level. These results demonstrate that our harness substantially strengthens the base models’ theorem-proving capabilities.

Solve rates for Kiro CLI, Codex, and Claude Code across five reasoning-effort levels on the IMO 2026 problems

Figure 1: Solve rates across five reasoning-effort levels on the IMO 2026 problems

Kiro CLI outperforms existing custom solutions

We compare Kiro CLI with existing proving systems on the IMO 2026 dataset. Across all six IMO 2026 problems, Kiro CLI completes the proofs in 1 hour 46 minutes. Compared with 8 hours 18 minutes for Humanfia, 8 hours 57 minutes for Pramaana, and 24 hours 56 minutes for AxiomProver, Kiro CLI is 4.7×, 5.1×, and 14.1× faster, respectively. The advantage is consistent across the individual problems, suggesting that the coordination framework improves not only success rate, but also the efficiency with which difficult proofs are found.

Proof-completion times for Kiro CLI compared with Humanfia, Pramaana, and AxiomProver on the IMO 2026 problems

Figure 2: Proof-completion times for Kiro CLI vs. existing custom solutions on IMO 2026

The core of the Kiro CLI prover: Coordination

The core of the Kiro CLI prover harness is coordination. Rather than treating theorem proving as a collection of independent lemma-solving jobs, it maintains a single evolving proof plan. Prover agents work on individual obligations and return checked lemmas, counterexamples, partial proofs, and structured reports about what is or isn’t working. A persistent coordinator reads this feedback together with the current Lean development and continuously revises the global proof architecture. It can preserve useful results from a failed branch and repair broken interfaces between lemmas. When a helper is false, it retracts both the helper and the downstream structure that depends on it. The coordinator can also introduce new abstractions, redirect workers, or leave productive branches alone. This allows local discoveries to influence the entire proof rather than being lost inside isolated attempts. In particular, the prover can recognize a flawed decomposition and restructure the proof around new evidence. Moreover, it can discover novel mathematical arguments.

Example 1: Repairing the proof architecture

We see this behavior in an Opus 5 run on the Quillen–Suslin theorem from FormalQualBench. At one point, a worker proposed simplifying the proof by keeping track of only part of the algebraic structure after a localization step. A later worker found a concrete counterexample showing that this simplification had thrown away information that was essential to completing the proof. In fact, the proposed intermediate lemma was stronger than what the original theorem required and would imply a substantially harder open statement.

Rather than repeatedly trying to prove the bad lemma, the coordinator recognized that the proof had been decomposed incorrectly. It recorded the counterexample, rejected later proposals that made the same mistake, and redesigned the intermediate interface so that both complementary pieces of the structure were preserved. After this information was carried through the remaining steps, the two pieces could be recombined into a global basis and the proof completed.

Example 2: Discovering novel proof strategies

Coordination can also lead Kiro CLI to discover new mathematical constructions. One example comes from the ArxivLean theorem Real Matrices as Linear Combinations of Three Orthogonal Matrices.

Both the original paper and our prover begin with the same standard simplification: reduce the matrix problem to a problem about its singular values. From there, however, the proofs diverge. The original paper first solves the easier case in which all singular values are distinct. When some values are repeated, it slightly perturbs the matrix to remove these repetitions, applies the distinct-value construction, and then uses a limiting argument to recover the original matrix. Our prover instead found a direct construction that works even when singular values are repeated. It groups the singular values into overlapping two-dimensional blocks and chooses intermediate values so that neighboring blocks fit together consistently. These local constructions then telescope into three global orthogonal matrices whose linear combination gives the original matrix.

Thus, it doesn’t merely translate the argument from the paper into Lean. It introduces different intermediate objects and reorganizes the proof around them, producing a route that directly handles the case for which the original proof uses perturbation and compactness.

Conclusions

Proof development isn’t only a collection of frontier model workers working individually. What matters is how we organize the work. MAESTRO in Kiro CLI is built around this observation: individual provers explore locally, while a persistent coordinator accumulates evidence and continuously reshapes the global proof. Through this coordination-heavy design, the prover has reached nearly saturating performance on several benchmarks of competition-level formal mathematics, while also showing promising results on substantially harder, research-level problems. You can find more technical details about MAESTRO in our paper, Maestro: Agentic Formal Proof Development by Coordination.


About the authors

Haoyu Li

Haoyu Li

Haoyu is a Ph.D. student in Computer Science at the University of Illinois Urbana-Champaign, where he researches techniques and algorithms for formal verification. His research focuses on trustworthy machine learning, AI safety, and formal methods. Haoyu was an intern at AWS AI Labs. Learn more at haoyuli02.github.io.

Zeren Shui

Zeren Shui

Zeren is an Applied Scientist at AWS AI Labs. His research interests span a broad spectrum of deep learning, large language models, agentic AI with a focus on the development of coding agents. Zeren holds a Ph.D. in computer science from University of Minnesota – Twin Cities.

Yuhao Zhang

Yuhao Zhang

Yuhao is a Senior Applied Scientist at AWS AI Labs, where he researches techniques and algorithms that increase the trustworthiness of AI coding agent. His primary research focuses on the fields of software engineering, programming languages, and artificial intelligence. Learn more at: https://foreverzyh.github.io/.

Ferhat Erata

Ferhat Erata

Ferhat is an Applied Scientist at AWS AI Labs. He combines large language models with formal reasoning to improve the safety and security of generative and agentic AI. His work includes autoformalization, verified code generation, and training reasoning models with verifier feedback. His offensive security research includes reverse engineering quantum circuits from power traces and detecting side-channel vulnerabilities in post-quantum cryptographic code. Over more than a decade in industry before his doctoral studies, he co-founded UNIT Research & Development, led the development of ModelWriter, a platform connecting text documents with formal models, and coordinated national consortia for EUREKA/ITEA industrial research projects. Ferhat holds a PhD in computer science from Yale University. His website is https://ferhat.ai/.

Murali Krishna Ramanathan

Murali Krishna Ramanathan

Murali is a Senior Principal Scientist at Amazon Web Services (AWS), where he works on AI-powered developer tools. Previously at Uber, he was the architect of Piranha, an open-source tool that automates the removal of stale feature-flag code, built before large language models made automated code transformation mainstream. Earlier, as a faculty member at the Indian Institute of Science (IISc), Bangalore, he conducted research in program analysis and automated testing, work recognized with a Google Faculty Research Award and ISSTA Distinguished Paper Awards. He holds a Ph.D. in Computer Science from Purdue University.

Hao Zhou

Hao Zhou

Hao is a Senior Applied Scientist at AWS AI Labs. His research interest is in machine learning, optimization, and big data analysis. He currently works on LLM training, inference, and agents, with a focus on repository-level coding tasks. He holds a PhD in Electrical Engineering from Northwestern University.

Part 4 – Cybersecurity journeys: What I’d tell my younger self

Post Syndicated from Shannon Brazil original https://aws.amazon.com/blogs/security/part-4-cybersecurity-journeys-what-id-tell-my-younger-self/

Over the past few weeks, I’ve introduced you to people who fell into security from the most unexpected places, who work in domains you’ve probably never heard of, and who were honest about what the job is like after the excitement fades and the real work begins. This is the final post in a four-part series following 11 Amazon Web Services (AWS) security professionals through the real stories of how they got here and what the work actually looks like. For this last one, I wanted to end where almost every one of my conversations ended up on its own: the things people wish they’d known sooner.

This one is about hard-won perspective, the kind you only get from doing the uncomfortable thing and coming out the other side. It’s about growth that doesn’t always feel like growth while it’s happening, and the advice these folks would hand to the earlier versions of themselves. If you’ve been with me since Part 1: I didn’t plan this, thank you for sticking around. I hope this last set of stories gives you something to carry into your own path.

Choosing the harder room

Justin Mills, Security Engineer

Justin Mills, Security Engineer

When I asked Justin Mills where someone should start, I expected a certification or a favorite course. His answer started with two words, “seek discomfort,” and his own career shows he means it. Justin is a security engineer with more than a decade in the field, and today he consults on how AI can strengthen security detections. Detections are the rules and models that watch a system’s activity and flag the signs of an attack, and both they and AI are changing faster than most of us can keep up with. What stuck with me was how much faster everything moved after he committed to doing the hard things: he earned two promotions in roughly 2 years.

So I asked him how, and his answer came back to the people around him. What surprised me was that he never went looking for them. Some came through luck, starting with the manager who hired him, and others arrived the way most good working relationships do: he got involved in projects, met the people on them, and the conversations just started flowing. He didn’t push or force any of it.

“Those who are meant to be part of your career story will be part of it, no shoehorning needed.” —Justin Mills

The other half of the equation was trust. Today, Justin works on hard problems, and his manager gives him room to work through them his own way, because he’s shown he can be trusted with that room. His manager even told him so directly: “I trust you to do the right thing and speak for us.” Earning that kind of trust changed how he approached the work. “That forced me to ask the right questions instead of defaulting to what I already knew,” he told me. There’s something important in that framing, because asking good questions isn’t a sign that you don’t belong; it might be the clearest sign that you do.

The lesson he had to learn later was almost the opposite of the one that got him started. Early on, growth meant taking on everything hard himself. As he moved up, it meant the reverse: learning to let go, treating delegation as a skill in its own right, and trusting the people on his team to carry work he could have gripped tightly. When I asked what he’d tell someone who feels stuck, he kept it short.

“Surround yourself with people who push you, not people who make you comfortable. Ask the right questions. Do the hard things. That’s the whole playbook.” —Justin Mills

Pandora’s box and a pile of dust

Tyler Gabor, Technician

Tyler Gabor, Technician

Tyler Gabor has always been the person who takes things apart. He started out selling and fixing cell phones, learning what made them tick, before moving into help desk work. From there he made his way into data center operations, where the hardware went from handheld to industrial and the stakes scaled up with it. “If it lights up or makes noise, I’m messing with it,” he told me, and he meant it.

The jump from help desk to the data center came from preparation. While he was working help desk from home, Tyler studied for a few foundational certifications, and when he could show he knew what he was talking about, he started putting his name in for roles. One of those led to AWS, though not directly: a staffing agency he partnered with placed him on a temporary assignment there, and from that point he had to prove himself all over again to earn a full-time offer. He admits it wasn’t easy and was sometimes scary, and he compares the whole stretch to the Olympics, trying out for a spot on Team Amazon alongside some of the most talented people in the field. “It was the audition of a lifetime,” he told me.

When Tyler explains his work to people outside tech, he reaches for mythology. “It’s kind of like fixing Pandora’s box,” he said, “while also making sure that Pandora doesn’t get outside of the box.” It’s a vivid image for a job that involves testing, repairing, and sanitizing server hardware so the data a machine once held can’t resurface. That can mean wiping a drive, degaussing it (using a powerful magnetic field to scramble the stored data), or physically destroying it. This is security you can physically touch, and as he put it, “It’s a little hard to get intellectual property out of a pile of dust.”

What pulled him in wasn’t a degree program or a bootcamp. It was watching high-profile leaks and breaches dominate the news as a student, which made it clear that security wasn’t an abstract discipline but something shaping real events in the real world. That curiosity never faded; it’s the same impulse that has him tinkering with hardware side projects to this day, like programmable NFC chips (the same short-range wireless technology behind tap-to-pay cards) that he brings to hacker conferences as business cards. That same urge to see what a piece of hardware is really capable of shows up in his day job, too.

One project crystallized the stakes for him. A piece of hardware came through that most people would have assumed was safe to discard, and on closer inspection it was still holding information that should never have left the building. The lesson stuck with him, and he put it the way only Tyler would.

“It doesn’t matter if the front door is locked if all the windows are open.” —Tyler Gabor

What surprised me most, though, was how much he talked about leadership, not in the org-chart sense, but the quiet kind that shows up when your team trusts you to make the right call on your own. He didn’t expect to be trusted with hardware of this caliber, and you could hear how seriously he takes that responsibility. His advice for anyone early in their career, or thinking about the switch, sounds simple but clearly took time to earn: just ask for help, because when you do, people usually give it. And if you’re already in and feeling stuck, prepare for the role you want so you’re ready when it’s time to step up to the plate.

“We are going to witness the greatest leap in human intelligence during our lifetime. As technology grows at an exponential speed, we’re going to need people to keep it in line.” —Tyler Gabor

A short word from Zlata

You might remember Zlata Pavlova from Part 1, where she shared her path from political science to open source investigations and into protective security. I wanted to bring her back for a moment, because something she said has echoed through nearly every conversation in this series: the thing that holds people back is rarely a lack of skill, it’s the quiet suspicion that they don’t belong. Her answer to impostor syndrome is to remind yourself that you do and to build connections early. She spent years assuming her teammates already knew what she knew, and it turned out they often didn’t, because their focus was elsewhere. She also told me the biggest surprise of her current role was how much of it is writing, and especially how much of that writing means translating technical findings for readers who aren’t technical. As you move up in security, the reports, advisories, and briefs you write become one of the clearest signals of your judgment, so practicing that skill early, whether through a blog post, a conference talk, or simply explaining your work to a friend outside the field, pays off for years.

“Don’t feel embarrassed sharing something you think people already know. They might not, and you might help somebody who’s just starting out.” —Zlata Pavlova

The thread

Something I didn’t expect when I started this series was how often people’s earlier jobs turned out to be an advantage rather than a detour. Zlata told me that people who come into security from non-technical careers often explain things better, because they remember what it was like to need the explanation. Tyler’s years in telecommunications, where customers handed him phones that held their whole lives, are what first got him thinking seriously about protecting people’s information. And in Part 3, Brandon’s time administering systems and shipping software gave him the empathy for builders that his application security work depends on. The backgrounds people bring with them are part of why they’re good at this work.

If you’re early in your career and wondering whether you belong here, the people in this series would tell you that you do. If you’re mid-career and feeling stuck, pay attention to whichever story felt familiar, because that’s usually a clue about where to look next. And if this series showed you a corner of security you didn’t know existed, that corner might be a good place to start.

If you want to start, here are a few places to begin:

I’m grateful to everyone who sat down with me and shared their story. Hearing them changed how I think about what this field looks like, and I hope reading them does the same for you as you work out your own path.

What would you tell your younger self about getting into security? Share it on LinkedIn!

This concludes the Cybersecurity Journeys series. Read the full series: Part 1: I didn’t plan this, Part 2: What even is this job?, Part 3: The work behind the work, and Part 4: What I’d tell my younger self.


Shannon Brazil

Shannon is a senior security engineer, managing a team on the AWS Customer Incident Response Team (CIRT), specializing in digital forensics and cloud security investigations. Known in the community as AWSlady, she is passionate about security education and mentoring the next generation of defenders.

How to run a DR readiness engagement

Post Syndicated from Peter Garnepudi original https://aws.amazon.com/blogs/architecture/how-to-run-a-dr-readiness-engagement/

Part 3 of 3

The framework from Part 1 (three business decisions) and Part 2 (progressive validation) works at any pace. You can apply it over months as part of normal operations, or compress it into a structured engagement that moves from undefined targets to a validated recovery commitment in weeks. This post walks through how to run that engagement yourself, with concrete steps, outputs, and a before-and-after example from a real enterprise engagement.

Step 1: Kickoff (define the scope)

Bring three groups into one room: the application owner who knows the technical architecture, the business owner who understands the consequence of downtime, and the platform team that owns the shared services your application depends on. If a dependency is owned by a team not represented in the room, that’s your first action item before proceeding.

Ask these questions in order:

  • What does the user need to be able to do for this function to count as “recovered”?
  • Which applications, shared services, and accounts are in the path of that user journey?
  • Who owns each piece, and have they confirmed their recovery timeline?

The output artifact is a boundary map: a document listing every service and account in the recovery path, with an owner assigned to each and their stated recovery timeline recorded. This single artifact becomes the foundation for every subsequent step.

Step 2: Assessment (surface the gaps)

Run a structured review of each component in the boundary map. For each dependency, ask: does it recover in the target AWS Region, does it recover within the stated recovery time objective (RTO), and does an upstream service’s recovery target cap the RTO you can actually commit to?

  • A Well-Architected Framework Review against the Reliability pillar gives you a systematic way to surface single points of failure, missing health checks, and unverified quotas.
  • An Operational Readiness Review checks whether the runbook’s assumptions hold: are the resources pre-provisioned, do upstream dependencies recover within the target, and are there any manual steps that depend on a specific person being available?

The output is a prioritized list of dependency gaps with owners assigned and a target fix date for each. “Done” at this step means every gap has an owner, a planned fix, and a committed date. Gaps without owners don’t get fixed.

Step 3: Validation plan (structure the tests)

Build a test calendar that progresses through the levels from Part 2. A practical sequence for a first engagement:

  • Week 2: Tabletop exercise on the highest-consequence function. Confirm ownership, escalation path, and expected duration per step.
  • Week 4-6: Planned failover after gap remediation. Measure actual recovery time against the committed RTO. Verify data consistency against the recovery point objective (RPO).
  • Week 8-10: Surprise drill (if the planned failover passed). Test coordination without advance notice.

Don’t schedule a game day with fault injection until you’ve passed a planned failover cleanly. Each level builds confidence for the next. Attempting a higher level before the lower one passes cleanly wastes effort and produces noisy findings that obscure the real gaps.

Step 4: Execute and learn (close the loop)

Each test produces findings. Document them in a format that maps each finding back to the decision it informs:

  • Boundary findings (a shared service discovered mid-test that wasn’t in the recovery scope) → feed back into Decision 1.
  • Consequence findings (recovery took longer than the business can tolerate) → feed back into Decision 2, where the business owner either accepts the longer timeline or invests in a faster architecture.
  • Dependency findings (an upstream service whose recovery target capped the application’s own, a resource that wasn’t pre-provisioned) → feed back into Decision 3.

The output is an updated recovery commitment that reflects what you actually measured, not what you assumed. This document replaces the prior version and becomes the baseline for the next cycle.

Step 5: Ongoing cadence (prevent drift)

Recovery posture drifts as applications change. A new microservice adds a dependency nobody mapped. A team deploys a configuration change that shifts traffic routing. A quota that covered peak load 6 months ago no longer does. Schedule a quarterly review that checks: has the boundary changed (new shared services, new accounts)? Has the business owner confirmed they still accept the RTO/RPO targets? Have any dependencies changed their recovery behavior?

A trigger for re-engagement outside the quarterly cycle: any architecture change that adds a cross-account dependency or changes the failover path.

What this looks like in practice

In the enterprise engagement referenced throughout this series, the team applied this pattern across its portfolio of applications and accounts. The first exercise missed a contractual recovery commitment of 5 hours, taking more than twice that long. Root causes mapped to all three decisions: the boundary excluded a shared authentication platform (most downstream applications couldn’t authenticate users), the dependency chain included an upstream service whose recovery target capped what the applications calling it could achieve, disaster recovery (DR) resources weren’t pre-provisioned, and DNS failover depended on a manual step that took over 3 hours under coordination pressure.

Over 4 months, the team redefined the boundary to include shared platforms, documented the full dependency chain per application, reconciled each application’s target against its slowest dependency, pre-provisioned the recovery environment, and replaced manual DNS failover with automated routing built on Amazon Application Recovery Controller (ARC) routing controls. ARC routing controls are on/off switches on a data plane that shifts traffic without depending on the Amazon Route 53 control plane, so a cutover doesn’t rely on modifying DNS records mid-event. This follows the Well-Architected guidance to rely on the data plane, not the control plane, during recovery (REL11-BP04), the same static-stability principle behind pre-provisioning. The second exercise passed. The organization operated a live customer-facing system from the recovery Region for the first time. Senior leadership cited the structured engagement as the factor that connected the two different outcomes.

Accelerating the timeline

You can run every step in this post with your own teams. If you want to compress the timeline, AWS Enterprise Support provides specific mechanisms that map to each step: a Technical Account Manager to facilitate the kickoff and maintain engagement momentum between sessions, Well-Architected and Operational Readiness Reviews for the assessment, specialist Solutions Architects for deep dependency analysis, and AWS Countdown for coordinated operational-readiness and engineering support during critical events, including planned failovers and DR game days. These are acceleration options, not prerequisites. The steps and outputs are the same either way.

The AWS Resilience Lifecycle Framework and AWS Resilience Analysis Framework provide the detailed practices behind each decision, and AWS Resilience Hub helps you assess applications against your defined RTO and RPO targets.


About the authors

Peter Garnepudi

Peter Garnepudi

Peter Garnepudi is a Senior Technical Account Manager and Enterprise Support Lead at Amazon Web Services. With 20 years in the IT industry and 9 years at AWS, Peter is a database specialist who works with Strategic, ISV, and Financial Services customers to optimize their cloud operations. His expertise includes database architecture, workload resilience, chaos engineering, and GenAI risk governance. He is based in Ottawa, Canada.

Jay Desai

Jay Desai

Jay Desai is a Technical Account Manager at Amazon Web Services in Toronto. He works with Financial Services customers to help them run secure, resilient, and well-architected workloads on AWS.

Validating your DR commitment: From controlled test to real-world readiness

Post Syndicated from Peter Garnepudi original https://aws.amazon.com/blogs/architecture/validating-your-dr-commitment-from-controlled-test-to-real-world-readiness/

Part 2 of 3

In Part 1 of this series, you defined three business decisions that shape your disaster recovery (DR) commitment: the recovery boundary, the business-anchored target, and the dependency-reconciled commitment. Those decisions give you a target on paper. Testing tells you whether it holds under real conditions.

This post covers a progressive approach to DR validation. Each level of testing surfaces a different class of gap, and each gap feeds back into the three decisions from Part 1 so your commitment gets stronger with every cycle.

Why a single test isn’t enough

A scheduled DR exercise with advance notice proves that the failover mechanism works. That’s a necessary starting point. You can confirm that traffic shifts correctly, data remains consistent in the recovery Region, and the application functions as expected after the transition.

A scheduled exercise doesn’t test the assumptions that only surface under pressure. It doesn’t show whether teams who haven’t rehearsed together can coordinate, or whether everyone agrees on who declares the failover and under what conditions. It also can’t confirm that you provisioned and validated your recovery environment before you triggered the failover. These are the conditions where real recoveries stall, and they’re invisible to a test where every team knows it’s coming and watches closely.

In the same engagement from Part 1, several applications had DR instances in a stopped state. The failover automation assumed it could launch instances, create database replicas, and scale container fleets on demand during the exercise. That approach worked only if the team confirmed two things in advance: that the recovery AWS Region had capacity for the requested instance types, and that the account quotas could absorb the full production load. The team didn’t verify these assumptions in advance.

The architecture was correct. The team didn’t verify those assumptions in advance, and it didn’t pre-provision the recovery environment. To close this gap, the team needs to pre-provision the recovery environment and configure auto scaling groups, confirm that its account quotas match production capacity requirements, and remove any runbook step that assumes your team needs to create resources during the event.

Four levels of progressive validation

The AWS Resilience Lifecycle Framework treats testing as a practice you deepen over time. The four levels below are a maturity model you can apply yourself, ordered from lowest-cost awareness to highest-fidelity validation. We’ve drawn it from patterns across the DR exercises we’ve seen. Each surfaces a different class of gap.

Level 1: Tabletop exercise. A facilitated walkthrough of the recovery plan without touching production. The goal is shared awareness and ownership clarity.

Questions to ask during the tabletop:

  • Who declares the failover, and under what conditions?
  • For each step in the runbook, who owns it and what’s their backup?
  • What’s the expected duration of each step, and what’s the total end-to-end timeline?
  • Which steps depend on a specific person being available?

In the same case study from Part 1, the first tabletop surfaced three boundary gaps: the shared authentication platform wasn’t in the recovery scope, a networking account owned by a different team had no failover runbook, and the DNS update step listed a generic team alias with no named owner.

Level 2: Planned failover. A scheduled failover with advance notice. The goal is confirming the mechanism works under known conditions.

What to measure:

  • Actual recovery time per application compared to the stated recovery time objective (RTO).
  • Data consistency validation in the recovery Region to confirm the recovery point objective (RPO).
  • Any step that requires manual intervention (potential automation candidate).
  • Any dependency that recovered later than expected.

In the same case study, the planned failover revealed the DNS misconfiguration: the shared platform’s Amazon Route 53 records still resolved to the primary Region 45 minutes into the test. Most downstream applications appeared healthy but couldn’t serve users. Actual recovery took more than twice the 5-hour target.

A successful planned failover produces a measured gap (or confirmation) between the RTO target you defined and your actual recovery time. That number is the starting point for every subsequent conversation with the business owner about whether the target is realistic.

Level 3: Surprise drill. A failover triggered without advance notice to the responding team. The goal is testing execution readiness, not capability.

This level finds the gaps that preparation hides: the runbook that only its original author can interpret, the step that depends on a specific person’s tribal knowledge, and the escalation path that nobody has practiced. In the case study, a later surprise drill revealed that only one engineer understood how to validate database consistency in the recovery Region. With that person unavailable, the team lost 90 minutes locating and interpreting the procedure.

Level 4: Game day with fault injection. An exercise where you inject specific dependency failures using AWS Fault Injection Service, which ships purpose-built scenarios such as “AZ Availability: Power Interruption” and “Cross-Region: Connectivity”. The goal is testing resilience under conditions you didn’t fully script.

You discover the dependency you assumed would be healthy, the monitoring gap that delays detection, and the cascade that your tabletop didn’t model. This is the closest approximation to a real event without waiting for one to happen.

What each level reveals

The tabletop finds the step no team owns. The planned failover finds the misconfiguration. The surprise drill finds the coordination failure. The game day finds the unmodeled dependency. Each level surfaces a different class of gap.

Run only the calm, scheduled test and you validate the mechanism while leaving the conditions that cause real recovery failures untested. The point isn’t to skip levels. You build confidence at each level before increasing realism. But stopping at the planned failover, a pattern we often see once the mechanism works, leaves team, process, and dependency gaps invisible until an event exposes them.

Closing the loop

Each test produces specific findings that map directly to the three decisions from Part 1.

  • A boundary gap (a shared service discovered mid-test that wasn’t in the recovery scope) feeds back into Decision 1.
  • A recovery that exceeds the time the business can tolerate feeds back into Decision 2, where the business owner either accepts the longer timeline or invests in a faster architecture.
  • A dependency that recovered slower than the target assumed feeds back into Decision 3 for a revised commitment that reflects what was actually measured.

In the same engagement, the first exercise surfaced all four patterns described across this series: a boundary gap (shared platform not in scope), a dependency gap (an upstream service whose recovery target capped the application’s own), a stability gap (resources not pre-provisioned), and a coordination gap (manual DNS failover). The team mapped each finding to a specific decision and a specific fix. Four months later, the second exercise passed. The organization operated a live customer-facing system from the recovery Region for the first time.

The value of progressive testing is not passing a single exercise. It’s building a practice where each cycle tightens the recovery commitment, and each finding has a clear path back to the decision it informs.

Where to start

If you haven’t run a tabletop yet, start there. Gather the owners of one critical business function and walk the recovery plan together. Ask: who declares the failover, what’s the first action, who owns each step, and what’s the expected duration of each. If the room disagrees with any answer, this indicates an opportunity to clarify and strengthen your recovery plan before touching production.

If you’ve passed a planned failover, consider whether a surprise drill would surface coordination gaps that preparation masks. In this engagement, the surprise drill cost the team 90 minutes that the planned failover never surfaced. Whatever level you’re at, document what each test reveals and feed it back into the three decisions. A posture that improves every test cycle is worth more than one that passes a single exercise and never evolves. In Part 3, we walk through how to structure a full DR readiness engagement: who to involve, what to produce at each step, and how to compress the timeline from months to weeks.


About the authors

Peter Garnepudi

Peter Garnepudi

Peter Garnepudi is a Senior Technical Account Manager and Enterprise Support Lead at Amazon Web Services. With 20 years in the IT industry and 9 years at AWS, Peter is a database specialist who works with Strategic, ISV, and Financial Services customers to optimize their cloud operations. His expertise includes database architecture, workload resilience, chaos engineering, and GenAI risk governance. He is based in Ottawa, Canada.

Jay Desai

Jay Desai

Jay Desai is a Technical Account Manager at Amazon Web Services in Toronto. He works with Financial Services customers to help them run secure, resilient, and well-architected workloads on AWS.

Three business decisions that define your DR success

Post Syndicated from Peter Garnepudi original https://aws.amazon.com/blogs/architecture/three-business-decisions-that-define-your-dr-success/

Part 1 of 3

A disaster recovery (DR) strategy is only as strong as the business decisions behind it. The architecture and failover mechanisms matter, but they execute against targets that you define. In our experience facilitating enterprise DR exercises, the difference between teams that meet their recovery targets and teams that don’t often comes down to three decisions made (or deferred) before the first architecture diagram: what you’re actually recovering, what the business needs from that recovery, and what your dependencies allow you to commit to.

This post helps you validate whether your current approach accounts for each one. It draws on a pattern from a real engagement where revisiting these decisions turned a missed recovery target into a successful test four months later.

This is Part 1 of a three-part series. It walks through those three decisions and shows how to make them before you choose an architecture pattern. Part 2 covers progressive validation testing. Part 3 covers how to run a DR readiness engagement. The approach builds on the AWS Resilience Lifecycle Framework and the AWS Resilience Analysis Framework.

Define the recovery boundary

The first decision is drawing the line around what “recovered” means. You cannot meaningfully define a recovery time objective (RTO) until you specify precisely what you need working for you to consider that objective met.

The Resilience Lifecycle Framework defines a business function as the end-to-end capability a user depends on (completing a transaction, filing a report, accessing an account). A single business function often spans multiple applications, accounts, and teams. When you set a recovery target at the application level without accounting for the shared services that the application calls, you might define the boundary too narrowly to deliver the capabilities your users require.

Here’s a pattern we observed in a DR exercise for a large enterprise customer. The customer tested dozens of applications across many AWS accounts. Each application had its own failover automation, and each one failed over correctly. But a shared authentication platform (running in a separate account, managed by a different team) used Amazon Route 53 simple routing pointed at the primary AWS Region. It had no health checks and no failover configuration. Because the team didn’t include it in the recovery scope, the platform remained in the primary Region while everything else moved. The result: Most of the downstream applications that depended on it couldn’t authenticate users. Although the team had successfully failed over the applications themselves, the business function (user login, transaction, confirmation) did not work. This single boundary gap, not the individual application architectures, was the primary driver of the missed recovery target.

The solution involved redefining the recovery scope rather than changing the architecture. The organization redefined recovery boundaries around business functions (what capabilities must you deliver to your users?) rather than individual applications (is this service running?). That single change pulled the shared authentication platform inside the scope and helped the team discover the dependency. A business impact analysis is the structured starting point for this work.

Anchor the target to business consequence

The second decision is settling what recovery means in terms the business owns. Business stakeholders strengthen RTO and recovery point objective (RPO) when they help set them. In organizations where these targets are defined by engineering alone, they can become disconnected from what the business actually tolerates. A 4-hour RTO means something different for a payments platform that clears interbank settlement than it does for an internal reporting dashboard, even when both share the same database engine and replication setup.

In the same engagement, the business carried a contractual recovery commitment of 5 hours. The business treated it as a firm commitment to their regulator. The engineering team treated it as a stretch goal based on the architecture’s theoretical capability. Neither side had validated whether 5 hours was achievable given the dependency chain. That disconnect only became visible when the first exercise took 9–11 hours. The target had never been tested against the actual recovery path, and the business stakeholder who owned the penalty had never been asked whether the engineering team’s assumptions matched theirs.

RPO is a statement about data loss. For a payments platform clearing interbank settlement, an RPO of 5 minutes might still be too wide if those minutes contain transactions that can’t be reconstructed. The question for the business owner is: Which records must survive a Regional impairment, and can you prove they will? The business owner uses that framing to decide whether asynchronous replication (with its window of in-flight data loss) is acceptable. The alternative is synchronous writes, with the latency cost that comes with them.

RTO is a statement about downtime tolerance. A 4-hour recovery time is a technical figure until someone attaches a consequence to each hour. For regulatory filings, the fourth hour might mean a missed deadline with financial penalties. For a storefront, every minute carries lost orders.

The AWS Resilience Lifecycle Framework recommends defining objectives around business outcomes directly (for example, the percentage of successful orders maintained when a component degrades). A service-level agreement (SLA) is a contract with a financial penalty. A service-level objective is a target you manage toward. Raising an SLA doesn’t make an application more resilient.

The output of this decision is a recovery requirement stated in language the business recognizes and engineering can build against: the function, the consequence of losing it, the data that must survive, and the tolerable downtime before the consequence lands.

Account for the dependency chain

The third decision is confirming that your recovery target accounts for everything in the critical path, not just the components you own. Your application recovers only as fast as its slowest dependency. If a service you call takes 4 hours to recover in the recovery Region, your achievable RTO is 4 hours regardless of your own architecture.

In the same engagement, one customer-facing application had committed to a 1-hour recovery target and had the architecture to meet it on its own. But it called a shared identity service, owned by a different team, to validate every user session. That identity service carried a 4-hour recovery target. No matter how fast the customer-facing application recovered, users couldn’t do anything until the identity service came back. The application’s real, achievable RTO was 4 hours, set entirely by a dependency its own team didn’t own and hadn’t accounted for when they committed to 1 hour.

This is why the dependency chain, not the application in isolation, defines the commitment you can make. Identify and document every service your business function calls, note the recovery target each one holds, and set your own target against the slowest link you depend on. Where another team owns a dependency, its recovery target becomes a commitment you need from them, in writing, not an assumption. The AWS Fault Isolation Boundaries whitepaper helps you reason about which dependencies hold during a Regional impairment, and which introduce risk. The distinction that matters is what your runbook asks a service to do. Reading existing data keeps working during an impairment: existing credentials still authenticate, and existing DNS records still resolve. Changing things is what carries risk, because creating or modifying AWS Identity and Access Management (IAM) roles and Route 53 records runs through control planes that live in a single Region. A recovery path that only reads existing state holds up. One that must create or change resources mid-event might not. That’s the case for settling those changes ahead of time.

Where you can, reduce the chain itself. When you pre-provision and run a dependency in the recovery Region before the event, you reduce how many services need to recover before you can restore your application. The shorter the chain of things that must recover in sequence, the closer your achievable target gets to the one you want to commit to.

Where to start

Start your validation by examining one business function. Assess whether: Is the recovery boundary drawn around the full user journey, including shared services in other accounts? Has a business stakeholder confirmed the consequence-per-hour attached to the current RTO and RPO? Does the target account for every dependency in the path, including global endpoints and any runbook step that assumes resource creation during failover?

If you can answer yes to all three with evidence, your targets are well-founded. If any answer is uncertain, you’ve identified where to focus next. In Part 2, we cover how to validate your answers through progressive testing that closes the distance between a controlled exercise and real-world readiness.


About the authors

Peter Garnepudi

Peter Garnepudi

Peter Garnepudi is a Senior Technical Account Manager and Enterprise Support Lead at Amazon Web Services. With 20 years in the IT industry and 9 years at AWS, Peter is a database specialist who works with Strategic, ISV, and Financial Services customers to optimize their cloud operations. His expertise includes database architecture, workload resilience, chaos engineering, and GenAI risk governance. He is based in Ottawa, Canada.

Jay Desai

Jay Desai

Jay Desai is a Technical Account Manager at Amazon Web Services in Toronto. He works with Financial Services customers to help them run secure, resilient, and well-architected workloads on AWS.

Part 3 – Cybersecurity journeys: The work behind the work

Post Syndicated from Shannon Brazil original https://aws.amazon.com/blogs/security/part-3-cybersecurity-journeys-the-work-behind-the-work/

The most important work I’ve done in cybersecurity would make for terrible television. There were no countdown clocks and no frantic typing, just a lot of careful reading, a lot of hard conversations, and the occasional quiet moment where everything finally clicked. That’s the part I want to show you this week, because the three people I talked to have built their entire careers on the work that happens off-camera, and it’s the part that decides who lasts in this field.

This is a four-part series following 11 AWS security professionals through the real stories of how they got here and what the work actually looks like. In Part 1: I didn’t plan this, I shared how three of them found their way in, and in Part 2: What even is this job?, we looked at roles most people don’t know exist. This week is about what the day-to-day actually feels like, told through three AWS security professionals whose work rarely makes it into the trailer but is the reason any of it holds together.

Unlimited ammo

Brandon Caldwell, Mgr, Security Engineering

Brandon Caldwell, Mgr, Security Engineering

Ask most people to picture an application security engineer at work and they’ll imagine someone hunched over a terminal breaking into systems. Brandon Caldwell‘s actual days (reviewing software while it’s being built so the flaws never ship) look almost nothing like that. He spends them in conversation: sitting with the engineers who built a product, walking through how it works, and asking question after question until he understands it well enough to see where it could break. “This job is at least 50 percent people,” he tells the folks he hires. “You have to people a lot.”

The hardest part of the day isn’t technical, it’s human. Brandon’s team has to hand builders news they don’t want to hear, and he’s blunt with new hires about what that really feels like.

“Our job is to go tell someone their baby is ugly. I use that phrase because it is meant to draw out empathy. This is the thing they love and spent time on, and you show up at the end and say that thing is ugly. Of course they are going to react.” – Brandon Caldwell


Learning to deliver that message without crushing the person across the table is, in his telling, the actual craft of the job. It’s why he prizes empathy, and why he says the engineers who struggle most in application security are the ones who got into tech specifically to avoid talking to people. The rest of the day is the patient, unglamorous work of reading someone else’s code closely enough to find the one flaw they didn’t know they’d left, then explaining it in a way that lands as help rather than judgment.

That explaining is more than a conversation. So much of security work gets written down: the finding documented so an engineer can act on it, the advisory that tells customers what to do, the policy that sets the bar for everyone who comes after. The ability to write clearly, to turn something messy and technical into a few sentences a reader can actually use, is one of the quiet differentiators as you move up in this field, and it’s a skill worth building long before anyone asks you to.

None of this is where Brandon started. He’s been curious about breaking things since he was a kid hex-editing the game Doom for unlimited ammo, taught himself reverse engineering (taking apart finished software to understand how it works) before it had a name, and spent years as an IT admin and then in incident response at Microsoft back when cybersecurity barely existed as a job title. What pulled him off the breaking side was noticing—after watching the same handful of mistakes scroll past again and again (things like SQL injection and cross-site scripting, two of the most common ways attackers exploit a poorly built website)—that he’d rather help teams avoid the break-in than perform it. Application security, he says, is his natural place.

As for breaking in yourself, Brandon’s advice follows straight from how he spends his days: practice interviewing and asking the right questions and spend real time around builders so you understand what goes into the thing you’re critiquing. When he hires, certifications are the last thing he looks at; he wants to see capture-the-flag competition results (hacking contests where you solve security puzzles to earn points) and public GitHub projects, because those prove you’ve done the work. Build something, break something, and get comfortable talking to the people who build.

Forget that

Kurt Heath, IT Support Associate

Kurt Heath, IT Support Associate

Kurt Heath‘s chosen corner of the work is offensive security: penetration testing (getting paid to legally break into systems to prove where they’re weak) and, eventually, red teaming. The day-to-day of getting good at it is far less cinematic than the label suggests. It’s hours of hands-on practice against deliberately vulnerable targets, reading about a technique and then trying it until it works, and the slow accumulation of the layers the job actually rests on: operating systems, networking, software development, and security fundamentals, all applied at once. He’s the first to say pen testing is “the sexy thing” that gets advertised, and the first to warn that it shouldn’t be anyone’s first stop, because of how much quiet groundwork sits underneath it.

A big part of his real workday now is deciding how, and how much, to lean on AI. Kurt treats it as a tool rather than a crutch: he uses it as a coding assistant when he has to work in an unfamiliar language, and to sort through large volumes of information quickly, but he won’t let it think for him. He still fills out forms like this blog’s questionnaire by hand, keeping a personal touch on his own work, and he’s wary of letting AI pull the trigger during an engagement, since an overzealous tool can act on a network however it sees fit, cause unintended operational damage, or wander out of scope and create a legal problem. The way he sees it, security is fundamentally a human problem: automation keeps getting better at the systematic parts, but thinking outside the box the way an attacker does stays stubbornly human.

How Kurt got here matters less to the day-to-day than how he keeps going, but it’s worth knowing that he’s largely self-taught. After a community college program didn’t work out, he taught himself from scratch, finishing shifts at an auto plant and then spending nights with YouTube and a search bar learning how computers and networks actually worked. The hardest part, he told me, was self-motivation, and what carried him through was a mix of sheer want and, eventually, other people: the friends and community he found in Discord servers, security podcasts, and interest channels who helped him turn raw drive into real focus. He even got a tattoo as a reminder that there is no plan B.

“There is no other option. There’s no what if. This is it. This is life now, and I’ll figure it out some way, somehow, no matter how long it takes.” – Kurt Heath


If you’re drawn to the same offensive work, Kurt’s advice is to resist starting there. Build the foundation first, often through a role like a security operations center, join the communities where people are learning in the open, and let the flashier work come after the groundwork is solid. Progress in this field—in his experience—is measured less by the title you land than by how much you grow and who you grow alongside.

The comfort lock

Seth Jeacopello, Mgr, Security Engineer

Seth Jeacopello, Mgr, Security Engineer

Seth works in application security too, but if Brandon’s days are mostly about people, Seth’s are mostly about code, which makes the two of them a useful picture of how differently the same job can look. When Seth Jeacopello gets stuck on a hard problem, he doesn’t stare at it harder, he picks up a lock. An actual physical lock with pins and tumblers, because he’s a serious lock-picking hobbyist, and over the years he’s learned that the best response to hitting a wall is to set the problem down and pick what he calls a comfort lock, one he knows well and can open easily. It isn’t about the lock; it’s about giving his brain a small win so he can return to the hard thing with fresh eyes.

The comfort lock exists because Seth’s actual day-to-day is easy to get lost in. In application security, a big part of the job is reading enormous amounts of someone else’s code looking for the one flaw that matters, and as a lifelong developer he genuinely loves reading code, which is exactly what makes it a trap. He described the pull of it to me: you convince yourself the vulnerability is just behind the next function, and the next, and suddenly two days are gone. The skill he’s spent decades sharpening isn’t finding things, it’s knowing when deep enough is deep enough and when to step back, which is where the lock comes in. After 26 years, the meta-skill of managing his own attention is as much a part of the craft as anything technical.

That longevity goes back a long way. Around 1999, still in his teens, he landed a software quality assurance internship, was hired as a developer by 17, and gravitated toward the security-flavored work from the start before moving into network security tooling, rising to chief architect and then chief technology officer along the way.

“We’re like three or four guys off the street coming in with a product, and this huge company was like, yeah, you three, go for it. That was probably one of the coolest things.” – Seth Jeacopello


His advice for building the instincts the daily work demands is refreshingly concrete: set up a deliberately vulnerable practice machine, break into it, and then go read the logs it generated, so you learn not just how an attack works but what it looks like from the defender’s side. That single exercise builds both the offensive and the investigative muscle at once, and it’s the kind of patient, repeatable practice that the day-to-day of this field actually rewards.

The thread

There’s a version of cybersecurity that makes for great television, fast-paced and full of eureka moments, and those moments do exist. But the three people I talked to this week live in a different version of the field, one built on patience, repetition, and a kind of stubborn care that doesn’t photograph well. For Brandon, the real work is the hard, human conversation of telling a builder their work is flawed without making an enemy of them. For Kurt, it’s the unglamorous hours of hands-on practice and the discipline to use AI as a tool without letting it think for him. For Seth, it’s reading one more page of code and knowing when to stop, with a comfort lock on the desk for when he can’t.

None of that makes a great action sequence, but all of it is what a security career actually looks like, and the good news for anyone starting out is that these are learnable habits, not inborn talent. Notice how much of the real work is human: empathy, patience, judgment, and knowing how your own mind works. The analytical, interpersonal, and self-management skills you already have are not a consolation prize, they are part of the job. If you want to begin building the rest, here are a few places to start:

  • TryHackMe and Hack The Box let you stand up deliberately vulnerable machines, break in, and study what you triggered, exactly the exercise Seth recommends
  • The OWASP Top 10 explains the most common application security flaws, including the ones Brandon saw on repeat
  • The OWASP Cheat Sheet Series turns those flaws into practical, task-level guidance on how to build and review software securely; the daily bread of Brandon’s and Seth’s work
  • The Darknet Diaries podcast tells the human stories behind real security incidents, and it’s a good way to hear what the work actually feels like

And if you want to build cloud security skills specifically, AWS has free and hands-on resources to help you start:

  • AWS Skill Builder offers more than 1,000 free self-paced courses, including security learning plans you can start today
  • AWS Builder Labs lets you practice in a real AWS console on hands-on security labs, the stand-it-up-and-try-it approach Seth recommends

In Part 4: What I’d tell myself, we close the series with the advice and hard-won lessons these professionals would hand to someone just starting out.

Doing the work behind the work? These three want you to know it counts. Share your story on LinkedIn!


Shannon Brazil

Shannon is a senior security engineer, managing a team on the AWS Customer Incident Response Team (CIRT), specializing in digital forensics and cloud security investigations. Known in the community as AWSlady, she is passionate about security education and mentoring the next generation of defenders.

Part 2 – Cybersecurity journeys: What even is this job?

Post Syndicated from Shannon Brazil original https://aws.amazon.com/blogs/security/part-2-cybersecurity-journeys-what-even-is-this-job/

If you’ve ever looked at a list of cybersecurity jobs and felt like none of them fit you, you might just be looking at a short list. Most career guides cover the same few roles, like the analyst watching alerts or the pen tester (penetration tester, someone hired to legally break into systems to find weaknesses) looking for a way in. Those jobs matter, but they’re a small slice of a field with room for a lot of different skills. The three people in this post do work you probably won’t find in those guides, and none of them started out qualified for it. What they had in common was a willingness to aim at something before they were ready and to build the skills on the way there. If you’ve ever talked yourself out of a role because you didn’t tick every box, their stories are worth a few minutes, because the move each of them made is one you can make too.

In Part 1: I didn’t plan this, I shared how three people found their way into security from a warehouse floor, a marketing desk, and a teenager’s bedroom. This week is about where people end up after they’re inside. I’ve been in security for over a decade, and while putting this series together I kept running into work I hadn’t come across before: measuring the risk of a million applications at once, building detection tools from the customer service side of the house, and thinking seriously about how to secure infrastructure that’s being launched into orbit.

The skill tree

Jason Lane, Security Engineer

Jason Lane, Security Engineer

When Jason Lane told me he plans his career like a Dungeons & Dragons character build, I assumed he was being metaphorical, but he meant it literally. “Think of it like a video game skill tree,” he said, and then walked me through the whole framework. You start by picturing your level 20 character, the version of yourself with the skills and the role you actually want, and then you work backward. At level 19, what choices get you to 20? At level 18, what needs to be true so you even have those options? You keep stepping back until you’re standing at level 1, looking at a path you built from the top down.

For Jason, level 20 was vulnerability research (finding unknown security flaws in software before attackers do). At the time he was a systems administrator, keeping servers and networks running, which gave him a solid grasp of how systems fit together but little exposure to picking apart the applications on top of them. That gap, application security, was the next node on his tree, so he called an IT recruiter, laid out his plan, and was honest about where he stood. The recruiter found a credit card processing company that needed an application security engineer and was open to someone who wasn’t one yet. For Jason it was a paid apprenticeship in the exact skill he was after; for the company it was a motivated engineer who would grow into the role. He describes it as a deal where both sides got what they needed, and that framing matters: he wasn’t settling for less, he was trading a steeper learning curve for a faster path to where he wanted to be.

He spent 4 years there taking applications apart to understand what made them breakable, and when he had learned what that role could teach him, he applied the same framework again. A friend offered to get him an interview at Google, and Jason’s honest reaction, that he wasn’t smart enough to work there, is the kind of self-doubt most of us recognize (also known as imposter syndrome). His friend didn’t buy it, the interviewers didn’t either, and they hired him. The lesson he took from it is one worth borrowing: the voice telling you that you don’t belong in the room is often wrong, and the way to quiet it is to put in the work until the skills speak for themselves. He went on to spend nearly 7 years doing Android vulnerability research, the kind of deep technical work most companies don’t have the budget for.

That track record opened the door to Amazon. When the team that measures application risk interviewed him, the hiring manager’s pitch was simple: we could really use your vulnerability research experience. Years of tearing systems apart to find what made them breakable turned out to be the perfect background for judging which systems were most at risk. Today, Jason measures risk across roughly a million applications at Amazon, evaluating which ones would cause the most damage if compromised and which are most likely to be targeted, so the organization can focus on the most dangerous ones first. When his manager recently presented the team’s work to a new senior leader, the example he used was a project Jason had built from scratch. Jason’s name wasn’t on the slide, and he told me he didn’t need it to be, but you could hear the pride in his voice.

When I asked what he’d tell someone with a big vision and no idea where to start, he said to get obsessively clear about where you’re going, because the gaps will show up on their own. He compared it to planning a 100-mile trip and realizing you aren’t wearing shoes; once you see it, you can’t unsee it. And if it turns out the dream is really a hobby, he’s at peace with that too.

“If you can wait for Saturday, maybe you should just wait for Saturday. There’s nothing wrong with that.” —Jason Lane

If you’ve been waiting for someone to hand you a career ladder, Jason’s approach offers another option: decide where you want to end up, and let that tell you what to learn next.

Full circle

Jason Goodman, Security Engineer

Jason Goodman, Security Engineer

Where Jason Lane planned his career from the finish line, Jason Goodman’s career has looked more like a pinball machine, bouncing through three security disciplines in about a year and picking up new skills along the way.

The foundation was laid long before he knew it. Eight years ago, before Amazon acquired Ring, he was working in customer service, troubleshooting doorbells and helping customers with their accounts, and before that he sold computers at Best Buy. “That was the most tech I had,” he told me, grinning. What pointed him toward security was partly a nudge from his father-in-law, who worked in IT, and partly what he was seeing on the phones every day: the occasional account access that didn’t look right, the quiet signals that something might be off. Those patterns made him want to understand the systems behind them, so while he was still in customer service he went back to school for a cybersecurity degree.

He didn’t wait until graduation to start building toward the move. He reached out to the manager of the internal threat team (the people who look for misuse of access from inside an organization) and asked a question anyone can ask: what could I do to keep progressing while I’m in school? That turned into stretch projects, and here is the part worth noticing, he earned the bigger ones by delivering the smaller ones first. His standout project was a fraud prevention dashboard that helped the team spot potential misuse of customer account access, a blind spot nobody had flagged. He had the credibility to be handed it because he’d already shown up for the low-stakes work, and it opened the door to his first security engineering role on that same team.

From there he moved to an adversarial threat detection team (the people who watch for outside attackers trying to get in), which meant flipping his mindset overnight: instead of asking what someone on the inside could do with their existing access, he was now asking how someone on the outside would try to break in. A later move took him into application security, where the work shifted from responding after the fact to building security into processes from the ground up.

“I’m in my third discipline in a year since transitioning out of customer service into security engineering.” —Jason Goodman

What I find fascinating is how much carried over from one role to the next. The Splunk queries (Splunk is a tool for searching and analyzing large volumes of log data) he learned in customer service carried into internal threat work, the detection logic from insider investigations carried into adversarial detection, and now in application security he has come full circle, working alongside the same customer service teams he started with, except this time he’s helping build the security around them.

Through all of it, he has leaned on four different mentors, and the way he found them is worth stealing. Finding one mentor who truly invests in you is hard; finding four is rare. Jason didn’t wait to be assigned anyone. He reached out to people whose work he admired, starting with that internal threat manager during school, asked specific questions, and stayed in touch over the years as he grew. Each mentor served a different purpose: one for navigating school, one for stretch projects, one for learning to think like an adversary, and one for getting grounded in application security. “I’ve always felt like I could take at least a little nugget from each one of them,” he said. Next on his list is offensive security, meaning pen testing and red teaming (simulating real attackers to test how well an organization defends itself). He figures starting on defense was the right call: learn what attackers do and how to stop them, then learn to think like one.

If your current role feels unrelated to security, Jason’s story is a good reminder that the skills you’re building now may be exactly the ones your first security job needs. The analytical habit of noticing when something is off, the patience to explain a problem clearly, even the instincts you pick up from a completely different field all transfer. The trick is learning to name what you already have and point it at the work you want.

The edge of the map

Harshvardhan Chunawala, Solutions Architect, AWS Startups

Harshvardhan Chunawala, Solutions Architect, AWS Startups

Harshvardhan Chunawala was still in high school when he started finding security flaws in real products and reporting them to the companies that built them. He had started out building websites and web applications, and after he understood how software was put together, he couldn’t stop wondering how it could be pulled apart. Rather than wait until he had a degree or a job to explore that, he began poking at real products on his own, and when he found a flaw he contacted the company and worked with them to fix it.

Today he advises startup founders at AWS on security and volunteers his time to teach cloud and security courses to students, who often ask him where to start when entry-level work is being automated. His answer has changed over the years. Now that generative AI can answer the surface-level questions, he believes curiosity matters more than credentials, because the people who stand out are the ones who keep asking questions after the tool runs out of answers.

“AI tools put everyone on the same starting line. Interest and passion has become the driving force.” —Harshvardhan Chunawala

What surprised me most, though, was where his own curiosity took him. While studying at Carnegie Mellon University (CMU), he ended up working on space missions, including time as a mission operator for the Iris lunar rover (Iris and CMU’s next rover, MoonRanger, appear with him in the photo above). If you had asked me a few years ago whether security belonged anywhere near a rover on the Moon, I wouldn’t have known what to say, but Harshvardhan would have answered right away: anything that can be commanded remotely can be commanded by the wrong person.

When I asked about the future of security, he took the conversation completely off the map. He talked about robots entering people’s homes and about data centers moving from land to underwater to space. A satellite has to be protected in orbit, where nobody can physically reach a compromised system, and on the ground, where the stations that command it are targets too. “So are cyber security specialists going to go under astronaut training?” he asked, grinning, and he was only half joking. He has already written a two-part guide to incident response for satellite operations on AWS, so for him, this corner of the map is already part of the job.

If you’re drawn to an area that doesn’t look like a security job yet, Harshvardhan’s story is worth taking seriously: the move he keeps making is to follow a question past the point where the easy answers run out, into territory nobody has fully mapped. You don’t need a satellite to do that. Pick the corner of technology that genuinely fascinates you, start asking how it could be attacked and defended, and you might find you’re standing at the edge of a role that doesn’t have a name yet.

The thread

These three stories share one thread: the job you end up loving might be one you haven’t heard of yet. Jason Lane mapped his moves in advance, Jason Goodman carried his skills through three disciplines in a single year, and Harshvardhan is already helping AWS customers secure satellites and the ground systems that run them. None of them could answer “what do you do?” in a single sentence, and I suspect that’s true for a lot of people doing interesting work in this field.

If you’re curious about the roles that exist beyond the ones you already know, here are a few places to start exploring:

In Part 3: The work behind the work, we pull back the curtain on what the day-to-day of security actually looks like compared to what people imagine.

Have you found a corner of security most people don’t know exists? Share your story on LinkedIn!


Shannon Brazil

Shannon is a senior security engineer, managing a team on the AWS Customer Incident Response Team (CIRT), specializing in digital forensics and cloud security investigations. Known in the community as AWSlady, she is passionate about security education and mentoring the next generation of defenders.

Part 1 – Cybersecurity journeys: I didn’t plan this

Post Syndicated from Shannon Brazil original https://aws.amazon.com/blogs/security/part-1-cybersecurity-journeys-i-didnt-plan-this/

If you’ve ever wondered whether your background qualifies you for a career in cybersecurity, you’re not alone — and you’re probably more prepared than you think.

My name is Shannon Brazil, and I work in security communications at Amazon Web Services (AWS). I have been in the security space for over a decade, and my own journey into it wasn’t exactly a straight line. I started in Statistics Canada helping people fill out export declaration forms over the phone, then got a college placement in IT, moved into Digital Forensics and Incident Response, and eventually found my way into the side of security that most people don’t think about: how we communicate about it, how we tell the stories, and how we help people understand what this work looks like from the inside.

Nearly every time I’m asked that “how do I get in?” question, my answer is different. Not because I’m making it up as I go, but because I try to make it relatable to the person asking. Every person I’ve met in this field started from a different and took a completely different road to get here. There is no checklist, no degree that unlocks the door, and no straight line.

I decided to create this series for Cybersecurity Awareness Month, however I didn’t want to write another “Top 10 Tips to Break Into Security” post. Instead, I interviewed with 11 security professionals across AWS, each from a different team and a different background, and asked them one basic question: how did you actually get here?

What came out of those conversations surprised me. The stories were funny, honest, sometimes hard to hear. Over the next four weeks, I will be sharing them in a four-part series, grouped by theme:

  1. I didn’t plan this (this post) explores the non-linear paths that brought people into security from places you wouldn’t expect.
  2. What even is this job?” dives into the lesser-known corners of security that most people don’t realize exist.
  3. The work behind the work pulls back the curtain on what the day-to-day looks like compared to what people imagine.
  4. What I’d tell myself wraps the series with advice, growth moments, and the kind of wisdom that only comes from having lived it.

Nearly every person I spoke to had a different story, but one thing kept coming up: nobody planned this. And that’s kind of the whole point. So let’s start there.

Mr. Robot and a box cutter

Justin Knight, Security Engineer

Justin Knight, Security Engineer

Justin Knight was packing boxes in an Amazon warehouse when his career in cybersecurity started. He just didn’t know it yet.

It was around 2015 or 2016, and he’d just started at Amazon, working fulfillment, scanning, packing, shipping, with no tech background and no degree in computer science. What he did have was a TV show.

“I saw Mr. Robot,” he told me, grinning. “And I was like, man, I want to do that.” I’ve heard a lot of origin stories in this field, but something about watching a fictional hacker-vigilante while surrounded by conveyor belts and cardboard, and deciding that’s the career for me, just felt different. Justin started teaching himself by turning YouTube into his classroom (Daniel Messler for the fundamentals, Stök for the hacker mindset, NetworkChuck for the energy), and spinning up a Kali Virtual Machine (a free operating system built for security testing) and diving into Hack the Box (an online platform where you practice cybersecurity kills by solving challenges) and TryHackMe labs (a beginner-friendly platform for learning cybersecurity through guided exercises), all while still packing boxes.

When an IT role opened up inside the warehouse, Justin jumped on it and started doing the tasks the team disliked the most: fixing broken laptop screens. He would proactively walk the floor looking for cracked displays before anyone even filed a ticket, replace printers (the silent soul-killer of every IT professional), and eventually became the “computer guy” while already looking for what was next.

Justin found a bug bounty role posted on LinkedIn, and even though he had no idea what bug bounty even was, he reached out anyway. The hiring manager had a one-on-one with him, saw the passion, and did something I’ve seen happen before but never gets old: he lowered the role level requirements to a lower seniority level so Justin could make the jump.

Four years later, Justin is still on that team. He just got back from DEFCON, his second time, where he met the very YouTubers who taught him everything he knows.

“I didn’t even know what bug bounty was until I joined the team.” —Justin Knight

What gets me about Justin’s story is that none of the traditional ingredients were there: no degree, no bootcamp, no network of security professionals opening doors. All Justin had was a show that lit a fire, a YouTube playlist, and the kind of curiosity that made him walk the warehouse floor looking for broken screens nobody asked him to fix.

The accidental investigator

Zlata Pavlova, Sr Intel Analyst

Zlata Pavlova, Sr Intel Analyst

If Justin’s story resonated with you, Zlata’s might hit even closer to home; she started with a marketing gig at a pen testing firm.

Zlata has a degree in political science. After school, she worked the kinds of jobs that have nothing to do with either politics or science: hospitality, restaurants, retail. At some point, she got interested in social media marketing, and that led to a contract with a small penetration testing (pen testing) firm. Not to do anything security-related, just to run their social media, maintain their website, and represent them at conferences. But something happened when she started spending time around people who think for a living. The pen testers, the red teamers, the people who look at a locked door and think “how would I get through that without the key?” Their energy was contagious.

“I had no idea that there’s actually a title or a role based on finding information online,” she said. “I didn’t know that could be transformed into a career.”

Zlata had always been good at sleuthing, finding things and connecting dots that other people missed. She just didn’t know there was a word for it: OSINT, or open source intelligence. Before long, she was supporting the red team’s operations, preparing phishing engagements and doing reconnaissance for physical security assessments, and eventually conducting the assessments herself. And then she found Trace Labs.

For those unfamiliar, Trace Labs runs capture-the-flag competitions focused on finding missing persons. Zlata started as a competitor, moved into judging, and then took it even further by volunteering with the National Child Protection Task Force, doing OSINT investigations on cases involving real victims.

One of those cases led to people being rescued and suspects being arrested. A political science major who took a marketing contract at a pen testing firm helped rescue real people from real danger, because she followed her curiosity into a field she didn’t even know existed.

“That was one of the biggest cases I personally worked on. Hearing that our efforts actually helped people being rescued and the bad guys arrested… that was really rewarding.” —Zlata Pavlova

Today, Zlata works on a technical risk team, bridging the gap between cybersecurity and physical security for executive protection. She came from a world where none of this was on the radar. And her advice for anyone feeling like they don’t belong?

“Impostor syndrome is real, but you belong here. Share what you learn. Don’t assume everyone already knows it. They might not.” —Zlata Pavlova

Access denied

Arman Sadri, Security Engineer

Arman Sadri, Security Engineer

Arman Sadri’s story starts in a place most cybersecurity career guides don’t typically cover: getting in trouble as a teenager.

As a teenager, Arman was into video games. Really into them. So into them that he ended up hacking into a major tech company and stealing hardware prototypes. He was 16, maybe 17. “I’m dumb. I’m a kid,” he told me, laughing about it now, but you could hear the weight of it in his voice. That experience taught him a lot, but it also left him terrified that no legitimate employer would ever give him a chance, especially not a Fortune 500 company.

He enrolled in Year Up, a program that pairs 6 months of college with 6 months of an internship, and did so well they had him teaching the classes. Amazon offered him a job before the internship even started, letting him skip ahead and start running. He landed in IT support (the help desk, the person who remotes into your computer when something breaks) and for a while, that was the job, but Arman had his sights set on security.

He applied to a mentorship program for aspiring security engineers and found himself on a 2-year waitlist, but he waited it out. When he finally got in, he met with a senior leader every week, showed what he could do, and coached the other mentees in the room. At the end of the program, there was a hiring challenge. No other candidate could complete it, and even after they extended it externally. Arman was the only one who finished.

Four months of silence went by before he got a response: “Hey, there’s this role opening up. I think you’d be a great fit.” The role didn’t exist before that conversation; they created it for him.

“If you knew by the 80th no it was a yes, you’d be so happy every time you got told no.” —Arman Sadri

What stays with me about Arman’s story is the refusal to let his past define his ceiling. He didn’t hide from it, he just outworked it. A 2-year waitlist, and he waited. A challenge nobody could finish, and he finished it. A role that didn’t exist, and now it does. And when I asked him what makes the biggest difference for someone trying to break in, he didn’t say certifications. He said: “What can I Google about your name and see? Show me what you’ve built. Show me the impact.”

The thread

These three stories share one thread: curiosity, persistence, and a willingness to start before feeling ready. If you’re looking for a place to begin:

In Part 2: What Even Is This Job?, we explore the roles most people don’t know exist in cybersecurity — and why that matters for your career.

Have your own unconventional path into security? Share your story on LinkedIn!


Shannon Brazil

Shannon is a senior security engineer, managing a team on the AWS Customer Incident Response Team (CIRT), specializing in digital forensics and cloud security investigations. Known in the community as AWSlady, she is passionate about security education and mentoring the next generation of defenders.

Building cloud-native PACS on AWS

Post Syndicated from ManojKumar MV original https://aws.amazon.com/blogs/architecture/building-cloud-native-pacs-on-aws/

Modernizing medical imaging infrastructure is a pressing challenge for multi-hospital networks. Cloud-native PACS (Picture Archiving and Communication System) on AWS can help address the challenge at scale. A hospital chain with multiple facilities generates millions of imaging studies annually: each CT produces 300 to 2,000 DICOM images, MRI generates 500 to 3,000 slices, and digital mammography produces 8 to 12 high-resolution images.

At this scale, a typical network accumulates 50 to 200 terabytes of new imaging data yearly, with retention mandated for 7 to 10 years. The traditional approach used on-premises PACS with SAN or NAS storage at each hospital independently. This worked when volumes were modest, but as chains grow through acquisition, the constraints of this siloed architecture become apparent.

In this post, we present a hybrid cloud architecture pattern for PACS on AWS. We describe the core components, explain how data flows from imaging devices to a centralized cloud archive, and outline storage tier options and capacity planning guidance. This post is for healthcare IT architects and solutions architects familiar with DICOM workflows.

Challenges that do not scale

  • Storage cost explosion: Enterprise SAN/NAS requires hardware refresh every 3-5 years with annual maintenance contracts consuming 15-20% of hardware cost. Organizations must over-provision storage for projected peak capacity years in advance.
  • Data silos: A patient scanned at Hospital A cannot have images viewed at Hospital B within the same chain.
  • Radiologist reporting bottleneck: When a radiologist is unavailable, studies pile up with no mechanism to route to available readers at other facilities.
  • Continuous archive growth: PACS/VNA storage must scale indefinitely with no capacity ceiling and no upfront provisioning of unused capacity.

How traditional PACS works today

The workflow begins when a clinician orders an imaging study. The Radiology Information System (RIS) fills the order and populates the modality worklist. The technologist selects a patient entry from the modality worklist and acquires a study. The scanner transmits DICOM objects to the PACS server through C-STORE on the hospital LAN (TCP port). A DICOM object contains image metadata and pixel data.

The PACS server ingests DICOM images and HL7 orders. It archives images on local SAN/NAS, indexes metadata, and notifies the radiologist worklist. The radiologist reviews images with patient history and creates a report. The report flows back to the EMR through HL7 messaging.

The following diagram shows the traditional on-premises PACS workflow and its limitations.

Traditional on-premises PACS workflow from imaging modality through DICOM C-STORE to the PACS server, radiologist, and EMR

Figure 1: Traditional on-premises PACS workflow

DICOM protocol: The language of medical imaging

DICOM (Digital Imaging and Communications in Medicine) is a widely adopted standard for storing, transmitting, and viewing medical imaging files. DICOM specifies a binary file format encapsulating pixel data and metadata and defines network services including DIMSE (DICOM Message Service Element) services: C-STORE (send), C-FIND (query), C-MOVE (retrieve), and C-ECHO (verify connectivity).

DICOM DIMSE services are designed for LAN. They facilitate interoperability and image exchange on the hospital campus.

DICOMweb is a set of RESTful services that web developers use to access DICOM-enabled systems with industry-standard toolsets.

Key components of a PACS architecture

Every PACS, regardless of vendor or deployment model, consists of six core building blocks. Cloud migration does not replace these components. Instead, it re-hosts and enhances them with cloud-native capabilities. The following diagram and table describe each component and its role in architecture.

Six PACS components

Figure 2: Six core components of a PACS architecture

Component breakdown

The following table summarizes each component, its role, and how it operates within the architecture.

Component Role How it works
Web Server Serves PACS viewer UI, authentication, session management Renders DICOM in browser with windowing, leveling, and measurement tools. Zero-footprint, no client install required.
VNA Server DICOM ingestion, format normalization, image streaming Receives C-STORE from modalities on LAN. Normalizes multi-vendor encoding. Compresses and stores objects.
Application server Worklist management, study routing, sync coordination Routes studies by urgency and subspecialty. Integrates with HIS/EMR through HL7 v2 or FHIR REST APIs.
Database Patient MPI, study location tracking, sync state Stores everything except pixels: demographics, modality, storage location. Supports cross-facility patient lookup.
Object Storage All DICOM images centralized, lifecycle-managed Replaces SAN/NAS with scalable pay-per-use storage. Lifecycle policies auto-tier by age and access.
PACS Viewer Local + Cloud dual viewer with transparent routing Routes requests to local or cloud viewer based on image availability. Clinicians remain unaware of data source.

How the components interconnect

An imaging device completes acquisition and sends DICOM objects to the VNA through C-STORE over the hospital LAN. The VNA normalizes encoding, applies compression, and writes standardized image bytes to storage.

The Application server updates the metadata database with the complete study record. It then evaluates routing rules to assign the study to the appropriate radiologist worklist based on urgency and subspecialty.

When a clinician opens a study, the PACS Viewer checks image location in the metadata database. Locally cached studies serve at LAN speed. Expired cache studies stream from the cloud viewer through a content delivery network. The clinician interacts with a single interface and remains unaware of the backend source.

Cloud-native PACS architecture on AWS

This architecture pattern applies to hospital networks that run a single PACS vendor consistently across all facilities. A common infrastructure across every site and the cloud is what allows the centralized system to discover and retrieve studies from any hospital in the network. The recommended architecture follows a hub-and-spoke model. Local PACS instances at each hospital (spokes) connect to a centralized cloud archive (hub) through AWS Direct Connect or AWS Site-to-Site VPN. This approach preserves quick image retrieval for daily clinical workflow while providing cross-facility interoperability, disaster recovery, and intelligent storage tiering.

The following diagram shows the centralized PACS architecture on AWS with hub-and-spoke connectivity.

Centralized PACS architecture on AWS using a hub-and-spoke model, with local hospital PACS instances connecting to a centralized cloud archive across two Availability Zones

Figure 3: Centralized PACS architecture on AWS

Architecture flow

Each hospital retains a local PACS with Web Server, VNA, Application server, and local database. Imaging modalities send DICOM objects to the local VNA over the hospital LAN. Studies are immediately available for radiologist reading at LAN speed.

A single PACS vendor is deployed consistently across all hospital sites and in the cloud. Because every site and the centralized cloud archive run the same system sharing a common metadata database, the cloud-based system can discover and retrieve studies created at any facility in the network. A radiologist at one hospital can query and open a study acquired at any other hospital, giving the enterprise a unified patient imaging record.

In the background, the images generated from new studies are replicated to Amazon Simple Storage Service (Amazon S3) through Direct Connect or Site-to-Site VPN. Clinical workflow is never blocked because sync happens asynchronously.

On the cloud side, the centralized PACS runs across two Availability Zones in AWS Region. Web Servers, VNA Servers, and App Servers on Amazon Elastic Compute Cloud (Amazon EC2) sit behind Network Load Balancers with automatic failover. Amazon Aurora PostgreSQL serves as the centralized metadata store with synchronous replication.

Amazon S3 stores DICOM images with lifecycle policies that automatically tier data by access patterns. Amazon CloudFront and AWS WAF deliver the cloud PACS viewer for teleradiology access with IP allow list and encryption.

Transparent viewer experience

When a clinician requests a study, the PACS application checks the metadata database for image location. If the local system has cached the study (the majority of daily requests), it serves the images from local disk at LAN speed.

If the local cache has expired, the cloud viewer streams from Amazon S3 through Amazon CloudFront with progressive loading.

High availability and disaster recovery

The cloud deployment spans two Availability Zones with automatic failover. Amazon S3 replicates objects across multiple Availability Zones. On the local side, the cache serves recent studies if cloud connectivity drops.

If a local server fails, requests route to the cloud where all recent data is already synced. If cloud connectivity drops, the local cache continues serving recent studies without interruption. If a single AZ fails, automatic failover routes traffic to the surviving AZ within seconds.

Data protection and security controls

Medical imaging data contains sensitive patient information including patient names, dates of birth, and clinical findings in DICOM metadata. Under the AWS shared responsibility model, AWS secures the cloud infrastructure, while the customer configures services, manages access, and implements audit controls.

The architecture uses AWS services including Amazon S3 (encrypted image storage), Amazon Aurora (encrypted metadata), Amazon EC2 (encrypted compute), Direct Connect (private connectivity), Amazon CloudFront (encrypted viewer delivery), and AWS Key Management Service (AWS KMS) (key management with rotation).

The architecture includes security controls that healthcare organizations can use as part of their security programs: encryption at rest and in transit across every layer, comprehensive audit logging with AWS CloudTrail and Amazon S3 access logs, least-privilege access through IAM with role-based controls, and continuous monitoring with AWS Config.

For data residency, deploying in a regional AWS location keeps sensitive patient data within national borders. S3 bucket policies can enforce region-level restrictions for organizations with specific data sovereignty requirements.

Storage tier planning

Running PACS on AWS provides the ability to use Amazon S3 storage tiers that align cost with access patterns. Traditional on-premises SAN/NAS stores data on a single expensive tier regardless of access frequency. Amazon S3 provides intelligent lifecycle management that reduces storage costs while improving durability.

Understanding access patterns is key

Traditional on-premises storage uses a single tier for data regardless of access frequency. Amazon S3 provides multiple tiers that align cost with how often data is accessed.

Medical imaging data follows a predictable decline in access frequency: frequent in the first months (reporting, follow-ups), dropping sharply after 6 to 12 months, and rarely accessed after 2-3 years. Mapping this pattern to storage tiers is a high-impact cost optimization decision.

This predictable decline in access frequency makes PACS an ideal workload for tiered storage. The key questions to answer are: how long do radiologists typically reference prior studies? What is your average follow-up window? What percentage of archived studies are ever retrieved after 12 months? These answers drive the lifecycle policy configuration.

S3 Standard: Hot storage for active studies

Studies in their first 6 months to 1 year are actively accessed. Radiologists reference them for follow-up comparisons. Clinicians review them during patient visits. Reporting workflows are still active.

These studies sit on S3 Standard, which provides millisecond access with high throughput. This is equivalent to the performance clinicians expect from traditional local SAN, but without the upfront costs, hardware refresh cycles, or capacity planning overhead.

S3 Glacier Instant Retrieval: For warm data

S3 Glacier Instant Retrieval (GIR) provides millisecond retrieval (the same access speed as S3 Standard) at significantly lower storage cost with nominal retrieval fees. For PACS workloads, this combination of low storage cost with millisecond retrieval is particularly well suited.

When you occasionally access studies older than 6 to 12 months for comparative reads, these make ideal candidates for GIR.

This combination of instant retrieval with archive-tier pricing makes GIR well suited for medical imaging, where occasional access to historical studies is clinically important but infrequent enough to benefit from reduced storage rates.

S3 Intelligent-Tiering: When access patterns are unpredictable

For datasets with unpredictable access patterns (research hospitals, teaching institutions), S3 Intelligent-Tiering automatically moves objects between tiers based on actual usage with no retrieval fees or operational overhead.

S3 Glacier Deep Archive: Long-term retention

Studies older than 5 years that require long-term retention move to S3 Glacier Deep Archive. Retrieval takes 12 to 48 hours, acceptable for infrequent retrieval needs. Storage cost is minimal.

Why this matters for PACS

Amazon S3 replicates objects across multiple Availability Zones within a region. With Cross-Region Replication (CRR), the same archive provides built-in disaster recovery across geographically separated regions. Most modern PACS solutions support S3-compatible APIs natively, requiring no custom middleware.

Amazon S3 stores every object redundantly across multiple physically separated Availability Zones within a region. With Cross-Region Replication (CRR), organizations can maintain a full disaster recovery copy in a secondary region with no additional infrastructure to manage.

The majority of modern PACS solutions natively support writing and reading data through S3-compatible APIs. This eliminates the need for complex storage integration configurations or proprietary connectors.

Cloud-only vs. hybrid: Making the decision

The choice between a fully cloud-based PACS and a hybrid (local + cloud) deployment is not driven by imaging volume. High-volume sites operate successfully in both models. The right answer depends on two factors specific to each facility.

Cloud-only PACS is a strong fit when:

  • Redundant, reliable connectivity is available. The facility’s region has well-established, high-bandwidth links to the cloud from at least two independent network carriers, ensuring no single point of failure for clinical workflows.
  • The PACS vendor offers a cloud-optimized solution. The solution delivers equal or faster performance when deployed in the cloud compared to on-premises. This is achievable today: vendors running entirely on AWS have publicly demonstrated faster image retrieval than traditional on-premises deployments, even at enterprise scale.

This model eliminates local infrastructure, removes hardware refresh cycles, and centralizes operations across all sites.

Hybrid PACS (local + cloud) is a strong fit when:

  • Connectivity is limited or single carrier. Regions where redundant high-bandwidth links are not yet available, or where network reliability does not meet clinical uptime requirements.
  • The PACS solution performs best with local caching. Some vendor architectures are optimized for local-first access, with a site cache providing sub-second retrieval for active studies while background sync handles cloud replication asynchronously.

This model ensures uninterrupted clinical performance regardless of WAN conditions and provides a natural migration path toward cloud-only as connectivity and vendor solutions mature.

Both architectures use AWS as the durable, long-term archive. The difference is where the active working set lives day-to-day.

Conclusion

The hybrid cloud architecture described in this document is designed to help address the core on-premises PACS challenges: storage cost explosion, data silos across facilities, radiologist routing bottlenecks, and unbounded archive growth.

Next step: Conduct a device inventory and access pattern analysis to turn this conversation into a numbers-driven plan.

 


About the authors

The state of AI for security: Measuring what matters most for building trust

Post Syndicated from Anshumali Shrivastava original https://aws.amazon.com/blogs/security/the-state-of-ai-for-security-measuring-what-matters-most-for-building-trust/

Security teams are starting to actively use AI for security work, including vulnerability triage, penetration testing, threat modeling, incident response, and code review. The promise is speed, but a security tool that moves fast and raises too many false alarms doesn’t save time. Engineers spend time on false alarms, on-call is noisier, and teams distrust findings that matter.

Today, we’re releasing Deception Benchmark, the first benchmark designed to measure that trust problem directly. It tests whether a model can distinguish real vulnerabilities from code that looks risky but is actually safe. The benchmark includes 14,822 samples across 16 languages and more than 70 Common Weakness Enumeration (CWE) categories. We evaluated 12 models from five providers and are releasing the dataset and whitepaper to the community. Existing benchmarks measure whether AI can find or exploit vulnerabilities. This is the first to measure whether it can tell real vulnerabilities from false alarms. Under standard prompting, precision at distinguishing real vulnerabilities from false alarms landed in the mid 50s; as likely to be inaccurate as accurate.

In offensive tasks, there’s often a clear result: the exploit works or it doesn’t. Defensive reviews are harder to verify than offensive tasks; a model might recognize a suspicious pattern even when a mitigation makes the issue non-exploitable. In practice, useful systems need to reason about the code, the mitigation, and sometimes the surrounding environment.

The measurement gap

The community has made progress on security evaluations. CyberGym tests agents on more than 1,500 realistic tasks. Meta’s CyberSecEval and CyberSecEval 2 measure exploit generation. CYBENCH evaluates capture the flag (CTF) challenges. SEC-Bench and VulnBench push toward authentic security workflows.

Recent work reinforces both the progress and the gap. ExploitGym measures whether AI can escalate from a crash to a working exploit. Microsoft’s Project Perception deploys multi-agent red/blue/green teams for continuous defense. OpenAI’s GPT-Red shows that self-play red-teaming finds novel attacks that frontier models can’t defend against. Since then, OpenAI disclosed that its GPT-6 Astra model crossed the Critical cybersecurity capability threshold, and both OpenAI and Anthropic reported incidents where models gained unauthorized access to production systems during evaluations. The offensive side is moving fast. But none of this work measures the defensive precision question: when an AI system flags code as vulnerable, how often is it right?

Introducing Deception Benchmark

14,822 samples, 16 languages, and more than 70 CWE categories. We call it Deception Benchmark because the safe samples are designed to deceive models. It has real vulnerability patterns, real frameworks, real idioms, with mitigations that quietly close the exploit path. The goal is to classify code as vulnerable or safe, with no hints.

Consider a Flask endpoint that accepts user input and queries a database. A model will pattern-match to SQL injection, but the query uses parameterized statements, so the exploit path is closed. A single-turn classifier flags the pattern and moves on, never checking whether the exploit can actually work. Production tools rely on multi-step loops and agentic workflows to compensate, but that scaffolding masks whether the model itself understands the code. This benchmark strips the scaffolding away and asks the model to make the call in a single pass, so what it measures is understanding, not how many tries a harness takes to get there.

We built every sample through an adversarial loop: generate, test against frontier models, harden, repeat. If a model gets it right easily, the sample doesn’t survive. The result is a benchmark calibrated to the frontier, not below it. Building it this way is expensive. Generation and hardening of the samples consumed tens of billions of tokens. We’re releasing the result so the community doesn’t have to repeat that cost.

This benchmark generates two challenge types. Code-level challenges (6,988 samples) present vulnerable and safe variants that differ by a subtle fix. Both look suspicious, only one is exploitable. Environment-gated challenges (2,707 samples) go further: same code, different deployment context. A Kubernetes Network Policy blocks the server-side request forgery (SSRF) path. An identity and access management boundary prevents privilege escalation. The pattern is visible in the source. The infrastructure makes it unexploitable. The model has to figure out which scenario applies.

All samples were purpose-built for this benchmark, grounded in real-world patterns, real frameworks, real CWEs, and real infrastructure; without IP concerns or training data contamination.

Large-scale quality data with LLMs and humans in the loop

Generating reliable labels at this scale is difficult: a single pass—by people or by models—leaves errors that skew scores. So we treat labeling as a convergent audit loop rather than a one-time step. Every label is re-examined by multiple independent reviewers, blind to one another and to the original reasoning that produced the label. Disagreements escalate to direct adjudication, where the original reasoning is evaluated against the challenge. Unresolved cases go to human review. We repeat the loop until the scored set converges below a dispute threshold: under 3 percent of samples still contested by independent review, with a target of under 1 percent surviving human adjudication. One choice makes this defensible: we never relabel a disputed sample. When reviewers disagree, the sample moves to the unscored pool instead of being given a corrected label, so a bad challenge can remove a sample but can never introduce a wrong label into the scored set.

A human review of 100 randomly drawn scored samples found no label errors. We describe the full process in the whitepaper.

The results

The benchmark is roughly balanced: half vulnerable, half safe, so a random classifier scores 50 percent. We report two error rates separately, because they fail in opposite directions. The false positive rate (FPR) is how often the model flags safe code as vulnerable. These are the false alarms that waste an engineer’s time. The false negative rate (FNR) is how often it misses a real vulnerability and calls it safe. Accuracy alone hides this: a model that labels everything vulnerable catches every bug (0 percent FNR) but flags all safe code (100 percent FPR) and still scores about 50 percent. We consider FPR below 10 percent and FNR below 10 percent the minimum bar for production use.

Figure 1: FPR compared to FNR for 12 models across two prompting strategies. No model reaches the generous bar

Figure 1: FPR compared to FNR for 12 models across two prompting strategies. No model reaches the generous bar.

Model

Prompt

Accuracy

FPR

FNR

GPT-5.6 Sol Direct 54.9% 92.5% 0.9%
GPT-5.6 Sol PoE 58.9% 58.6% 23.1%
GPT-5.5 Direct 56.9% 87.8% 1.3%
GPT-5.5 PoE 62.9% 63.6% 12.4%
GPT-5.4 Direct 60.2% 81.0% 1.5%
GPT-5.4 PoE 77.7% 10.1% 33.6%
Llama 3.3 70B Direct 58.8% 84.2% 1.1%
Llama 3.3 70B PoE 72.2% 10.2% 44.2%
Claude Haiku 4.5 Direct 55.6% 92.1% 0.0%
Claude Haiku 4.5 PoE 75.6% 22.4% 26.3%
Claude Opus 4.6 Direct 55.9% 91.3% 0.1%
Claude Opus 4.6 PoE 75.8% 42.7% 7.0%
Claude Opus 4.7 Direct 58.3% 85.5% 0.9%
Claude Opus 4.7 PoE 75.9% 32.0% 16.8%
Claude Opus 4.8 Direct 53.8% 95.7% 0.2%
Claude Opus 4.8 PoE 75.8% 32.5% 16.4%
Claude Opus 5 Direct 77.3% 41.5% 5.2%
Claude Opus 5 PoE 79.3% 24.9% 16.8%
Claude Sonnet 5 Direct 62.9% 74.7% 2.2%
Claude Sonnet 5 PoE 74.7% 31.8% 19.2%
Amazon Nova 2 Lite Direct 56.3% 89.2% 1.2%
Amazon Nova 2 Lite PoE 70.1% 45.2% 15.5%
Mistral Large Direct 52.2% 99.0% 0.0%
Mistral Large PoE 65.5% 49.3% 20.6%

Among the general-purpose frontier models tested, no configuration achieves both FPR and FNR less than 10 percent on this benchmark.

Every model has the same failure mode. With direct prompting, they catch up to 95 percent of real vulnerabilities but also flag 41–99 percent of safe code. Precision runs from 52 percent to 71 percent, clustered in the mid-50s; effectively as likely to be inaccurate as accurate. The models see a vulnerability pattern and stop reasoning. Proof-of-exploit prompting cuts false positives by 17–74 points but misses 7–44 percent of real vulnerabilities. The environment-gated challenges are worse: models flag the code and ignore the Kubernetes Network Policy next to it. No tested configuration keeps both false positives and false negatives below 10 percent.

These results reflect general-purpose models in single-turn prompting. Purpose-built systems with multi-step validation and tool use are a different operating point that we didn’t measure, and if a harness can close the gap between pattern recognition and genuine understanding, this benchmark is the place to demonstrate it. Two cautions before assuming it already does. Agentic verification is proven mostly on offensive tasks, where success can be confirmed: the exploit fires or it doesn’t. Judging that code is safe has no such oracle. Extra iterations re-sample the same judgment rather than confirm a negative, and a harness still inherits the base model’s understanding. If the model can’t separate an effective mitigation from an ineffective one in a single pass, more passes won’t add the missing knowledge. That’s what this benchmark measures: the model’s intrinsic ability to understand code, tested at the single-turn baseline where no scaffolding can mask the gap.

For security teams evaluating AI tools today: ask your vendors how their system performs on tasks like this, not just whether it finds vulnerabilities, but how often it’s wrong. Pair any AI-assisted review with human verification on high-risk code paths, and use Deception Benchmark to hold your tools accountable.

Availability

We built Deception Benchmark to simplify measuring this problem in a reproducible way. The public release includes the samples and evaluation workflow. We don’t release the labels, so submissions can be scored consistently over time without turning the benchmark into a memorization exercise.

Of the 14,822 samples, 9,695 are scored; the remaining 5,127 are held out and unscored, mixed in with the rest of the benchmark. The goal is straightforward: make it more difficult to optimize the benchmark compared to improving the underlying system. We describe that design in more detail in the whitepaper.

Deception Benchmark is available on GitHub, along with the whitepaper and submission instructions for verified scoring. If you’re building security tooling, you can download the dataset, run your system against the benchmark, and submit predictions for scored evaluation.

If you have feedback about this post, submit comments in the Comments section below.


Anshumali-Shrivastava

Anshumali Shrivastava

Anshumali is an Amazon Scholar and Full Professor of Computer Science at Rice University. His research on dynamic sparsity, sketching, and hashing pioneered techniques now central to efficient LLM training and inference. A two-time founder — ThirdAI (acquired by ServiceNow) and XMAD.ai (acquired by Workato) — he bridges theoretical computer science and practical AI systems at scale.

Neha Rungta

Neha Rungta

Neha is a scientist and builder who has spent her career making machines reason about complex systems at scale. Her work spans automated reasoning, formal verification, security, and AI, shaping systems including Cedar, IAM Access Analyzer, and Continuum. Today, she is forging the next generation of machine reasoning, combining LLMs, formal methods, and agentic systems.

AWS recognized as a Leader in the 2026 Gartner Magic Quadrant for Strategic Cloud Platform Services for the 16th consecutive year

Post Syndicated from Erika Ehrli original https://aws.amazon.com/blogs/big-data/aws-recognized-as-a-leader-in-the-2026-gartner-magic-quadrant-for-strategic-cloud-platform-services-for-the-16th-consecutive-year/

On September 1, Gartner published its Magic Quadrant for Strategic Cloud Platform Services (SCPS). Amazon Web Services (AWS) is the longest-running Leader in this Magic Quadrant, with Gartner naming AWS a Leader for the sixteenth consecutive year.

In the report, Gartner once again placed AWS highest on the Ability to Execute axis. We believe this reflects our commitment to help customers innovate faster, operate more securely, and build at any scale, particularly as agentic AI drives the need for a data foundation that is production-ready.

Here is the graphical representation of the 2026 Magic Quadrant for Strategic Cloud Platform Services.

For the full evaluation and methodology, download the complete 2026 Gartner Magic Quadrant report and read our lead announcement post.

Your AI strategy is only as good as your data strategy

Your agents are only as powerful as the data they rely upon. Agents need access to your data and shared context to reason accurately and deliver reliable responses.

Today the knowledge agents need is scattered across databases, data lakes, warehouses and third-party applications with no shared context or governance. And the scale of the problem is new. Agents generate 10 to 100x more queries than humans. This means your data architecture must be agent-ready from day one. If it isn’t, your AI investments underperform.

AWS gives your agents an open data foundation with governed context intelligence, built to scale while optimizing the cost of AI. Agentic data capabilities meet industry-specific compliance, security, and schematic requirements so you can move to production with confidence.

An open data architecture for your data and AI

Agents need to discover and access your data, wherever it is stored. That’s why AWS delivers an open architecture on Apache Iceberg so agents can use data across these silos. We offer the broadest native Iceberg support of any major cloud provider, with native Iceberg compatibility across every layer of the data stack – ingestion, storage, catalog, and analytics.

Amazon Simple Storage Service (Amazon S3) supports Apache Iceberg natively. S3 Tables delivers fully managed Apache Iceberg tables that automate compaction and maintenance as data grows. It works with any Iceberg-compatible engine, from Spark to Redshift, and supports natural language queries through MCP.

Amazon SageMaker lakehouse architecture is built with Apache Iceberg. It enables Amazon S3, Amazon Redshift, Amazon OpenSearch Service, Amazon EMR, and Amazon Athena to access the same Iceberg tables through a unified catalog, from a single governance layer. Zero-ETL integrations and federated querying remove remaining barriers across on-premises and third-party cloud sources.

AWS MCP Server, part of the Agent Toolkit for AWS, gives any tool (Amazon Quick, a third-party agent, or a developer’s IDE) governed access to your data through a single path with inherited permissions. It standardizes tool discovery, authentication, and contextual data access for AI agents interacting with AWS services.

AWS embraces open standards for flexibility and the best value. This includes PostgreSQL via Amazon Aurora and Amazon RDS, Apache Kafka via Amazon MSK, OpenSearch via Amazon OpenSearch Service, Apache Spark via Amazon EMR and Trino via Amazon Athena.

From data to contextual intelligence

Agents need more than data access to be accurate. They need contextual understanding of your data and the business rules governing how it should be used before they can make trusted decisions.

This is why we introduced AWS Context, a new service that automatically maps the relationships across your existing data into a knowledge graph and provides agentic search so AI agents in the organization can access governed data relationships, business rules, and domain knowledge at runtime.

For governance, AWS Glue Data Catalog provides a single catalog for AWS and third-party Iceberg tables, while AWS Lake Formation enforces row-, column-, and cell-level access control so the right data reaches the right agent with the right permissions. AWS Glue Data Quality and SageMaker ML Lineage Tracking add the governance layer that production AI demands.

Foundational excellence at scale

Agentic AI workloads require resources that are always available, dynamically allocated, and optimized for price-performance. AWS delivers the most powerful combination of services and capabilities for automatic resource allocation, zero-tuning price performance, and the reliability that millions of customers have trusted for over 20 years.

AWS Databases offer a high-performance, secure foundation to power agentic AI and data-driven applications at any scale. Amazon Aurora delivers unparalleled high performance and availability at global scale for PostgreSQL, MySQL, and DSQL. Amazon DynamoDB and Amazon ElastiCache serve up to tens of billions of requests per second at microsecond to single-digit millisecond latency at any scale, operating at agent speed. With native vector search built into Aurora PostgreSQL, DynamoDB, and ElastiCache, you can perform vector search — from billions to trillions of vectors — and integrate effortlessly across AWS services to build agentic applications.

Amazon S3 has evolved to support the demands of AI with purpose-built storage tiers. S3 Files gives agents a shared file system directly on S3 data, so an entire agent fleet can read inputs, write outputs, and persist memory with no duplicated data and no new APIs to learn. S3 Vectors is the first cloud object store with native support to store and query vectors. It cuts the cost of uploading, storing, and querying vector data by up to 90%, making it practical to build the large-scale vector datasets that give AI agents memory, context, and semantic search.

For search and retrieval, AWS provides purpose-built vector engines that bring intelligent search to your data where it already lives. With OpenSearch Service Serverless, your agents take advantage of lexical, vector, hybrid, and agentic search in a single system with high throughput, low latency, and relevant results at scale.

Key takeaways

The companies moving fastest with AI are the ones that treated data readiness as strategy from the start. We believe the Gartner recognition of AWS as a Leader for 16 consecutive years reflects the breadth and deepest set of core public cloud services and capabilities, including the data foundation that makes this possible.

Ready to see the full evaluation? Download the 2026 Gartner Magic Quadrant for Strategic Cloud Platform Services.


Gartner does not endorse any company, vendor, product or service depicted in its publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner publications consist of the opinions of Gartner’s business and technology insights organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this publication, including any warranties of merchantability or fitness for a particular purpose.

Gartner and Magic Quadrant are trademarks of Gartner, Inc., and/or its affiliates.

This graphic was published by Gartner, Inc. as part of a larger research document and should be evaluated in the context of the entire document. The Gartner document is available to download: Complete 2026 Gartner Magic Quadrant report.

Gartner, Magic Quadrant for Strategic Cloud Platform Services, By Alessandro Galimberti, Carolin Zhou, Douglas Toombs, Dennis Smith, Ed Anderson, Tobi Bet, Chuck Lawton , 1 September 2026


About the author

Erika Ehrli

Erika is Head of Product Marketing for Data for AI at Amazon Web Services, where she leads technical product marketing and go-to-market strategy across the AWS analytics, database, and storage portfolios. In this role she helps organizations build AI-ready data foundations for agentic AI and analytics workloads.

Investigate DMS migration issues with AWS DevOps Agent

Post Syndicated from Chitresh Saxena original https://aws.amazon.com/blogs/devops/investigate-dms-migration-issues-with-aws-devops-agent/

Migrating a production database is a high-risk operational event. AWS DMS is a cloud service that migrates relational databases, data warehouses, and other data stores into the AWS Cloud or between environments. It moves the rows reliably, but the failures that page an on-call engineer rarely happen during the data copy. They occur in the hours after the cutover. A query that was fast on the old engine starts doing full scans. A connection pool sized for the old database becomes exhausted. A downstream service that nobody mapped begins timing out.

These are operational problems, not data problems. They resolve slowly because engineers must correlate many sources under time pressure: DMS task state, Amazon CloudWatch metrics, Amazon RDS Performance Insights, logs, and deployment history.

AWS DevOps Agent can investigate and troubleshoot your database migration issues. DevOps Agent is your always-available teammate that accelerates and validates the deployment of code changes, then keeps your applications running optimally across AWS, multicloud, and on-prem environments. It learns about your resources and their relationships, then correlates telemetry, code, and deployment data to pinpoint root causes and recommend fixes. When issues arise, it autonomously investigates and resolves them.

In this post, you’ll learn how to extend AWS DevOps Agent into a DMS migration specialist. You deploy a sample Model Context Protocol (MCP) server that gives the agent read-only, migration-specific tools and a library of runbooks. You’ll then watch the agent investigate real migration issues and reach grounded root causes on its own. The accompanying GitHub repository includes the full source and a deployment guide.

Prerequisites

Before you begin, make sure you have the following:

  • An AWS account with AWS DevOps Agent turned on and an Agent Space created. Note the Agent Space ID.
  • An active AWS DMS replication task that migrates to Amazon Aurora PostgreSQL-Compatible Edition, with data validation turned on. The repository includes scripts that provision a test migration if you need one.
  • Permissions to deploy AWS CloudFormation stacks that create AWS Lambda functions, IAM roles, and an S3 bucket.
  • The AWS Command Line Interface (AWS CLI) v2 configured, and Python 3.10 or later. No Node.js or CDK is required.

Architecture

DevOps Agent runs inside AWS. It reaches the MCP server over HTTPS and authenticates each request with AWS Signature Version 4 (SigV4). SigV4 is the same IAM mechanism that every AWS API uses, with no keys or shared secrets. The sample deploys your MCP server onto AWS Lambda, behind a Lambda function URL with the AWS_IAM auth type. That auth type accepts only SigV4-signed requests from authorized principals.

The following diagram shows the request path from the agent to the tools.

Architecture diagram

You deploy the server and its IAM roles as one AWS CloudFormation stack. You then register the function URL with DevOps Agent and add the tools to the allow list in your Agent Space. When a symptom appears, you create an investigation. The agent assumes the role, calls the tools, correlates the results, and returns the root cause.

Implementation walkthrough

DevOps Agent supports custom tools through MCP. After you deploy the sample server and register it, the agent calls its tools during an investigation, the same way it calls a CloudWatch tool. Every tool in this server calls only Describe*, Get*, List*, Lookup*, and TestConnection APIs. None of them modifies a resource, which is the property that lets you give them to an autonomous agent.

The MCP exposes 20 tools across the migration lifecycle. The following table lists the ones the agent reaches most often.

Tool Returns
validate_migration_data Validation state distribution, failed and suspended tables (report and skill prompt)
get_validation_failures Tables in non-Validated states with failed and suspended record counts
check_connection_health Endpoint connectivity (waits for the real test result), SSL mode, failure messages
analyze_cdc_latency Source vs target CDC latency, backlog, and an assessment
check_replication_instance_health Replication-instance CPU, memory, swap, storage, network with flags
capture_aurora_performance Aurora CloudWatch metrics and Performance Insights waits and SQL
check_stabilization Post-cutover regressions, missing alarms, recommendations (report and skill prompt)
summarize_task_health One-call roll-up across status, validation, latency, and instance health
list_runbooks / get_runbook Browse the runbook catalog and fetch one by id

The server also ships 46 runbooks covering data validation, full load, CDC, connectivity, replication-instance health, Aurora target health, and cutover readiness.

Database migration stages

You can apply this approach across the migration timeline:

  • Pre-cutover readiness: Confirm endpoint connectivity and that every table has reached the Validated state before you commit to the switch.
  • Issue investigation during cutover: You give the agent a symptom, such as a validation mismatch or a latency spike, and it finds the root cause instead of several engineers chasing parallel theories.
  • Post-cutover stabilization: Detect regressions and missing alarms on the new database before they turn into outages.

What the agent sees, and what it does not

Out of the box, DevOps Agent reads Amazon CloudWatch, AWS CloudTrail, and the AWS APIs in your account. However, for a DMS migration, the agent needs to read migration-specific reasoning an operator applies: how to read a DMS validation state distribution, how to tell a source-side change data capture (CDC) bottleneck from a target-side one, or which alarms a freshly promoted Aurora instance should have but it only has access to raw CloudWatch metrics.

You close that gap with the sample MCP server:

  • Read-only tools that turn DMS, CloudWatch, and Performance Insights data into pre-aggregated reports. The tools do the counting deterministically, and the agent does the interpretation.
  • Runbooks the agent can browse and follow, each grounded in public AWS documentation.

Deploy the MCP server

The MCP server runs on AWS Lambda, the serverless compute service that runs your code without provisioning servers. AWS DevOps Agent reaches it through a Lambda function URL, a dedicated HTTPS endpoint for the function. You do not run or host anything yourself; the deployment creates the Lambda function and its endpoint in your account.

The endpoint is not open to the public. It uses the AWS_IAM auth type, so it accepts only requests that DevOps Agent signs with an IAM role in your account, as described in the architecture section above.

Deploying and registering are two distinct steps. Deploying creates the server: the Lambda function, the function URL, and the IAM roles. Registering tells DevOps Agent that the server exists and adds its tools to the allow list. You can select either of the two options as indicated below.

Option A: Deploy MCP and register via AWS Management Console

Deploy the MCP and perform the MCP registration manually via console:

./deploy.sh us-east-1 --skip-register

The script deploys the MCP server and prints its Function URL and role ARN, then leaves the registration to you. With the server deployed, register it in the console by following Connecting MCP servers:

  1. Sign in to the AWS Management Console and open the AWS DevOps Agent console.
  2. On the Capability Providers page, find MCP Server under Available providers and choose Register.
  3. On the MCP server details page, enter a Name, the Endpoint URL (the function URL from the stack output), and an optional Description.
  4. For the authorization method, choose AWS SigV4, enter the role ARN from the stack output, set the Region to us-east-1, and set the service name to lambda.
  5. Open your Agent Space, go to the Capabilities tab, and add the tools to the allow list.

Option B: Deploy MCP and register with one click deployment

Alternatively, you can deploy the MCP and perform the MCP registration with one-click deployment. Complete the following steps to deploy and register the server.

  1. Clone the repository and change to the deployment directory:
git clone https://github.com/aws-samples/sample-dms-devops-mcp.git
cd sample-dms-devops-mcp/lambda_mcp
  1. Run the deployment script with your target Region:
./deploy.sh us-east-1

The script prompts for your Agent Space ID. It then packages the Lambda code, deploys the CloudFormation stack, and registers the server with your Agent Space, allow-listing all of its read-only tools.

  1. Verify the deployment. In the DevOps Agent console, open your Agent Space, go to the Capabilities tab, and confirm the tools appear under MCP Servers.

If registration returns HTTP 403, confirm the IAM role grants both lambda:InvokeFunctionUrl and lambda:InvokeFunction. Granting only the first returns 403. The template grants both.

Review and investigation

To validate the approach, we ran it against a live migration: an Amazon Relational Database Service (Amazon RDS) for MySQL source replicating to Aurora PostgreSQL through a DMS task with data validation turned on, 1,000 rows across four tables (customers, products, orders, and order_items). Every query and response below is from a real DevOps Agent investigation against that environment. You start each one by entering a prompt in the DevOps Agent web app, the conversational interface where you investigate issues and review findings.

A consistent pattern shows up across all five investigations: the agent chooses a different set of tools for each question. It is reasoning about which tool fits, not running a fixed script.

The five scenarios are not demo picks. Each one represents a class in the DMS failure taxonomy the server’s 46 runbooks cover: data validation, full load, change data capture, connectivity, replication-instance health, Aurora target health, and cutover readiness. That taxonomy is the operational surface of a real migration, so a reader who follows these five is rehearsing the categories they are most likely to hit, not a curated happy path. Each scenario below is one representative of its class; the full catalog is available to the agent through list_runbooks.

Pre-Migration: Confirm cutover readiness

Before a cutover, you want a clear go or no-go.

Query

We are about to cut over the DMS migration (task dms-mcp-test-task, us-east-1, target Aurora dms-mcp-test-target). Before we switch the application, confirm whether this migration is ready: are the endpoints healthy and has all data validated? Give me a clear go or no-go.

Response

The agent chose exactly the go/no-go gate tools: check_connection_health and validate_migration_data for the readiness signals, plus get_task_status, get_validation_failures, list_table_statistics, and analyze_cdc_latency to confirm the full picture. The connection-health tool waits for the real endpoint test result rather than reporting that a test merely started, and the validation tool confirms whether DMS actually compared source and target rows and found them equal, not just that the full load reached 100 percent. In our test runs the agent returned this go/no-go in under two minutes, against the fifteen to thirty minutes an operator typically spends cross-checking endpoint tests, task status, and per-table validation state by hand. (Illustrative from our runs, not a benchmark.)

Cutover readiness check

Figure 1. The readiness check, showing endpoint health and validation state feeding a go/no-go assessment.

Migration: Investigate validation failures

To create a fault, we changed two rows directly on the Aurora target, bypassing DMS, which moved the customers and orders tables to the Mismatched records state.

Query

Our DMS migration to Aurora PostgreSQL (task dms-mcp-test-task in us-east-1, Aurora instance dms-mcp-test-target) is reporting data validation failures on some tables. Investigate the root cause and tell me how to fix it.

Response

The agent ran a deep investigation across 73 journal records, calling 11 of the registered tools (33 tool calls in total). It started with get_task_status, get_validation_failures, validate_migration_data, and list_table_statistics to establish which tables had diverged, then used search_task_logs, correlate_cloudtrail_changes, get_recent_task_events, analyze_cdc_latency, capture_aurora_performance, describe_endpoints, and get_premigration_assessment to build the timeline. It identified both affected tables and produced these findings:

 Finding: CDC changes not applied to target before validator compared rows
 Source modifications occurred on customers and orders at ~05:37:18 UTC.
 The validator compared at 05:37:41 UTC, 23 seconds later, before CDC applied
 them to Aurora. Evidence: CDC captured 17 source events but target id was 0
 (no changes applied), and TARGET_APPLY logs showed "waiting for data from
 upstream" through the window.
 Finding: ValidationQueryCdcDelaySeconds set to 0 allows the validator to race
 ahead of  CDC replication.

That second finding is the difference between a dashboard and an investigation. The agent did not just report which tables failed. It named the exact DMS task setting (ValidationQueryCdcDelaySeconds) behind the transient failures and explained the mechanism, which is the fix an operator can act on. In our test runs the agent reached the ValidationQueryCdcDelaySeconds root cause in a single investigation of about three minutes, work that manually means correlating the failure table, CDC latency, CloudTrail, and task logs across four consoles, commonly thirty minutes or more. (Illustrative from our runs, not a benchmark.)

Validation failure investigation

Figure 2. The DevOps Agent investigation for the validation failure, showing the tool timeline and the root-cause findings.

Migration: Assess replication latency

During ongoing replication, you want to know whether CDC is keeping up and where any delay sits.

Query

I want to understand the replication performance of our DMS task dms-mcp-test-task in us-east-1. Is the change data capture keeping up, and is the replication instance healthy or is it a bottleneck? Summarize the latency picture.

Response

The agent picked up the performance-specific tools: analyze_cdc_latency, check_replication_instance_health, and describe_replication_instance, with get_task_status and list_table_statistics for context. The latency tool compares source latency with target latency and returns an assessment, because the two together tell you where the delay sits. When source and target latency track each other, the bottleneck is the source side. When target latency runs well above source latency, the bottleneck is the target apply side. The instance-health tool flags CPU, memory, swap, and storage pressure that would make the instance itself the limit. In our test runs the latency assessment came back in roughly a minute, versus the manual path of pulling source and target CDC latency and instance metrics from CloudWatch and reasoning about which side leads. (Illustrative from our runs, not a benchmark.)

The latency tool also knows when it cannot answer. When the metric window is too sparse to separate source-side from target-side delay, it returns an insufficient_data verdict and asks for a longer window instead of forcing a conclusion from a handful of datapoints. This is deliberate: a confident but wrong root cause is worse than a request for more data. The agent surfaces that verdict to you rather than inventing a bottleneck, which is what makes its confident answers trustworthy when it does give them.

Replication latency assessment

Figure 3. The replication latency assessment, comparing source and target CDC latency and instance health.

Migration: Run an open-ended investigation

Sometimes the operator does not know what is wrong yet. This is where the runbooks earn their place.

Query

Something seems off with our DMS migration (task dms-mcp-test-task, us-east-1, Aurora target dms-mcp-test-target) but I am not sure what. Investigate broadly, use any available runbooks that match what you find, and report the most important issue with how to fix it.

Response

Given no specific symptom, the agent ran the broadest investigation of the suite: 12 distinct tools across 43 records. It swept the task status, validation state, connectivity, latency, replication instance, endpoints, and logs, then called list_runbooks, recognized the validation symptom it had found, and fetched get_runbook for the matching runbook, which returned the full procedure. It produced a finding about a datatype or precision difference in the MySQL to PostgreSQL migration and followed the runbook to the recommended remediation. In our test runs this broad sweep of twelve tools resolved to a single prioritized finding in a few minutes, against the open-ended manual triage it replaces, which has no fixed time because the operator does not yet know where to look. (Illustrative from our runs, not a benchmark.)

Tools the agent chose: validate_migration_data, get_validation_failures,
list_table_statistics, check_connection_health, describe_endpoints,
describe_replication_instance, analyze_cdc_latency, summarize_task_health,
search_task_logs, correlate_cloudtrail_changes, list_runbooks, get_runbook

Open-ended investigation

Figure 4. The open-ended investigation, showing the agent browse the runbook catalog and follow the matching runbook.

Post-Migration: Review stabilization and monitoring

After cutover, the question shifts from “did the data move” to “is the new database healthy and watched.”

Query

We just cut over to Aurora PostgreSQL (instance dms-mcp-test-target) from a DMS migration (task dms-mcp-test-task, us-east-1). Assess the target database health now and tell me what monitoring or alarms are missing that we should add before production traffic ramps up.

Response

The agent selected the stabilization tool set: check_stabilization, capture_aurora_performance, summarize_task_health, get_validation_failures, and check_pending_maintenance. It read the Aurora target health (CPU, connections, read and write latency, buffer cache hit ratio, and the top Performance Insights wait event), then assessed what monitoring was missing for a database about to take production traffic. The point of this phase is the interpretation: a buffer cache hit ratio that stays low after warmup points to missing indexes, and a new database with no alarm on connection count or query latency is one bad query away from an unmonitored outage. In our test runs the stabilization review returned target health and the specific missing alarms in about two minutes, versus manually inspecting Aurora metrics and Performance Insights and then deciding which alarms a freshly promoted database still lacks. (Illustrative from our runs, not a benchmark.)

Post-cutover stabilization review

Figure 5. The post-cutover stabilization review, with target health and the monitoring gaps the agent flagged.

Improve the agent with Skills and runbooks

A finding like the ValidationQueryCdcDelaySeconds race condition should improve the next migration, not be relearned. The agent’s migration judgment is captured in two places. DevOps Agent Skills are Markdown instruction sets that load automatically when relevant and tell the agent when to call a tool and how to read its output. The runbooks are fetched on demand: the agent calls list_runbooks to browse the catalog, then get_runbook to pull the procedure that matches what it found, as it did in scenario 5. You can fold each new finding back into a skill or runbook, so the agent improves with every migration.

To make the flywheel concrete, here is the runbook the agent fetched in the open-ended investigation. When it found the validation symptom, it called get_runbook and received this procedure, authored from earlier findings and grounded in public AWS documentation:

---
id: validation-mismatched-records
title: "Validation: mismatched records on a table"
severity: HIGH
triggers:
  - "ValidationState=Mismatched records"
  - "ValidationFailedRecords>0"
tools:
  - get_validation_failures
  - list_table_statistics
  - analyze_cdc_latency
  - correlate_cloudtrail_changes
---

# Validation: mismatched records on a table

A table shows Mismatched records, meaning source and target rows differ.
The row-level diffs are recorded in the awsdms_validation_failures_v1
control table on the target.

## Phase 1 — Assess
- Run get_validation_failures to see which tables are in Mismatched records
  and the failed-record counts.
- Run list_table_statistics to confirm the per-table validation state.

## Phase 2 — Investigate
- Query awsdms_validation_failures_v1 on the target for the failing rows/columns.
- Run analyze_cdc_latency: if validation runs during heavy CDC, transient
  diffs can appear while changes are in flight.
- Run correlate_cloudtrail_changes to check for an out-of-band write or reload.
- Check for data-type, precision, timezone, or character-set differences.

## Phase 3 — Report
- State which tables diverged and by how many records.
- Name the most likely cause (type/precision, timezone, encoding, out-of-band write).
- Recommend revalidating the table after the cause is corrected.

## Remediation (operator action)
- Correct the underlying difference, then revalidate with validate-only.

> All steps use read-only MCP tools. Remediation actions are operator tasks
> and are not performed by the tools.

The judgment for when to apply a runbook lives in a DevOps Agent Skill, a Markdown instruction set that loads automatically when relevant. The data-validation skill, for example, encodes the rules the agent follows before it draws a conclusion:

# Skill: DMS Data Validation Assessment

## Critical Rules
- Every finding MUST cite actual numbers from the metrics report, never generalize.
- Do NOT fabricate or estimate any metric. If data is missing, say "Data not available."
- Do NOT hardcode thresholds, use relative comparisons (% of total, trend direction).
- Validation metrics come from TWO sources: the DMS table-statistics API AND
  CloudWatch. Cross-reference both.

## Concepts to Evaluate — ValidationState machine
- Validated          = healthy, all rows confirmed matching
- Mismatched records = ACTION REQUIRED, source/target differ, check failure table
- Suspended records  = source churn too high, DMS cannot compare
- No primary key     = CANNOT VALIDATE, table lacks a PK
Flag if Mismatched + Suspended + Error tables exceed 5% of the total.

Every new finding folds back into a runbook or a skill, so the next migration starts from what the last one learned. The ValidationQueryCdcDelaySeconds race condition from scenario 2 becomes a trigger the agent recognizes on sight, rather than something it has to rediscover.

When to use this approach

This pattern fits issue investigation, pre-cutover readiness gates, and post-cutover stabilization reviews, where an operator hands the agent a symptom and wants a grounded root cause from read-only tools. It is not a replacement for continuous monitoring or alarms, and it does not ship logs for long-term retention. Use it alongside your existing CloudWatch alarms and dashboards, not instead of them.

Clean up

To avoid ongoing charges, delete the resources you created. Delete the CloudFormation stack with aws cloudformation delete-stack --stack-name dms-mcp-test-mcp-server. Remove the MCP server from your Agent Space and deregister it. If you provisioned a test migration with the repository scripts, run the included cleanup script.

Conclusion

Migration issues usually stem from operational problems, not data problems, and AWS DMS does not catch them. In this post, you saw how to extend AWS DevOps Agent to investigate them autonomously, using a sample MCP server that exposes read-only, migration-specific tools and runbooks. Across five real investigations, the agent chose the right tools for each question, found the affected tables, named the exact task setting behind a validation race condition, and followed a runbook to a fix. Because the tools are read-only and access is least-privilege IAM, you can give them to an autonomous agent without widening your operational scope.

To get started, deploy the MCP server from the GitHub repository, register it with your Agent Space, and turn on DMS data validation before your next cutover.

About the authors

Chitresh Saxena

Chitresh Saxena
Chitresh Saxena is a Senior AI/ML Specialist, specializing in generative AI solutions and dedicated to helping customers successfully adopt AI/ML on AWS. He excels at understanding customer needs and provides technical guidance to build, launch, and scale AI solutions that solve complex business problems.

Neel Sendas

Neel Sendas
Neel Sendas is a Principal Technical Account Manager at AWS, leading Cloud Operations for some of AWS’s largest enterprise customers across ML governance, cloud finance, and operational resilience at scale. He is also a core member of AWS’s Machine Learning Technical Field Community, helping shape the roadmap for AWS AI/ML services.

Tipu Qureshi

Tipu Qureshi
Tipu Qureshi is a Senior Principal Technologist in AWS Agentic AI, focusing on operational excellence and incident response automation. He works with AWS customers to design resilient, observable cloud applications and autonomous operational systems.

 

Hybrid cloud orchestration: Modernizing on-premises infrastructure management with AWS

Post Syndicated from Sandeep Singh original https://aws.amazon.com/blogs/architecture/hybrid-cloud-orchestration-modernizing-on-premises-infrastructure-management-with-aws/

This post demonstrates how to build a hybrid cloud orchestration solution that manages distributed on-premises infrastructure at scale using AWS serverless technologies. If you manage geographically dispersed data centers with thousands of servers that require bare-metal configuration, deployment, and ongoing lifecycle management, this solution provides centralized control while maintaining on-premises execution. Many of these environments also need the Kubernetes control plane itself to stay on-premises. This can be for data sovereignty, regulatory, or policy reasons, or because the network to AWS is disconnected, disrupted, intermittent, or limited (DDIL). Amazon EKS Anywhere runs the entire cluster on your own hardware, and this solution orchestrates it at scale from AWS. For on-premises workloads that can use a managed Amazon Elastic Kubernetes Service (Amazon EKS) control plane in the cloud, Amazon EKS Hybrid Nodes is the recommended approach.

In Part 1 of this series, you’ll learn the core architecture patterns for building an event-driven orchestration engine using AWS Lambda, AWS Step Functions, and Amazon DynamoDB. With this foundation, you can automate server lifecycle management through vendor-agnostic APIs, deploy EKS Anywhere clusters consistently across sites, and establish centralized observability for your entire infrastructure. In subsequent posts, we walk through the implementation with code examples, deployment templates, and detailed workflows for server and cluster management.

The challenge: Managing distributed on-premises infrastructure at scale

Managing distributed on-premises infrastructure at scale presents these challenges:

Inconsistency across locations: Different hardware vendors, network architectures, and compliance requirements lead sites to develop their own procedures. The same Kubernetes cluster deployment can produce different results at each site, such as the cluster version installed or the set of add-ons enabled. Without centralized orchestration, identical operations succeed at some locations but fail at others.

Manual lifecycle bottlenecks: The infrastructure lifecycle spans multiple layers requiring manual intervention. Hardware operations include BIOS configuration, firmware updates, and power management. OS operations cover installation and patching. Kubernetes operations encompass cluster creation, version upgrades, and scaling. Application operations involve deployment and maintenance. While manageable for individual servers, these processes become overwhelming bottlenecks when multiplied across thousands of geographically distributed machines.

Fragmented visibility: When management tools operate independently at each site, aggregating data across the entire environment becomes challenging. Operators struggle to answer enterprise-wide questions: How many servers are running outdated firmware? Which clusters are approaching capacity? Without centralized observability, identifying issues and planning capacity requires manual investigation across multiple locations.

Scalability limitations: Orchestration tools designed for a single data center encounter fundamental limitations at enterprise scale. Coordination mechanisms that work for dozens of servers fail when managing thousands. State synchronization becomes unreliable. Maintenance windows that are straightforward for a single site become logistical challenges across hundreds of locations.

Core technologies for hybrid orchestration

To address these operational challenges, four core technologies work together to deliver centralized orchestration with distributed execution:

Hybrid connectivity: Secure network connectivity between AWS and on-premises sites forms the foundation for centralized orchestration. AWS Direct Connect provides dedicated private connections, while AWS Site-to-Site VPN offers encrypted tunnels over the internet. This connectivity allows AWS services running in your virtual private cloud (VPC) to coordinate lifecycle operations with on-premises infrastructure.

AWS architecture stack: The AWS serverless stack along with Amazon EventBridge sets the foundation for an event-driven orchestration engine. Additional compute services include AWS CodeBuild for build processes, AWS Batch for long-running jobs, and AWS Systems Manager for on-premises tasks. These services provide a framework that can handle different execution runtimes while AWS manages the underlying infrastructure.

Redfish APIs: Redfish (a standard protocol for hardware management developed by the DMTF) delivers vendor-agnostic APIs for hardware management, allowing standardized control of bare-metal servers. Through Redfish, BIOS configuration, firmware updates, power management, and health monitoring operations are executed across diverse hardware environments.

Amazon EKS Anywhere: EKS Anywhere creates and operates Kubernetes clusters on your own infrastructure, using the same Amazon EKS Distro that powers Amazon EKS in the cloud. It supports several infrastructure providers, including the bare-metal provider this solution uses. Cluster lifecycle operations and maintenance are your responsibility, which is the work the orchestration engine automates across sites. If you have on-premises or edge environments with reliable connectivity to an AWS Region, Amazon EKS Hybrid Nodes is the recommended alternative. For the full set of options, see Amazon EKS deployment options.

Architecture overview

High-level architecture showing the AWS orchestration engine, on-premises EKS Anywhere clusters, and the hybrid connectivity linking them

Figure 1: High-level architecture of the hybrid cloud orchestration solution

The architecture consists of three primary layers: a centralized orchestration engine on AWS, distributed on-premises infrastructure running EKS Anywhere clusters, and hybrid connectivity linking the two environments. Serverless technologies coordinate lifecycle operations across hundreds of sites while maintaining comprehensive state tracking through an Inventory Management System.

Foundational concepts

The architecture is built on several foundational concepts that organize how resources are managed, and operations are coordinated.

Site: A physical location or logical grouping housing on-premises infrastructure. Sites provide an organizational framework for distributed operations, supporting location-specific policies, connectivity requirements, and compliance controls (for example, central, regional, or edge data centers).

Server: Bare-metal servers within sites that provide the physical compute, storage, and networking foundation for containerized workloads. Hardware resources are managed through vendor-agnostic Redfish APIs.

Cluster: EKS Anywhere Kubernetes clusters deployed on hardware resources, consisting of both management clusters (for orchestration operations) and workload clusters (for hosting applications).

Order: A trackable infrastructure lifecycle operation that executes as a workflow. When an operator requests an action like rebooting all servers in a site, an order is created with a unique ID. This emits an event, which Amazon EventBridge routes to the corresponding AWS Step Functions workflow. Operators can monitor progress by checking the order status, which is updated in response to state-change events emitted by the running workflow.

Inventory Management System: Centralized state repository

The Inventory Management System is the central state repository, using DynamoDB tables to track infrastructure resources and their relationships across hundreds of distributed sites.

DynamoDB tables maintain information about sites, hardware, clusters, orders, and a catalog of reusable configurations. Sites organize resources by location, storing network configurations, gateway addresses, and regional information. Hardware inventory captures server configurations (BIOS and firmware versions, encrypted credentials), network details (IP addresses, MAC addresses), operational status, physical location (rack number, mounting position), and cluster membership. Clusters maintain Kubernetes configurations, node group details, addon versions, and relationships to management clusters. Orders track operation lifecycles from initiation through completion, capturing the operation type, target resources, execution status, and workflow outputs. The catalog stores vetted blueprints and templates that standardize deployments across the infrastructure.

As infrastructure changes occur, the inventory reflects the current state of resources and their dependencies, acting as the single source of truth for operational history and resource relationships.

Event-driven orchestration engine

With centralized state tracking using the Inventory Management System, the orchestration engine coordinates infrastructure operations through an API-driven framework built on AWS serverless technologies. This architecture delivers scalable, event-driven orchestration without operational overhead.

API layer

The API layer exposes a RESTful interface through Amazon API Gateway for create, read, update, and delete (CRUD) operations on infrastructure resources. A unified operator portal serves as the front end for this API, giving operators a self-service interface to perform lifecycle operations without requiring CLI or direct API knowledge. Lambda functions process incoming requests, validate parameters, and integrate with the order management system to initiate operations.

Orchestration layer

Step Functions executes specialized state machines that integrate with AWS services for compute, storage, and networking operations, providing retry logic, error handling, and state checkpointing.

Step Functions supports a callback pattern where a workflow can pause, hand off a task to an external system with a unique token and resume only when that system calls back with the token. This is critical for hybrid cloud orchestration because it allows workflows to pause execution and wait for external systems to signal completion. This capability addresses the challenge of coordinating AWS-based workflows with on-premises systems that may take hours to complete operations like firmware updates or cluster deployments. A workflow can hand off a task to on-premises infrastructure, pause, and resume only when the on-premises system reports back.

The Distributed Map state scales operations from individual resources to thousands across multiple sites. For example, a workflow that manages the power state of a single server can scale to manage power states across thousands of servers simultaneously.

Amazon EventBridge provides event-driven automation capabilities, triggering workflows based on infrastructure state changes. When inventory records are updated, Amazon EventBridge Rules evaluate the changes and invoke appropriate Step Functions workflows. This decouples components and supports reactive automation patterns, such as automatically scaling clusters when capacity thresholds are reached or starting maintenance workflows when hardware health checks fail.

Security and configuration

Security and configuration management are handled through multiple AWS services. AWS Systems Manager Parameter Store provides centralized configuration storage, while AWS Secrets Manager securely manages sensitive credentials and secrets. AWS Identity and Access Management (IAM) roles provide fine-grained access control across components, with IAM Roles Anywhere extending AWS access to on-premises clusters without requiring long-term credentials.

AWS Systems Manager hybrid activations register on-premises instances with AWS, allowing the Systems Manager agent to manage on-premises infrastructure alongside cloud resources. This delivers a unified management interface for configuration, patching, and command execution across both environments.

AWS Private Certificate Authority manages certificates for secure communications between orchestration components and on-premises infrastructure. Each component operates with least-privilege permissions, accessing only the resources required for its specific function.

Order management: Coordinating operations at scale

Order management flow where an API request maps through Amazon EventBridge rules to Step Functions workflows and updates order status in DynamoDB

Figure 2: Order management flow from API request to workflow execution

The orchestration engine coordinates operations through an order management system built on Amazon EventBridge rules that map API operations to Step Functions workflows. When an API request initiates an operation like `/clusters/{id}/terminate`, an Amazon EventBridge Rule routes the request to the corresponding workflow based on the resource and operation type. The system creates a record in DynamoDB and returns an order ID immediately, while the workflow executes asynchronously.

This event-driven system listens and responds to events throughout the operation lifecycle. As workflows execute, AWS-managed events from Step Functions and custom events from workflow logic progressively update the order status in DynamoDB. This allows operators to initiate operations without waiting for completion, which may take minutes to hours depending on the complexity of the operation.

The following core capabilities are enabled by order management:

Order lifecycle tracking: Operators can query order status through the API to monitor progress and track the complete audit trail from creation through execution to completion or failure.

Callback support: Orders support callbacks to both other workflows and external webhooks. Workflows can trigger other workflows upon completion, while webhook endpoints receive notifications upon state changes or completion. This supports integration with external systems such as ticketing platforms, notification services, or custom dashboards.

Conflict management: Integration with the inventory system prevents conflicting operations by denying new orders if another one is running on the same resource, preventing scenarios like cluster scaling during an upgrade.

Extensibility: New resource types and operations can be added by implementing Step Functions workflows and registering Amazon EventBridge rules that map API endpoints to workflows. The core order tracking logic remains unchanged.

Lifecycle management framework

The lifecycle management framework addresses two primary resource types, each with distinct operational requirements: bare-metal hardware and Kubernetes clusters.

Hardware management

Hardware lifecycle management flow using vendor-agnostic Redfish APIs to run firmware, power, and BIOS operations across on-premises servers

Figure 3: Hardware lifecycle management across distributed sites

The solution provides hardware lifecycle management across distributed on-premises sites through a vendor-agnostic approach integrated with the Inventory Management System.

Supported hardware lifecycle operations

  1. Firmware management: Automated updates and configuration management.
  2. NIC upgrades: Network interface card firmware updates.
  3. Power management: Remote reboot, shutdown, and power cycling.
  4. Health: Processor, memory, and disk health checks.
  5. BIOS configuration: Define and apply specific golden templates.

This approach automates traditional manual hardware management, so operations can efficiently handle hundreds of servers across multiple distributed sites.

Cluster management

Cluster lifecycle management flow where the orchestration engine assembles a configuration file and hardware inventory and runs EKS Anywhere commands through Systems Manager and Batch

Figure 4: EKS Anywhere cluster lifecycle management across sites

Cluster management uses Amazon EKS Anywhere for consistent Kubernetes operations across sites. To create a cluster from bare metal servers, a configuration file and a hardware inventory CSV that lists the servers and their network details are prepared and passed to the EKS Anywhere CLI, which network boots them, installs the operating system and Kubernetes, and brings up the cluster. For the full set of steps and configuration options, see the EKS Anywhere bare metal documentation.

EKS Anywhere supports two cluster types:

  • Management: Dedicated clusters that host orchestration components to manage the lifecycle of workload clusters.
  • Workload: Application-hosting clusters managed by their corresponding management cluster.

This mapping of management to workload clusters is maintained in the Inventory Management System to give a unified view of cluster distribution across the infrastructure. When an operator requests a cluster through the API, the orchestration engine assembles the required inputs: the configuration file comes from a blueprint in the Cluster Catalog, and the hardware CSV comes from the servers recorded in the Inventory Management System. A workflow then runs the EKS Anywhere commands through Systems Manager (SSM) and Batch, which execute them against the on-premises servers.

Scalable operations

Cluster operations must execute in the proper sequence across the distributed environment, handling dependencies between clusters and their components. For instance, cluster creation begins with hardware selection based on placement strategy, pre-flight checks, bootstrapping an Admin machine, executing on-premises commands and awaiting completion, add-ons installation, and post-deployment health checks. The orchestration engine handles this using Step Functions with child workflows, callback patterns, and dependency mapping.

Supported cluster lifecycle operations

  1. Cluster creation: Automated provisioning of management and workload clusters with customizable configurations.
  2. Cluster scaling: Dynamic addition or removal of worker nodes based on capacity requirements.
  3. Cluster upgrades: Coordinated Kubernetes version upgrades with minimal disruption.
  4. Cluster termination: Graceful cluster decommissioning with proper resource cleanup.

These automated workflows reduce the operational complexity of managing Kubernetes at scale and support consistent cluster operations from edge locations to central data centers.

Monitoring and observability

Managing geographically distributed infrastructure requires centralized observability since operators often need to investigate issues across individual sites, correlating data from different hardware vendors and software layers.

This solution addresses the fragmented visibility challenge by aggregating telemetry from on-premises clusters into managed AWS services. AWS Distro for OpenTelemetry (ADOT), deployed as a collector on each EKS Anywhere cluster, scrapes and forwards metrics from the server, Kubernetes, and application layers to Amazon Managed Service for Prometheus in the AWS Region. Amazon Managed Grafana then provides unified dashboards and alerting across the entire distributed environment.

With this approach, operators can monitor server availability (through Redfish events or Prometheus node-exporter), Kubernetes cluster health (through kube-state-metrics), and application-level metrics from one place, regardless of the underlying hardware vendor.

For a detailed implementation walkthrough, including Redfish event subscription patterns, OpenTelemetry collector configuration, Prometheus alerting rules, and Grafana dashboard setup for distributed sites on EKS Anywhere, see our related post: Building observability on Amazon Managed Grafana built on EKS Anywhere.

Hybrid integration patterns

Although EKS Anywhere clusters run on-premises, applications on them can depend on capabilities that span the cloud boundary: DNS resolution across both environments, TLS certificates, access to AWS APIs, and persistent storage. AWS offers services designed for this hybrid integration, and the orchestration engine can apply them automatically from its inventory as clusters and applications change.

Automated DNS management

When the state of a cluster, server, or application changes, Amazon DynamoDB Streams automatically trigger Lambda functions that update DNS records in Amazon Route 53 private hosted zones. Route 53 Resolver endpoints make these records resolvable from both AWS and on-premises, which supports service discovery without manual DNS configuration.

Certificate lifecycle operations

AWS Private Certificate Authority acts as a managed CA for the clusters, so there is no need to run a certificate authority at each site. cert-manager and the AWS Private CA Issuer request, renew, and distribute certificates from it automatically, which helps avoid outages from expired certificates.

Secure AWS access

Workloads on the clusters often need to call AWS APIs, such as sending Fluent Bit logs to Amazon Simple Storage Service (Amazon S3), publishing metrics to Amazon Managed Service for Prometheus, or pulling images from Amazon Elastic Container Registry (Amazon ECR). AWS IAM Roles Anywhere issues short-lived AWS credentials in exchange for a certificate the workload already holds, so no long-lived keys are stored at each site. It accepts that certificate only if it chains to a trusted source, so the orchestration engine registers each cluster’s own CA certificate as its trust anchor when the cluster comes up.

Persistent storage integration

External storage solutions such as Portworx can be integrated for stateful applications. DynamoDB Streams trigger automated interactions with storage provider APIs during node provisioning and cleanup operations and perform the configuration and reclaiming of storage resources.

The event-driven approach makes it possible for dependent infrastructure components to remain synchronized with the actual state of clusters and hardware, reducing operational overhead and minimizing configuration drift.

Conclusion

This blog post explores the architecture and capabilities of the hybrid cloud orchestration solution that modernizes on-premises infrastructure management using AWS technologies and EKS Anywhere. We’ve demonstrated how you can build a scalable, event-driven orchestration engine that manages your distributed infrastructure across hundreds of sites while maintaining operational consistency.

What’s next

In this post series, we’ve focused on the architectural patterns and capabilities that enable enterprise-scale hybrid cloud orchestration. To get started today, review the Amazon EKS Anywhere documentation and set up a bare-metal cluster or use the Docker provider for development and testing. In Part 2, we walk through the implementation of the orchestration solution with infrastructure-as-code templates, Step Functions workflow definitions, and operational runbooks you can adapt to your environment. Follow the AWS Containers blog for the next installment.


About the authors

Build your own continuous modernization pipeline with AWS Transform custom

Post Syndicated from Janardhan Molumuri original https://aws.amazon.com/blogs/devops/build-your-own-continuous-modernization-pipeline-with-aws-transform-custom/

Introduction

Development velocity has reached new heights with AI-driven development tools and practices. Organizations are generating code faster than ever before. But that speed carries risk. Researchers Anderson, Parker, and Tan warned in MIT Sloan Management Review, “Legacy systems tend to carry hidden debt; layering AI-generated code on top of them creates additional tangled dependencies.” The faster you generate code, the faster technical debt compounds — especially in brownfield environments where outdated frameworks, deprecated libraries, and undocumented services already carry years of accumulated risk.

As organizations accelerate their software development, manual or periodic processes to synchronize dependencies and update documentation no longer keep pace, and technical debt piles up faster than ever. Continuous modernization built into your pipeline enables you to maintain up-to-date dependencies and documentation across repositories on every commit, preventing future tech debt and improving AI agent accuracy and accountability.“

You can embed AI-powered code transformations directly into your CI/CD pipelines, turning modernization from a periodic project into an automated, ongoing practice. AWS gives you two ways to get there. AWS Transform – continuous modernization is the fully managed option, delivering continuous modernization automatically with no pipeline for you to build or maintain. The Do-It-Yourself (DIY) approach assembles the same practices yourself using AWS Transform custom and your existing CI/CD platform. Choose DIY when you need to fit modernization into a specific pipeline (GitHub Actions, AWS CodePipeline, Jenkins, GitLab CI, and so on), or want to customize the workflow with existing tools like Dependabot.

In this post, we cover the DIY approach on how to set up a continuous modernization pipeline using AWS Transform custom and demonstrate it in action.

The Do It Yourself (DIY) path – continuous modernization pipeline with AWS Transform custom

Sample application: instrumentShop

For this walkthrough, we use a dated Java application called instrumentShop (Figure 1) — a Java microservices application built with Spring Boot that simulates an online instrument shop to demonstrate four practices: automated dependency remediation, auto-documentation on every commit, scaling transformations across repositories, and continual learning.

Architecture overview
instrumentShop Java application architecture: a Spring Gateway routing traffic to four REST services (Agents, Instruments, Consumers, Products), with a Thymeleaf client, PostgreSQL persistence, and Hystrix circuit breaking.

Figure 1: instrumentShop Java application architecture

The instrumentShop application is a Spring Boot microservices application with a Spring Gateway (v1.5.19) routing traffic from a single HTTP/8010 entry point to four REST services: Agents, Instruments, Consumers, and Products. A Thymeleaf client provides server-side rendering, PostgreSQL 13.1 handles persistence via JDBC, and Hystrix provides circuit-breaking for inter-service calls. A ShopTester utility generates HTTP traffic for testing.

This application is a strong candidate for continuous modernization:

  • Spring Boot 1.5.19 is years past end of life and carries known CVEs
  • Hystrix has been in maintenance mode since Netflix deprecated it in 2018
  • Cross-service coordination — dependency updates must propagate across multiple microservices
  • Transitive dependency risk — PostgreSQL JDBC drivers and other transitive dependencies accumulate security advisories over time

A typical workflow for the continuous modernization pipeline is shown below (Figure 2):

  • A developer pushes code to main — GitHub Actions triggers the auto-documentation workflow, generating updated architecture docs and technical debt reports.
  • Dependabot detects a vulnerable dependency — A PR opens automatically. GitHub Actions triggers the dependency remediation workflow, runs AWS Transform custom to remediate the code, validates with tests, and pushes the result back to the PR.
  • A platform team defines a new transformation (e.g., “Upgrade Spring Boot to the latest stable release “) — The scheduled GitHub Actions workflow runs the transformation weekly in non-interactive mode across all instrumentShop microservices and other repositories in the portfolio.
  • The agent learns — Knowledge items from each execution improve future runs, reducing manual intervention over time.

AWS Transform continuous code modernization workflow
Figure 2: AWS Transform continuous code modernization workflow

Prerequisites

  • Before setting up the continuous modernization pipeline, ensure you have the following:
  • An active AWS account with permissions for AWS Transform custom
  • AWS Transform CLI installed and configured in your development environment
  • Authentication with AWS credentials configured locally and proper IAM permissions to call AWS Transform
  • Git installed for cloning sample repositories
  • GitHub Dependabot enabled on your repository for automated vulnerability detection

Continuous modernization through CI/CD in action

Continuous modernization shifts code transformation from a periodic project into an automated, pipeline-driven practice. Instead of scheduling a “modernization sprint” once a year, your CI/CD pipeline identifies and remediates technical debt on every commit, every dependency alert, and across every repository.

We implement this through four practices, each powered by AWS Transform custom running as a step in GitHub Actions workflows.

Note: This post uses GitHub Actions because the instrumentShop demo repository is built with it. The same AWS Transform CLI (atx) commands work with AWS CodePipeline, Jenkins, GitLab CI, CircleCI, or any CI/CD system that runs shell commands. Continuous modernization is a practice, not a tool choice.

Important: Every atx custom def exec invocation in this post uses the –trust-all-tools flag, which allows the agent to execute tools without interactive confirmation. This is required for non-interactive CI/CD execution. Review your organization’s security policies before enabling this flag in production pipelines.

1. Dependency analysis and remediation

GitHub Dependabot scans your repository for known vulnerabilities and generates alerts when a new vulnerability is added or your dependency graph changes—for example, when you push commits that update packages or versions. However, resolving these alerts requires more than bumping a version number. Upgrading a dependency can introduce breaking API changes, require code modifications, or demand configuration updates.

AWS Transform custom helps handle the code changes needed to resolve the alerts. It runs via a GitHub Actions workflow that triggers automatically to:

  • Fetch the list of latest Dependabot alerts
  • Run AWS Transform custom to analyze the alerts and apply code transformations
  • Run your build and test suite to validate the changes
  • Create a new pull request for each resolved alert

The workflow calls a shell script that invokes the AWS Transform CLI in headless mode with retry logic. Place this script at the root of your repository:

run_dependabot_alert_fixes.sh:

#!/usr/bin/env bash
set -euo pipefail

# -------------------------------------------------------------------
# run_dependabot_alert_fixes.sh
# Runs the Dependabot alert remediation transformation in headless mode.
# Retries up to MAX_RETRIES times on failure.
#
# Usage:
#   ./run_dependabot_alert_fixes.sh [-n <transformation-name>] [-p <path>] [-c <build-command>]
#
# Defaults:
#   -n  Remediate-Critical-GitHub-Dependabot-Alerts-Java-Maven
#   -p  .                   (current directory)
#   -c  mvn clean install   (Maven build)
# -------------------------------------------------------------------

TRANSFORMATION_NAME="Remediate-Critical-GitHub-Dependabot-Alerts-Java-Maven"
CODE_PATH="."
BUILD_CMD="mvn clean install"
MAX_RETRIES=3

while getopts "n:p:c:" opt; do
  case $opt in
    n) TRANSFORMATION_NAME="$OPTARG" ;;
    p) CODE_PATH="$OPTARG" ;;
    c) BUILD_CMD="$OPTARG" ;;
    *) echo "Usage: $0 [-n <transformation-name>] [-p <path>] [-c <build-command>]" && exit 1 ;;
  esac
done

echo "=== AWS Transform Custom ==="
echo "Transformation: $TRANSFORMATION_NAME"
echo "Code path:      $CODE_PATH"
echo "Build command:  $BUILD_CMD"
echo "============================"

attempt=1
while [ $attempt -le $MAX_RETRIES ]; do
  echo "--- Attempt $attempt of $MAX_RETRIES ---"

  if atx custom def exec \
    -n "$TRANSFORMATION_NAME" \
    -p "$CODE_PATH" \
    -c "$BUILD_CMD" \
    -x -t; then
    echo "=== Transformation completed successfully ==="
    exit 0
  fi

  echo "Attempt $attempt failed."
  attempt=$((attempt + 1))

  if [ $attempt -le $MAX_RETRIES ]; then
    echo "Retrying in 10 seconds..."
    sleep 10
  fi
done

echo "=== All $MAX_RETRIES attempts failed ==="
exit 1

This script accepts optional flags to override the transformation name (-n), code path (-p), and build command (-c). The -x flag enables non-interactive mode and -t enables --trust-all-tools, both required for CI/CD execution. On failure, it retries up to three times with a 10-second backoff.

Your CI/CD workflow must configure AWS credentials and install the AWS Transform CLI before invoking this script. With this setup, Dependabot alerts are reviewed continuously for any changes — not just a version bump, but the complete code adaptation required to make the upgrade work.

2. Auto documentation

Documentation is one of the most neglected aspects of modern software development. Documentation increases accuracy and acts as a contract between requirements and implementation. AWS Transform custom codebase analysis capability generates structured documentation covering architecture, technical debt, code metrics, and migration planning on every incremental update ensuring every Agent or human that modifies the codebase is working from a true “current state”.

By embedding this as a post-push step in your CI/CD pipeline, your documentation stays current automatically. The workflow triggers on every pull request to main, runs your build and test suite, then calls a shell script that invokes AWS Transform custom to generate documentation and commits it back to the PR branch.

Place this script at the root of your repository:

run_code_analysis.sh:

#!/usr/bin/env bash
set -euo pipefail

# -------------------------------------------------------------------
# run_code_analysis.sh
# Runs an AWS Transform custom transformation in headless mode.
# Retries up to MAX_RETRIES times on failure.
#
# Usage:
#   ./run_code_analysis.sh [-n <name>] [-p <path>] [-c <build-cmd>] [-U <pr-url>]
#
# Defaults:
#   -n  GitHub-PR-Context-Codebase-Analysis
#   -p  .                   (current directory)
#   -c  mvn clean install   (Maven build)
#   -U  (empty)             PR URL
# -------------------------------------------------------------------

TRANSFORMATION_NAME="GitHub-PR-Context-Codebase-Analysis"
CODE_PATH="."
BUILD_CMD="mvn clean install"
PR_URL=""
MAX_RETRIES=3

while getopts "n:p:c:U:" opt; do
  case $opt in
    n) TRANSFORMATION_NAME="$OPTARG" ;;
    p) CODE_PATH="$OPTARG" ;;
    c) BUILD_CMD="$OPTARG" ;;
    U) PR_URL="$OPTARG" ;;
    *) echo "Usage: $0 [-n <name>] [-p <path>] [-c <build-cmd>] [-U <pr-url>]" && exit 1 ;;
  esac
done

echo "=== AWS Transform Custom ==="
echo "Transformation: $TRANSFORMATION_NAME"
echo "Code path:      $CODE_PATH"
echo "Build command:  $BUILD_CMD"
echo "PR URL:         $PR_URL"
echo "============================"

attempt=1
while [ $attempt -le $MAX_RETRIES ]; do
  echo "--- Attempt $attempt of $MAX_RETRIES ---"

  if atx custom def exec \
    -n "$TRANSFORMATION_NAME" \
    -p "$CODE_PATH" \
    -c "$BUILD_CMD" \
    -g "additionalPlanContext=$PR_URL" \
    -x -t; then
    echo "=== Transformation completed successfully ==="
    exit 0
  fi

  echo "Attempt $attempt failed."
  attempt=$((attempt + 1))

  if [ $attempt -le $MAX_RETRIES ]; then
    echo "Retrying in 10 seconds..."
    sleep 10
  fi
done

echo "=== All $MAX_RETRIES attempts failed ==="
exit 1

This script accepts optional flags for the transformation name (-n), code path (-p), build command (-c), and PR URL (-U). Pass the PR URL to the agent via the -g flag as additionalPlanContext, giving it awareness of the pull request context when generating documentation. On failure, it retries up to three times with a 10-second backoff.

Your CI/CD workflow must configure AWS credentials and install the AWS Transform CLI before invoking this script. The workflow commits the generated documentation back to the PR branch automatically, keeping your architecture docs and technical debt reports current with every code change.

Every push now updates the documentation (Figures 3 and 4) — reducing knowledge silos and preserving institutional knowledge.

A GitHub pull request triggering the auto-documentation workflow.

Figure 3: PR triggering auto-documentation

Generated documentation output showing architecture and technical debt reports.

Figure 4 – Generated documentation output

3. Scale across repositories

For organizations with hundreds of microservices, transforming one repository at a time doesn’t scale. AWS Transform custom non-interactive mode combined with GitHub Actions matrix strategy allows you to orchestrate transformations across your entire portfolio in parallel. You can run them on demand or on a recurring schedule, so modernization runs as a continuous practice rather than a one-time project.

# .github/workflows/scale-modernization.yml
name: Scale Modernization
on:
  schedule:
    - cron: '0 6 * * 1'
  workflow_dispatch:

jobs:
  transform-repos:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        repo:
          - magnefique-studios/instrumentShop
          - magnefique-studios/orderService
          - magnefique-studios/paymentGateway
    steps:
      - name: Checkout ${{ matrix.repo }}
        uses: actions/checkout@v4
        with:
          repository: ${{ matrix.repo }}
          token: ${{ secrets.GH_PAT }}

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1

      - name: Install ATX CLI
        run: curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash

      - name: Run transformation
        run: |
          atx custom def exec \
            --transformation-name "spring-boot-3-upgrade" \
            --code-repository-path "." \
            --build-command "mvn clean install" \
            --non-interactive \
            --trust-all-tools

Tip: GitHub Actions matrix strategy runs each repository in parallel automatically — no separate orchestration layer needed. For larger portfolios, you can also wrap this in AWS Batch or AWS Fargate for large-scale parallel execution. The AWS Transform web console tracks progress across all repositories in a single view.

4. Continual learning

Each time AWS Transform custom completes a transformation, a memory agent scans the full execution trajectory and extracts lessons. Lessons include patterns that the agent learned, decisions that the agent made during planning, and feedback you provide during execution. AWS Transform custom automatically attaches these lessons to your transformation definition, which improves accuracy in subsequent runs.

AWS Transform custom applies lessons automatically, and each lesson belongs to a category that groups related lessons for review. You can browse and archive any lesson you do not want AWS Transform custom to apply to future runs.This keeps a human in the loop on what the agent “remembers” which matters when the same transformation runs across many repositories with different conventions.

In practice, this means your “Spring Boot 3 Upgrade” transformation gets sharper with each execution. The first repository surfaces the edge cases; once you review the resulting lessons and archive the ones that do not fit, subsequent runs handle those edge cases without intervention.

For production use, you can combine these practices into a single workflow file:

Note: The individual workflows shown in Practices 1–3 are presented separately for clarity. Combine them into a single workflow file as shown here, or keep them as separate workflow files depending on your team’s preference.

# .github/workflows/continuous-modernization.yml
name: Continuous Modernization
on:
  push:
    branches: [main]
  pull_request:
    types: [opened]
  schedule:
    - cron: '0 6 * * 1'

jobs:
  dependency-remediation:
    if: github.actor == 'dependabot[bot]'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.head_ref }}
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1
      - name: Install ATX CLI
        run: curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash
      - name: Remediate dependency changes
        run: |
          atx custom def exec \
            --transformation-name "dependency-remediation" \
            --code-repository-path "." \
            --build-command "mvn clean install" \
            --non-interactive \
            --trust-all-tools

  auto-documentation:
    if: github.event_name == 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1
      - name: Install ATX CLI
        run: curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash
      - name: Generate documentation
        run: |
          atx custom def exec \
            --transformation-name "codebase-documentation" \
            --code-repository-path "." \
            --build-command "echo 'docs-only'" \
            --non-interactive \
            --trust-all-tools

  weekly-modernization:
    if: github.event_name == 'schedule'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: us-east-1
      - name: Install ATX CLI
        run: curl -fsSL https://transform-cli.awsstatic.com/install.sh | bash
      - name: Run modernization scan
        run: |
          atx custom def exec \
            --transformation-name "tech-debt-analysis" \
            --code-repository-path "." \
            --build-command "mvn clean install" \
            --non-interactive \
            --trust-all-tools

Conclusion

Continuous modernization moves code transformation out of periodic sprints and into your CI/CD pipeline. By combining GitHub Dependabot’s vulnerability detection with AWS Transform custom agent, orchestrated through GitHub Actions, you can:

  • Remediate dependency vulnerabilities automatically — beyond version bumps to full code adaptation
  • Keep documentation current with every commit, preserving institutional knowledge
  • Scale transformations across hundreds of repositories with consistent quality
  • Improve continuously as the agent accumulates knowledge items from each execution

The instrumentShop sample application demonstrates that even a moderately complex microservices architecture — with end-of-life Spring Boot versions, deprecated libraries like Hystrix, and multiple interconnected services — can be continuously modernized without dedicated modernization sprints.

Ready to get started? This post walked through the do-it-yourself path with AWS Transform custom. If you would rather have continuous modernization delivered as a fully managed service, explore AWS Transform continuous modernization. Either way, visit the AWS Transform documentation to start your continuous modernization journey.

Janardhan Molumuri

Janardhan Molumuri is a Principal Technical Leader at AWS with over two decades of engineering leadership experience, advising customers on cloud and AI Adoption strategies and emerging technologies including generative AI. He has passion for thought leadership, speaking, writing, and enjoys exploring technology trends to solve problems at scale.

Maxine Rosa

Maxine Rosa is a Sr World Wide Generative AI Specialist at AWS focused on developer tooling including AWS Transform and Kiro. With a background in Software Engineering, Solution Engineering and Go-to-Market strategy, she helps AWS customers adopt Generative AI tooling into their current Software Development Lifecycle.

Kola Akinnibi

Kola Akinnibi is an Associate Solutions Architect at AWS focused on observability, partnering with ISVs and large enterprises to bring end-to-end monitoring to AI agents and modern applications. He helps customers design observability solutions that scale, and has a passion for sharing technical content.

Renuka Krishnan

Renuka Krishnan is a Senior Specialist Solutions Architect at AWS, specializing in code modernization using agentic AI and AWS services. She has over 15 years of experience architecting and implementing solutions, and works with customers to accelerate application development and modernization through AI-powered solutions.

Venugopalan Vasudevan

Venugopalan Vasudevan (Venu) is a Principal Specialist Solutions Architect at AWS, where he leads Generative AI initiatives focused on Amazon Q Developer, Kiro, and AWS Transform. He helps customers adopt and scale AI-powered developer and modernization solutions to accelerate innovation and business outcomes.

Closing the AI agent trust gap with graduated autonomy

Post Syndicated from Dev Arora original https://aws.amazon.com/blogs/architecture/closing-the-ai-agent-trust-gap-with-graduated-autonomy/

How much to trust an AI agent is now a daily operational question. Agents read customer data, open tickets, process refunds, and delete accounts, yet most teams pick up a binary: full access or read-only. Full access is risky because agents fail unpredictably. Read-only leaves most of the agent’s value unused. The distance between what an agent could do and what an operator trusts it to do is the agent’s trust gap.

In this post, we describe graduated autonomy, an architectural pattern that closes the gap. Agents earn expanded permissions through sustained reliability and lose them when performance degrades. Amazon Bedrock AgentCore, a platform to build, connect, and optimize agents at scale with any framework or model, provides the runtime, gateway, policy, and evaluation capabilities. Amazon DynamoDB stores trust state. AWS CodePipeline gates delivery on evaluation results. We cover each layer’s responsibility and the key design decision behind it.

The agent trust gap

Identity and access management answers “who can do what?” once, at provisioning. That model assumes that the principal behaves consistently. A large language model agent breaks it: the same agent can be accurate Monday and hallucinated Tuesday after a prompt change or model update.

Closing the gap requires three capabilities raw API logs rarely provide:

  • Visibility. API logs tell engineers what happened but tell a compliance officer nothing about whether an action was safe.
  • Decision provenance. Tracing an action back to the signal that triggered it, the alternatives considered, and the confidence held.
  • Reversibility. Pre-action state capture, so operators can recover from incorrect actions.

The framework that implements this pattern delivers all three through six architectural layers.

Solution overview

The six layers:

  • Scoring engine computes trust from configurable dimensions.
  • Tier system translates sustained scores into autonomy levels.
  • Pre-execution layer blocks dangerous actions before they run.
  • Enforcement layer applies tiers through Cedar policies at the infrastructure level.
  • Post-execution layer evaluates outcomes, records of provenance, and feeds signals back to scoring.
  • Delivery gate keeps degraded agent versions out of production.
Architecture diagram of the trust framework as a clockwise closed loop: the scoring engine produces a weighted trust score from five dimensions, the tier system converts sustained scores into autonomy tiers T1 through T4, the pre-execution and enforcement layers apply the current tier through in-process checks and Cedar policies, and the post-execution layer returns outcome scores, honeypot results, and human overrides to the scoring engine, with an audit trail at the center recording every decision.

Figure 1: The trust framework’s closed loop.

Each layer is replaceable: the scoring model, tier thresholds, pre-execution signals, and evaluation criteria are configuration, not code. Each layer also embodies one deliberate design decision, developed in the following sections:

Layer Key design decision
Scoring engine Safety is an independent floor, never averaged away by strong metrics
Tier system Start every agent at T1. Promote slowly, demote immediately
Pre-execution layer Fast in-process filters are backstopped, never solely trusted
Enforcement layer Deny by default, enforced outside the agent’s process
Post-execution layer Audit records capture pre-action state, making recovery possible
Delivery gate One unauthorized tool call in adversarial tests blocks release

The scoring engine

The scoring engine computes a weighted score from 0 to 100 per agent over a rolling window of 50 actions, from five dimensions:

Dimension Weight What it measures
Accuracy 25% Task completion correctness against expected outcomes
Safety 20% Boundary respect, adversarial content detection, permitted tool adherence
Consistency 20% Behavioral predictability, inverse of tool-use pattern drift
Compliance 20% Reasoning quality before acting, guardrail adherence
Efficiency 15% Execution without unnecessary retries or resource waste

The composite drives dashboards and tier assignment, but safety acts as an independent floor, so a dangerous individual metric never hides strength elsewhere.

The tier system

Every new agent starts at T1, regardless of test performance:

Tier Score range Permissions
T1: Probation 0 to 40 Read and list only. Two tools visible.
T2: Supervised 41 to 70 Add write operations. Human approves high-risk.
T3: Trusted 71 to 90 Execute and modify. Anomalies flagged for review.
T4: Autonomous 91 to 100 Full access. Post-hoc audit only.

Three rules govern transitions:

  • Promotion requires sustained performance. The score must stay above the promotion threshold for the entire rolling window.
  • Demotion is immediate. When safety drops below its floor or injection is detected, the agent moves down.
  • Hysteresis prevents oscillation. Promotion into a tier requires a score 5 points above that tier range floor. Demotion happens at the range floor itself. An agent at a boundary cannot flap between tiers.

Trust state lives in Amazon DynamoDB as a current state record plus a time-series history per agent. Enforcement components read the current tier on every invocation, a lookup DynamoDB typically serves in single-digit milliseconds.

The pre-execution layer

Post-execution evaluation cannot undo damage, so the pre-execution layer evaluates every tool’s call and can block it before execution. It scores six signals independently:

  • Adversarial injection detection. Pattern matching against known injection phrases. One match triggers an instant block and a trust penalty.
  • Sensitive target detection. Regex matching credentials, tokens, and private keys in tool arguments.
  • Dangerous tool detection. Flagging tools that match destructive operation patterns.
  • Behavioral consistency. Comparing the current tool call against the agent’s historical tool-use distribution.
  • Confidence calibration. Comparing stated confidence against historical accuracy. Overconfident failures are penalized at twice the normal rate.
  • Reasoning quality. Checking whether the agent provided reasoning before acting.

These checks are fast first-pass filters, not a complete defense. The enforcement layer’s deny-by-default policies backstop anything they miss.

The enforcement layer

The pre-execution layer is application code inside the agent’s process. The enforcement layer operates outside the agent, at the infrastructure level.

AgentCore Gateway, a capability of Amazon Bedrock AgentCore, sits between the agent and its tools. It routes every MCP tool invocation through Policy in Amazon Bedrock AgentCore, which evaluates Cedar policies with forbid-wins semantics. One satisfied forbid overrides any number of permits. Tier maps to policy state:

  • Probation: A forbid policy blocks write, execute, and delete tool actions.
  • Promotion: The forbid policy is removed, and broader permits take effect.
  • Demotion: The forbid policy is re-applied.

With the policy engine in enforce mode, the Gateway lists only tools that policy could permit, so the tier’s unconditional forbids keep blocked tools out of the listing. The agent is unlikely to call a tool it has never seen. Listing is a meta-action: each invocation is still evaluated separately with full request context, including input parameters. Cedar denies by default. Enforcement never depends on the agent’s choosing to behave. For model-level content safety, Amazon Bedrock Guardrails complements Policy in AgentCore, filtering harmful content and masking sensitive information independent of tier.

The post-execution layer

After every tool call, the system scores the outcome across eight signals, from confidence calibration and behavioral drift to human overrides and retry detection. Every action generates an audit record following the Think, Plan, Act, Observe, Score chain:

  • Think: The agent’s reasoning chain.
  • Plan: Tool selected, input prepared, pre-execution score.
  • Act: Cedar policy matched, Gateway route processed.
  • Observe: Success or failure, output data.
  • Score: Trust impact, per-dimension scores, tier change.

The Plan and Act records capture pre-action state, which is what makes recovery from an incorrect action possible. Operators ask questions in plain English, and a provenance query endpoint returns a human-readable explanation of any decision. Audit entries persist to DynamoDB.

The delivery gate

Each change to the agent’s prompt, configuration, or tool definitions triggers an AWS CodePipeline run. The run deploys the candidate to staging and runs it against ground-truth fixtures with Amazon Bedrock AgentCore Evaluations, a capability of Amazon Bedrock AgentCore. The fixtures include adversarial cases such as prompt injection and data-exfiltration requests. A single unauthorized tool call in any adversarial case fails the gate. The version that passes becomes the last known stable version.

Production monitoring and recovery

The framework injects synthetic honeypot cases with known expected behavior into a small share of traffic. Validation checks the tool-call trajectory (expected tools, expected order, no forbidden tools) rather than nondeterministic natural-language output, so a mismatch signals a real anomaly. Honeypot results stay out of production metrics. When safety drops below the floor, demotion narrows the agent’s permissions, and the framework redeploys the last known stable version. Together they restore known-good code alongside a tighter permission set. The framework also alerts operators.

Operator judgment feeds directly: the rolling rate at which operators reject proposed actions caps the effective safety metric, so 30 percent rejections cap safety at 70. An emergency stop pushes a single Cedar deny-all policy. Once the policy is active, typically within seconds, the Gateway denies all tool invocations without a redeployment. In multi-agent systems, a delegated action’s effective tier is the minimum across the delegation chain, closing the delegation privilege-escalation path.

Conclusion

In this post, we described graduated autonomy, an architectural pattern for closing the agent trust gap. With this pattern in place, your agents hold the autonomy their track record supports.

To get started, take the dimension weights and tier boundaries from the two tables in this post as a starting template for one agent in your fleet. Start that agent at T1. Then follow the Amazon Bedrock AgentCore Evaluations documentation to build the delivery gate, and the Policy in Amazon Bedrock AgentCore documentation to write the tier policies. You can explore these capabilities in the Amazon Bedrock console and on the Amazon Bedrock AgentCore detail page.

For deeper dives into the building blocks this pattern uses, read Secure AI agents with Policy in Amazon Bedrock AgentCore and Build custom code-based evaluators in Amazon Bedrock AgentCore.


About the authors

Amazon MSK Service 101: How many partitions does an Amazon MSK topic need?

Post Syndicated from Yashika Jain original https://aws.amazon.com/blogs/big-data/amazon-msk-service-101-how-many-partitions-does-an-amazon-msk-topic-need/

Customers new to Amazon Managed Streaming for Apache Kafka (Amazon MSK) often ask how many partitions their topics need. Choosing the right partition count is one of the most impactful architectural decisions you make, because it directly affects throughput, scalability, and operational complexity.

In Apache Kafka, a topic is the fundamental unit for categorizing data streams, but to achieve high scalability and performance, Kafka divides topics into smaller, independent units called partitions.

In this post, we provide practical guidance for determining the ideal partition count for your use case.

Understanding Kafka partitions

In
Apache Kafka, a partition is the unit of storage and parallelism. Each partition is an ordered, immutable log that can store records as they are produced to a topic. When you create a topic, Kafka distributes its partitions across the brokers in the cluster. Partitions allow Kafka to scale in three key ways:
  • Parallelism – Within a consumer group, each partition can be read by only one consumer at a time. Each partition maps to a dedicated log file in storage on the broker, and Kafka manages these logs through separate processing threads. This architecture allows more partitions to support more consumers processing data in parallel, with each partition’s log being independently managed for read and write operations.
The following diagram shows how Kafka distributes partition replicas across a three-broker cluster, with each broker serving as a leader for some partitions and a follower for others.
Partitions 0, 1, and 2 replicated across three brokers, each a leader for some partitions and a follower for others

Figure 1: Partition replicas distributed across a three-broker cluster

The following diagram illustrates how producers append new records to the end of a partition log, while consumers read sequentially from their current offset position.

Producers append records to the tail of partition logs while consumers read sequentially from their offset position

Figure 2: Producer writes and consumer offset positions in two partition logs

  • Throughput – Producers and consumers can read and write data in parallel across partitions, increasing overall throughput.
  • Scalability – Partitions allow Kafka to spread data and load across multiple brokers instead of concentrating it on a single node.

However, increasing partitions comes with trade-offs. Each partition adds metadata overhead, consumes memory, and requires file handles on the broker. While more partitions improve throughput and parallelism, they also increase the operational burden on the cluster. Too many partitions can lead to longer leader election times during broker failures, increased end-to-end latency, and higher memory consumption for both producers and consumers managing connections to multiple partitions.

Trade-offs when choosing partition count

Choosing a partition count is a balancing act between parallelism and resource utilization.

Benefits of more partitions

Using more partitions can significantly improve throughput by allowing Kafka to distribute read and write traffic across more brokers. This is particularly useful for high-volume ingestion pipelines and real-time analytics workloads. More partitions also allow consumer groups to scale horizontally, because the maximum number of active consumers in a group is limited by the number of partitions. In addition, choosing a partition count that is evenly divisible by the number of brokers helps provide balanced leadership and replica distribution, reducing the risk of uneven load.

Operational costs of more partitions

However, higher partition counts also come with costs. When a broker fails or undergoes maintenance, Kafka must perform recovery operations for each affected partition. During recovery, Kafka elects new leaders for partitions that were hosted on the unavailable broker and replicates data from the remaining in-sync replicas to newly assigned brokers. This process involves copying partition data across the network to restore the replication factor, which can be resource intensive. As the number of partitions increases, these recovery operations take longer because each partition requires its own leader election and data replication cycle.

You might encounter clusters with very high partition counts that experience extended recovery times during rolling upgrades, even when overall traffic volumes are modest. Amazon MSK Express brokers address this challenge by recovering 90x faster and providing 180x faster elasticity when scaling out clusters. This significantly reduces the operational impact of high partition counts during maintenance windows and failure scenarios.

Infrastructure cost implications

Beyond operational complexity, more partitions can directly increase infrastructure costs. Amazon MSK publishes partition-per-broker limits that vary by instance type. When the total partition count (including replicas) exceeds what the current broker fleet can support, you must add brokers to stay within recommended limits, even if throughput alone does not warrant the additional capacity.

Amazon MSK partition-per-broker guidelines

Amazon MSK publishes recommended partition-per-broker guidelines to help you operate clusters reliably. These values are strict limits. Exceeding them can lead to operational challenges, particularly during broker replacement or rolling upgrades, and can block cluster operations such as configuration updates or scaling down.

Express brokers support up to 5x more partitions per broker compared to Standard brokers. For example, the largest Standard broker (kafka.m7g.16xlarge) supports a recommended maximum of 4,000 partitions per broker. The equivalent Express broker (express.m7g.16xlarge) supports up to 20,000 recommended partitions per broker. This higher partition density means partition-bound workloads can be hosted on fewer brokers, improving price-performance by up to 50% for such workloads.

We recommend setting Amazon CloudWatch alarms on PartitionCount per-broker metrics to proactively monitor your partition distribution. When an alarm triggers, evaluate your partition strategy and consider rebalancing partitions across brokers, consolidating topics, or scaling out your cluster to stay within recommended limits. For detailed guidance, see Right-size your cluster: Number of partitions per Standard broker and Express broker partition quota.

Practical guidance for choosing a partition count

There is no single formula that works for every Kafka workload. In practice, you typically combine several considerations when sizing partitions.

  • Start with throughput requirements – The first step is to determine your per-partition throughput capacity, which then informs how many partitions you need.

For Express brokers, use the per-broker throughput capacity as the primary means for sizing your cluster. Express brokers feature a fully managed storage layer, so you do not need to separately account for storage I/O constraints. The published per-broker limits represent the effective capacity available to your workload.

For Standard brokers, the achievable throughput depends on additional factors beyond the broker instance size. These factors include provisioned EBS storage throughput, the number of consumer groups reading from the broker, and how much data is served from memory versus disk. Storage I/O is consumed when producers write, when data replicates between brokers, and when consumers read data that is not in memory. For this reason, validate the effective per-partition throughput for Standard brokers through load testing in your environment.

Once you know your per-partition throughput, calculate the required number of partitions: Number of partitions = Peak throughput of the topic ÷ Throughput per partition

For example, if a topic must handle 40 MB/sec at peak and your testing shows each partition can sustain 5 MB/sec, you would need: 40 ÷ 5 = 8 partitions. Always validate these assumptions with load testing, as actual throughput varies based on your workload characteristics. For initial sizing estimates, refer to the Amazon MSK Sizing and Pricing worksheet and the Amazon MSK Best Practices documentation.

  • Consider your consumer parallelism needs – If you know the number of consumers required during peak processing times, use that as your partition count. We don’t recommend having more active consumers in a consumer group than partitions. For example, if you have 5 partitions, only 5 consumers can actively process data. Additional consumers remain idle. These idle consumers still maintain active TCP connections to the brokers, sending frequent heartbeats and group coordination requests. This might result in unnecessary overhead on broker resources and contribute to high CPU usage despite low egress traffic.
Consumer group with more consumers than partitions, leaving the extra consumers idle

Figure 3: Idle consumers when a consumer group has more consumers than partitions

  • Producer throughput and partition keys – When sizing partitions, consider producer-side throughput in addition to consumer parallelism. If producers generate data faster than a single partition can handle, additional partitions can help distribute write traffic across brokers. Partition keys also play a critical role. Poorly distributed or low-cardinality keys can create hot partitions and limit throughput. In such cases, increasing the number of partitions alone does not improve throughput unless records are evenly distributed.
  • Plan for even distribution and future growth – Kafka works best when partitions can be spread evenly across brokers. Instead of focusing on specific numbers, aim for partition counts that divide reasonably well across your expected broker count. This reduces reassignment churn when brokers are added or replaced. But avoid excessive over-partitioning. It’s reasonable to leave some headroom for future growth. However, creating thousands of partitions “just in case” often causes more harm than good. Increasing partitions later is supported, but it can affect ordering guarantees and may require consumer changes. Start with a conservative number, monitor real traffic patterns, and scale gradually.

From an operational perspective, Amazon MSK provides recommended partition-per-broker guidelines based on broker instance type. Exceeding these guidelines increases operational risk and can block cluster operations such as version upgrades, scaling, or configuration changes. Large partition counts can also increase consumer group rebalance duration, temporarily pausing message processing and increasing end-to-end latency.

Keep in mind that partitioning improves scalability, but it does not address application-level bottlenecks such as slow consumers, inefficient processing logic, or downstream system constraints.

Conclusion

Determining the right number of partitions for an Amazon MSK topic is a foundational design decision. It affects throughput, scalability, failure recovery, and day-to-day operability of your Kafka cluster. Start by understanding your throughput and consumer parallelism needs, respect Amazon MSK partition-per-broker guidelines, avoid excessive over-partitioning, and validate assumptions through load testing. Most importantly, there is no universal “correct” number, only a number that fits your workload, operational goals, and cost.

For more information, see the Amazon MSK Developer Guide and Recommended best practices for Amazon MSK.


About the authors

Yashika Jain

Yashika Jain

Yashika is a Senior Cloud Analytics Engineer at AWS, specializing in real-time analytics and event-driven architectures. She is committed to helping customers by providing deep technical guidance, driving best practices across real-time data platforms and solving complex issues related to their streaming data architectures.

Ali Alemi

Ali Alemi

Ali is a Principal Streaming Solutions Architect at AWS. Ali advises AWS customers with architectural best practices and helps them design real-time analytics data systems which are reliable, secure, efficient, and cost-effective. Prior to joining AWS, Ali supported several public sector customers and AWS consulting partners in their application modernization journey and migration to the Cloud.

From clickops to governed IaC: CloudFormation drift detection in practice

Post Syndicated from Leen Alattas original https://aws.amazon.com/blogs/devops/from-clickops-to-governed-iac-cloudformation-drift-detection-in-practice/

AWS environments that have grown organically over time often share a common characteristic: infrastructure provisioned through the AWS Management Console, SDKs, or CLI without corresponding Infrastructure as Code (IaC) templates. This practice is commonly referred to as “ClickOps,” a term describing any infrastructure change made outside of a codified, version-controlled workflow. Whether changes happen through the console, the AWS CLI, or application SDKs, the result is the same: resources exist without a declarative template to describe their intended state. 

Over time, these manual changes accumulate, creating environments where Amazon Virtual Private Cloud (Amazon VPC) configurations, Amazon Elastic Compute Cloud (Amazon EC2) instances, and Amazon Simple Storage Service (Amazon S3) buckets exist without a single AWS CloudFormation template to describe them. 

Organizations that find themselves in this position have a clear opportunity. CloudFormation’s IaC Generator provides a practical starting point for bringing existing infrastructure under declarative management. It scans an AWS account and produces CloudFormation templates from existing resources, solving the first and most fundamental challenge: you cannot govern infrastructure you cannot see. 

However, generating a template is only the beginning. What follows is the operational thinking behind turning a generated template into something a team can govern and automate: the decisions, trade-offs, and organizational habits that determine whether IaC adoption succeeds long-term. 

IaC Generator: making the invisible visible 

CloudFormation’s IaC Generator scans an AWS account and produces CloudFormation templates from existing resources: Amazon VPCs, subnets, Amazon EC2 instances, Amazon S3 buckets, AWS Identity and Access Management (IAM) roles, and more. It solves the foundational problem of any ClickOps-to-IaC migration: establishing visibility into what exists and how it is configured. 

How it works at a high level 

Scan — IaC Generator discovers resources in the account by querying AWS Cloud Control API, identifying what exists regardless of how it was provisioned. 

Generate — It produces CloudFormation templates that represent the current state of those resources, mapping properties, dependencies, and relationships. 

Review — Teams evaluate the generated templates, reconcile any gaps, and decide how to bring each resource under management. 

This process eliminates weeks of manual documentation work. Instead of engineers mapping infrastructure by hand, IaC Generator produces a baseline in minutes. For a team managing 200+ resources across multiple VPCs, this can compress a multi-sprint effort into a single planning session. 

Understanding what the generator produces 

The generated templates capture the current state of resources, including every manual configuration and accumulated change. Before acting on a generated template, teams should understand what it represents and what it does not. 

Important: IaC Generator does not cover all resource types supported by CloudFormation. Before committing to an import path for any resource, verify that the resource type is supported. Coverage continues to expand, but teams should confirm support for their specific resource types before planning their migration approach. 

The generated template provides an inventory of infrastructure and surfaces implicit dependencies that were never documented. However, a template in a repository does not prevent out-of-band changes, enforce review processes, or protect against drift. Visibility is the prerequisite for control, not a substitute for it. 

Import or recreate: making the right decision for each resource 

When bringing existing resources under CloudFormation management, teams must decide on a per-resource basis whether to import a resource into a stack or to recreate it cleanly. The right choice depends on the specific characteristics of each resource: its criticality, how much operational disruption is acceptable, the complexity of its dependencies, and the technical limitations of the tooling. CloudFormation does not support partial adoption of an existing resource: a resource is either fully imported into a stack or newly provisioned through a stack. This is what makes the decision binary and per-resource rather than incremental. 

A note on configuration drift in this context: configuration drift occurs when the actual state of a resource diverges from what is defined in a template. A resource that was provisioned manually may be in a perfectly valid operational state, but it has no template against which to measure compliance. The goal of importing is to establish that baseline, not to imply the current configuration is inherently flawed.

Factor  Import existing resource  Recreate with new stack 
Resource criticality  High: production, live data, tight dependencies  Lower: dev/test, stateless, easily replaceable 
Manual changes  Significant: many out-of-band modifications  Minimal: resource is close to desired state 
Downtime tolerance  Zero: any interruption is unacceptable  Acceptable: brief maintenance window tolerable 
Template fidelity  Lower: generated template may be imperfect  Higher: full control over the final template 
Dependency complexity  High: cross-service dependencies difficult to isolate  Lower: resource can be isolated and rebuilt cleanly 

Technical limitations to consider 

Beyond operational factors, the IaC Generator has technical constraints that should inform the import-versus-recreate decision: 

  • Resource type coverage: Not all resource types supported by CloudFormation are supported by IaC Generator. Before committing to an import path, verify that the specific resource types are supported. If a critical resource type is not covered, the template must be written manually. 
  • Write-only properties: Some resource properties (such as passwords or secrets) are write-only and cannot be read back during scanning. Generated templates show placeholder values for these, requiring manual reconciliation. In production environments, this may require integration with AWS Secrets Manager or a similar secrets management solution. 
  • Hard-coded values: Generated templates produce literal values rather than parameterized inputs. Plan for a refactoring pass to introduce parameters, mappings, and conditions. 
  • Cross-account and cross-region references: IaC Generator operates within a single account and region. Resources with dependencies spanning accounts or regions require additional manual template work. 

For production resources, stateful workloads, and resources with complex dependency graphs, import is generally the appropriate default. The import operation brings resources under CloudFormation management without recreating them, preserving their current state. The trade-off is that the generated template becomes the starting point, and teams must reconcile any gaps between that template and actual resource state before making subsequent changes. 

Recreation is more appropriate when a resource can tolerate a brief maintenance window, when accumulated manual changes make a clean start more efficient than reconciliation, or when the architecture is being redesigned as part of the migration. 

The most effective approach is to segment the inventory by resource type, criticality, and configuration complexity, then match the strategy to each segment. An Amazon VPC that has been modified extensively over three years presents a different challenge than an Amazon S3 bucket created last month. 

Organizing stacks for operational reality 

A common challenge after bringing resources under CloudFormation management is determining the appropriate stack boundaries. Placing all resources into a single monolithic stack creates operational risk: changes to VPC and subnet infrastructure can inadvertently affect application resources, a rollback on an application deployment can revert infrastructure changes, and accountability becomes diffuse. When ownership is unclear, incident response slows. 

Organizing stacks around lifecycle, ownership, and change frequency addresses this challenge. The key principle is to group resources that share the same rate of change and the same responsible team: 

When these criteria conflict, ownership takes precedence: a shared resource should reside in the stack owned by its primary responsible team, with cross-stack references providing access to consuming teams. 

  • VPC and subnet infrastructure changes infrequently and is typically managed by a platform or infrastructure team. 
  • Application infrastructure changes frequently and is managed by the application teams that deploy to it. 
  • Security controls warrant their own stacks under security team ownership, insulated from application deployment cycles. 

Cross-stack references, through CloudFormation exports and imports, preserve these boundaries while maintaining relationships between stacks. A VPC stack exports Amazon VPC and subnet IDs; application stacks import them. This separation means that application deployments do not modify network configuration, and VPC or subnet changes do not require redeploying application stacks. 

Note: this separation does not eliminate all cross-cutting concerns. Changes to security groups or network ACLs, for example, may still require coordination with application teams. The goal is to reduce unintended coupling, not to eliminate all interdependency. 

This structure makes governance at scale tractable. When stacks have clear boundaries and named owners, drift detection becomes actionable. Teams know exactly who owns a drifted resource and who needs to respond. 

Drift detection: moving from reactive to continuous 

Defining drift: Configuration drift occurs when the actual state of a resource diverges from what is declared in its CloudFormation template. Drift can originate from manual console changes, AWS CLI or SDK operations, automated processes that modify resources outside of CloudFormation, or any action that bypasses the IaC workflow. Drift is not inherently a failure; it often reflects legitimate operational decisions made under time pressure. The challenge is maintaining awareness of these changes so they can be evaluated and reconciled deliberately. 

CloudFormation’s native drift detection tells teams whether resources match their templates. What it cannot do on its own is provide continuous monitoring. Manual, on-demand checks are valuable, but they are reactive. By the time a team runs one, the drift may have already caused a downstream issue. 

Automating drift detection with Amazon EventBridge 

Continuous drift detection requires three capabilities: scheduled detection runs, event capture when drift is found, and routing of alerts to the appropriate team. Amazon EventBridge provides the orchestration layer that connects these capabilities: 

  • Schedule drift detection: Configure an EventBridge rule with a cron expression to trigger the DetectStackDrift API on critical stacks at regular intervals (for example, every 6 hours for production stacks, daily for non-production). This is a custom configuration, not a built-in default; teams define the schedule based on their operational requirements. 
  • Capture drift events: CloudFormation emits events to the default EventBridge event bus when drift detection completes. Create rules that filter for CloudFormation Stack Drift Detection Status Change events where the drift status is DRIFTED. 
  • Automated remediation (with caution): For well-understood, low-risk drift patterns in non-production environments, EventBridge can trigger an AWS Lambda function that applies a drift-aware change set. However, automated remediation in production environments requires careful consideration. See the guidance below on remediation policy. 

Remediation policy: a deliberate decision 

Whether drift triggers a notification or an automated correction should be a deliberate, documented policy decision. Several factors argue for caution with automated rollbacks: 

  • Drift is typically detected well after it occurred. The change was not random; a person or process determined it was necessary at the time. 
  • Automatically reverting a change without understanding why it was made can reintroduce the problem it was intended to solve. 
  • In production environments, the safest default is to alert the owning team and let them evaluate whether the drift should be reconciled into the template or reverted. 

Automated remediation is most appropriate in controlled environments (development, staging) or for narrowly-scoped, well-understood drift patterns where the risk of unintended consequences is minimal. 

Drift-aware change sets 

Drift-aware change sets extend drift awareness into the deployment pipeline. Before applying changes, a drift-aware change set evaluates the actual current state of a stack rather than the last known state. This is critical when someone made a manual change under operational pressure but has not yet reconciled it. A routine deployment should not silently overwrite a deliberate operational decision. 

This capability supports the position that drift should generally be reconciled deliberately rather than reverted automatically. When a drift-aware change set reveals unexpected state, the deploying team can pause, investigate, and decide whether to incorporate the drift into the template or proceed with the planned change. 

Over time, drift data provides organizational insight beyond individual resource compliance. The same resource drifting repeatedly, or the same team consistently making out-of-band changes, points to gaps in process, tooling, or team capacity. That signal is valuable only if someone is reviewing it systematically. 

The operational maturity journey 

Moving from ClickOps to fully governed CloudFormation management is not a single migration event. The progression moves through four recognizable stages: 

 

Level  Stage  What it means 
Level 1  Visibility  The team knows what exists. IaC Generator provides templates that represent the infrastructure. Necessary, but not sufficient. 
Level 2  Control  Resources are under CloudFormation management. Changes route through templates and change sets. Drift is detectable. 
Level 3  Automation  Drift detection runs on schedule. CI/CD pipelines incorporate drift awareness. Governance is a property of the deployment process. 
Level 4  Governance  Compliance policies are enforced automatically. Drift outside defined parameters triggers remediation or escalation. Infrastructure state is continuously validated against policy. 

Moving from visibility to control is primarily an organizational challenge. It requires three deliberate shifts: 

  1. Ownership

Every CloudFormation stack needs a named team responsible for its drift state. Establish this accountability through: 

  • A mandatory team-owner tag applied to every stack. 
  • Integration with AWS Service Catalog to enforce ownership metadata from provisioning onward. 
  1. Process

Changes need to be routed through CloudFormation, not around it. Any change made outside of the IaC workflow (whether through the console, CLI, or SDK) is a potential source of drift. Governance controls include: 

  • AWS CloudTrail with EventBridge rules that flag API calls made outside of CloudFormation. 
  • A defined reconciliation window (for example, 24 hours for production hotfixes) that acknowledges operational reality while maintaining accountability. 
  1. Feedback loops

Point-in-time drift snapshots are useful, but trends over time are more valuable for identifying systemic issues. Build feedback mechanisms that surface patterns: 

  • Use Amazon Athena to query historical drift data for recurring patterns. 
  • Feed drift metrics into existing operational review cadences. 

Conclusion 

IaC Generator makes the invisible visible. It turns infrastructure provisioned outside of IaC workflows into CloudFormation templates that can be versioned, reviewed, and automated. The template is not the destination; it is the starting point for building infrastructure that teams can change with confidence and govern at scale. 

The real work is organizational: assigning stack ownership, routing changes through CloudFormation, building continuous drift awareness, and treating drift data as a signal about process gaps rather than as a compliance checkbox. Organizations that approach this as a cultural shift alongside a technical migration are the ones that sustain the gains long-term. 

Getting started 

For teams ready to implement this approach, the following resources provide step-by-step guidance: 

  • Implement drift notification routing: Use AWS Chatbot with EventBridge to route alerts to team channels, or trigger ticket creation via AWS Lambda. 

Leen AlAttas is a Technical Account Manager in the AWS Enterprise Support organization based in Riyadh, Saudi Arabia, where she has spent the past year helping enterprise customers optimize their cloud operations. She specializes in security and works closely with organizations to strengthen their AWS security posture. 

John Chebib is a Senior Technical Account Manager at AWS based out of Bahrain. He works with customers providing technical assistance and architectural guidance on various AWS services. He brings several years of experience in data analytics and architectural roles for various large-scale enterprises.

AI-powered clinical trial eligibility and safety using Amazon Bedrock AgentCore

Post Syndicated from Sachin Jain original https://aws.amazon.com/blogs/architecture/ai-agents-for-clinical-trial-screening/

AI agents built on Amazon Bedrock AgentCore let clinical trial teams make fast, accurate enrollment decisions while keeping clinicians in control through human-in-the-loop oversight. According to the Tufts Center for the Study of Drug Development, 80 percent of clinical trials miss their enrollment timelines, and each day of delay costs an estimated $500,000.

Today, eligibility decisions rely on manual chart review across fragmented sources — EHR notes, lab results, imaging reports, and medication histories. Study teams spend hours reconstructing each candidate’s history and mapping it to protocol criteria. As protocols grow more complex, this doesn’t scale: screen failure rates stay high and enrollment targets slip.

We show how to architect a Clinical Trial Eligibility and Safety Agent on AWS that assembles patient evidence, evaluates it against protocol criteria, and presents screening recommendations with citations, while clinicians retain final authority and full audit trails. It combines AWS HealthLake for FHIR-native data access, Amazon Bedrock AgentCore for multi-step reasoning, and Amazon Bedrock AgentCore Evaluations for scoring each decision via LLM-as-a-judge and human-in-the-loop. This post is for solution architects, engineering teams, and technology leaders applying AI to clinical trial operations on AWS.

AI agents for clinical trial screening

AI agents with Human-in-the-Loop (HIL) are well-suited for clinical trial eligibility and safety decisions because they address information fragmentation while preserving human clinical judgment. The core problem isn’t a lack of data, but that eligibility and safety signals are scattered across EHR notes, lab portals, imaging reports, and medication histories, forcing study teams to reconstruct each participant’s clinical picture. A knowledge graph addresses this by storing clinical data as entities and the relationships between them, representing each patient, molecule, endpoint, and market as a node with relationships stored as edges. To answer an eligibility or safety question, the agent traverses these edges, going from a diagnosis to its associated labs or a medication to its known interactions, rather than re-querying and joining disconnected sources each time. This structure supports the agent’s preparatory work:

  • Organizing evidence from fragmented sources into a knowledge graph, linking patients, molecules, endpoints, and markets as interconnected nodes.
  • Mapping patient information against protocol criteria.
  • Surfacing relevant passages with citations for clinician review.
  • Highlighting uncertainties that require human judgment.

Critically, the clinician remains the decision-maker. The agent organizes the supporting information. These systems augment rather than replace clinical reasoning — proposing preliminary assessments, flagging edge cases, providing confidence scores, and learning from feedback.

As protocols grow more complex with precision oncology and biomarker-driven eligibility, agents manage multi-step logic and maintain consistency across sites, while deferring final judgment to clinical staff.

Architecture overview

This proposed architecture illustrates how core AWS services can be combined to create an end-to-end clinical trial screening pipeline. AWS HealthLake serves as the FHIR-native clinical data foundation, ingesting and normalizing patient records from disparate EHR systems, lab portals, and imaging archives into a unified, queryable data store. Amazon Bedrock AgentCore orchestrates the multi-step workflow assembling patient profiles, matching them against trial protocols, detecting safety signals, and generating evidence-backed screening recommendations. An Amazon Bedrock Knowledge Bases stores trial protocols, inclusion/exclusion criteria, and safety guidelines. The entire pipeline feeds into a clinician review dashboard where investigators examine agent reasoning, verify citations, and render final decisions. Actions are captured in an immutable audit trail for regulatory compliance.

Architecture diagram showing the clinical trial screening pipeline with AWS HealthLake, Amazon Bedrock AgentCore, and Amazon CloudWatch

Architecture workflow

The screening pipeline operates in the following steps. Each step maps to a distinct phase of the eligibility and safety assessment, from data ingestion through clinician review and continuous monitoring.

Step 1: Clinical data ingestion

AWS HealthLake ingests patient records from EHR systems, lab portals, imaging reports, and medication histories, then normalizes them into FHIR R4 resources for standardized, queryable access.

Step 2: Agent orchestration

Amazon Bedrock AgentCore orchestrates three specialized agents, each scoped to a distinct phase of the screening pipeline. They operate within the Amazon Bedrock AgentCore Runtime, which connects to tools through MCP Gateway, maintains session memory so agents reference earlier findings without re-querying, and enforces identity-based access control for least-privilege data access. A built-in code interpreter handles dynamic calculations such as eGFR or BMI derivation.

Pre-screening agent: The first gate. It resolves three threshold questions: Is the patient’s informed consent valid and current? Does their high-level profile (age, diagnosis category, geography) align with basic enrollment parameters? Have they completed any required washout period? Patients who clear all three advance. Those who don’t receive a documented rejection citing the failing criterion.

Detailed screening agent: The core clinical reasoning engine. It walks through all inclusion and exclusion criteria, retrieving the relevant FHIR resources — Observation for labs, Condition for diagnoses, MedicationStatement for medications — and evaluating each against the protocol threshold. It also reviews organ function, adverse drug reactions, and contraindicated conditions, cross-references medications against the investigational product for interactions, and assesses the overall comorbidity profile for risk combinations no single criterion would catch. The output is a structured determination (Eligible, Ineligible, or Requires Review) with a per-criterion evidence matrix, confidence scores, and a reasoning summary citing source records.

Site & enrollment agent: Once a patient clears screening, it handles operational logistics — matching the patient to the most appropriate site by proximity, capabilities, and investigator availability, then confirming open enrollment capacity. If the preferred site is full, it identifies alternatives and flags the study coordinator.

All three agents operate behind Amazon Bedrock Guardrails, which enforce:

  1. PII/PHI filtering to protect patient health information.
  2. Content safety controls to help prevent clinically inappropriate outputs.
  3. Grounding checks to keep responses anchored in retrieved evidence rather than model parametric knowledge.
  4. Denied topic boundaries to keep agents within their screening scope.

Step 3: LLM-as-judge evaluation

Amazon Bedrock AgentCore Evaluations scores every screening decision using a combination of built-in and custom evaluators across three dimensions:

  1. Clinical accuracy: Correctness of the eligibility determination against patient data, faithfulness to source evidence (not hallucinated justifications), logical coherence across reasoning steps, and context relevance confirming the right protocol and patient records were retrieved.
  2. Operational effectiveness: Response completeness and clarity for coordinators reviewing dozens of patients daily, appropriate use of FHIR queries and knowledge base tools, and end-to-end goal success (did the agent complete the full screening workflow?).
  3. Safety compliance: Custom evaluators verify that safety-critical criteria (lab thresholds, restricted medications, contraindicated conditions) were never skipped, that uncertainties are explicitly acknowledged rather than resolved with false confidence, and that all safety flags route to the appropriate review tier.

Decisions that pass evaluation with high confidence proceed to the clinician dashboard. The system flags those that fall below quality thresholds and routes them to human review with the specific evaluation concern highlighted.

Step 4: Human-in-the-loop review and enrollment

Flagged cases and agent recommendations flow into a tiered clinical review structure:

  1. PI review queue: Principal Investigators review flagged decisions from the LLM Judge, examining the agent’s reasoning chain, verifying citations against source records, and rendering a final determination.
  2. Study coordinator dashboard: Coordinators manage trial logistics, scheduling, and the day-to-day enrollment pipeline, using the agent’s structured outputs to accelerate their workflow.
  3. Patient communication: Outreach and consent updates are coordinated through the dashboard, keeping patients informed of their screening status.
  4. Escalation to medical director: Complex or high-risk cases that exceed the PI’s comfort level are escalated to the Medical Director for final adjudication.

Clinicians retain complete override capability at every stage. When a clinician overrides an agent recommendation, approving a patient the agent flagged or rejecting one it cleared, the system captures the corrected decision and the clinician’s reasoning. These corrections expand the ground truth dataset used by Amazon Bedrock AgentCore Evaluations and surface patterns that inform prompt and retrieval tuning, creating a continuous learning loop where human judgment directly improves agent performance over time.

Step 5: Observability and continuous monitoring

Amazon CloudWatch provides end-to-end observability across all agents, surfacing agent traces (step-by-step execution logs), latency metrics, error rates (failed tool calls, guardrail blocks), judge scores (pass/flag rates per agent), HITL metrics (override rates, review latency), and alarm-based escalation when safety thresholds are breached.

Although the current implementation focuses on screening and enrollment, the same agent orchestration framework, evaluation pipeline, and compliance infrastructure support future post-enrollment monitoring agents such as adverse event detection from lab results and clinical notes, protocol deviation tracking, retention risk prediction, and re-screening triggers when clinical changes affect ongoing eligibility. Each inherits the existing scoring, logging, and auditability without requiring a separate governance framework.

Evaluating agent performance in clinical trial screening with human oversight

The screening pipeline’s credibility rests on two layers: an automated evaluation layer that scores every decision, and a human-in-the-loop (HITL) layer that gives clinicians final authority. LLM-as-Judge (Step 3) decides which cases clinicians see and how they’re prioritized. The HITL workflow (Step 4) decides how clinicians act. Together they form a continuous loop where human judgment both safeguards and improves agent performance. Using Amazon Bedrock AgentCore Evaluations, you build a framework spanning three dimensions: clinical accuracy, operational effectiveness, and safety compliance with built-in and custom evaluators that run continuously.

Clinical accuracy and reasoning

Built-in evaluators check whether the agent gets the determination right and whether its reasoning holds up: Correctness (accurate against the patient’s labs, diagnoses, and medications), Faithfulness (reasoning stays grounded in patient data and protocol, not plausible-sounding invention), Coherence (no logical contradictions across steps), Context relevance (the right protocol and records were retrieved), and Goal success rate (the full workflow ran end to end). Custom LLM-as-Judge evaluators add clinical specifics: Eligibility accuracy (each inclusion/exclusion criterion evaluated correctly) and Criteria coverage (no criteria skipped, especially safety-critical lab thresholds and restricted medications).

Operational effectiveness

Accuracy alone is insufficient, output must fit workflows where coordinators review dozens of patients daily. Helpfulness, conciseness, and relevance confirm a clear, scannable, on-topic determination. Instruction following verifies the expected structured format (patient summary, criteria checklist, determination, justification, safety flags, next steps). Tool selection and parameter accuracy check the agent invoked the right tools with correct inputs.

Safety and responsible behavior

Safety carries the strictest thresholds. Harmfulness detection flags clinically dangerous content; Stereotyping detection makes sure decisions aren’t influenced by demographics beyond protocol requirements. Both trigger immediate review. Custom evaluators target the highest-risk failures: Safety flag detection confirms every significant concern surfaced (contraindicated medications, out-of-range labs, disqualifying conditions, drug interactions), with a single miss treated as critical; Uncertainty acknowledgment makes sure the agent recommends human review on missing or ambiguous data rather than making an overconfident call.

The human-in-the-loop safeguard

When a wrong eligibility call can affect patient safety, human judgment is the final safeguard. A score below threshold routes the case to the HITL workflow.

The three agents together produce an eligibility determination with a confidence score. At trial onset, the clinician sets a confidence threshold. Cases below it or flagged by evaluation reach the clinician dashboard with the specific concern highlighted. Clinicians review the full reasoning and approve, reject, or request more information from the same interface. Their corrections are stored alongside machine-approved records, feeding back into future determinations and continuously improving accuracy.

Review and approval workflow

Review is tiered by complexity: automated pre-screening filters clearly ineligible candidates. Low-complexity cases get expedited review, medium-complexity follow standard protocols, and high-complexity edge cases escalate to senior clinicians. Cases unreviewed beyond set timeframes escalate automatically. Final enrollment decisions, low-confidence cases, experimental therapies, and complex histories require human approval. Routine high-confidence checks proceed automatically.

Audit trails

The system generates immutable audit records in Amazon DynamoDB for every decision, capturing clinician ID, timestamp, patient and trial IDs, outcomes, AI recommendations, and complete workflow execution history. These records are designed to support FDA 21 CFR Part 11 requirements for electronic records and signatures, providing documentation for regulatory inspections and quality assurance. Readers should consult their compliance team and conduct their own assessment. See the AWS compliance resources for further guidance.

Security and compliance

Clinical trial data is among the most sensitive in healthcare. HIPAA, FDA 21 CFR Part 11, GxP, and GDPR require strict controls over how patient data is stored, accessed, and processed, and AI agents reasoning over that data introduce new security considerations. This solution protects data at every layer while maintaining the audit trails and privacy standards regulators require.

AWS HealthLake is HIPAA-eligible with encryption at rest and in transit, access controls, and SMART on FHIR authorization. Amazon Bedrock is HIPAA-eligible, SOC 2 attested, ISO and CSA STAR Level 2 certified, and never shares customer data with model providers. AWS PrivateLink keeps traffic off the public internet.

Amazon Bedrock AgentCore enforces agent boundaries at runtime through declarative authorization policies — readable, deterministic rules, outside application code, defining what the agent can access, invoke, and retrieve. AgentCore runs within your Amazon Virtual Private Cloud (Amazon VPC) for network isolation, and AWS CloudTrail records API calls for an immutable audit trail that can support FDA compliance requirements.

Amazon Bedrock AgentCore Evaluations scores each decision using built-in and custom evaluators with an LLM-as-a-Judge approach. Continuous sampling detects drift, and Amazon CloudWatch alerts teams when quality drops below thresholds — ongoing evidence the agent performs within validated parameters, supporting GxP with minimal manual testing.

Conclusion

In this post, we showed how combining the FHIR-native data foundation of AWS HealthLake
with the multi-step reasoning capabilities of Amazon Bedrock AgentCore turns manual,
fragmented clinical trial screening into an AI-assisted workflow that reduces patient matching
time from days to minutes. Clinical trial enrollment remains one of drug development’s most
resource-intensive bottlenecks, and delayed starts carry heavy financial consequences from lost
patent-protected sell time and operational burn. Clinicians receive organized evidence,
transparent reasoning, and actionable recommendations while retaining full decision authority
and audit traceability.

The impact extends beyond speed: more consistent criteria interpretation across sites, earlier
detection of safety contraindications, and lower screen failure rates. As oncology trial eligibility
criteria grow in complexity — with fewer than 5% of cancer patients enrolling under strict
requirements — this human-in-the-loop approach offers a scalable, compliance-aligned path to
faster, higher-quality recruitment.

Call to action

Ready to accelerate your clinical trial operations? Take the next step:

Powering agentic AI with real-time streaming data on AWS

Post Syndicated from Mazrim Mehrtens original https://aws.amazon.com/blogs/big-data/powering-agentic-ai-with-real-time-streaming-data-on-aws/

Two years ago, the conversation about streaming data and generative AI centered on a straightforward question: how do you feed real-time context into a large language model (LLM) so it can answer questions using fresh data? We explored that question in our 2024 blog post, “Exploring real-time streaming for generative AI applications,” which introduced patterns for connecting streaming pipelines to foundation models.

The landscape has shifted. Today’s generative AI systems don’t only answer questions. They observe, reason, and act. Agentic AI applications have moved from research prototype to production reality. Agentic AI-powered data pipelines now monitor streaming telemetry, detect anomalies, decide on remediation strategies, and execute actions without human intervention. They maintain memory across sessions, query live data sources on demand, and coordinate with other agents to solve complex problems.

This shift demands a fundamentally different relationship between streaming infrastructure and AI. It’s no longer enough to inject context into a prompt. You need architectures where streaming data continuously powers autonomous agent action and keeps a real-time lakehouse fresh for training and retrieval. That data also flows into multiple consumption patterns, such as generative business intelligence (BI) for humans, standardized protocols for agent queries, and proactive memory hydration for low-latency agent context.

This post introduces three architectural patterns that together form a unified streaming backbone for the agentic AI era:

  1. Streaming feature engineering → real-time inference → action: Continuous data flows build features, invoke AI models, and act in a single pipeline.
  2. Event-driven agent invocation: Streaming pipelines detect patterns across millions of events and trigger agentic workflows with full context already assembled.
  3. Real-time context synchronization: Change data capture (CDC) and streaming pipelines keep agents’ memory current, so agents can respond instantly rather than making expensive external calls.

The following sections explore each pattern in depth.

Pattern 1: Streaming feature engineering → real-time inference → action

You’re watching a live football match. As a striker receives the ball in the box, AI-generated commentary appears on screen: “This is Smith’s third touch in the penalty area in the last 3 minutes. His conversion rate from this zone is 34% this season.” That insight was computed from streaming event data, passed through a feature pipeline, and fed to a generative AI model. All of this happened within the time it takes the striker to turn and shoot.

This pattern combines two capabilities that are often treated separately: using real-time data to continuously improve AI models, and using real-time data to invoke those models for immediate action. The streaming pipeline does both: it builds the features that train the model and the features that drive inference.

Streaming events (user interactions, sensor readings, game events, and transaction records) flow into Amazon Managed Streaming for Apache Kafka (Amazon MSK) or Amazon Kinesis Data Streams. Amazon Managed Service for Apache Flink processes these events through windowed aggregations (tumbling windows, sliding windows, or session windows) to produce features: rolling averages, counts, ratios, behavioral sequences, or other derived signals relevant to your use case.

These features serve two paths simultaneously:

The inference path: At the end of each window (or on each event, depending on your latency requirements), features are passed to a generative AI or machine learning (ML) inference endpoint: Amazon Bedrock for generative output, or Amazon SageMaker for custom models. The model produces a result (commentary, a recommendation, a personalization decision, or a risk score) and the pipeline acts: posting content to a user, updating a recommendation feed, sending a notification, or writing to a downstream system.

The training path: The same streaming features are continuously written to a real-time data warehouse or lakehouse such as Apache Iceberg tables on Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), that keeps training datasets fresh. Amazon SageMaker lakehouse architecture provides unified access for training jobs and fine-tuning pipelines. As new data streams in, your models can be retrained or fine-tuned on data that’s minutes old rather than days old. This matters for domains where patterns shift quickly, such as fraud detection, personalization, and industry dynamics.

Amazon S3 Tables handles the Iceberg table management automatically, including compaction, snapshot management, and metadata optimization. Your team focuses on feature logic rather than storage operations. The AWS Glue Data Catalog makes these tables discoverable across training jobs, inference pipelines, and analytics consumers. Glue Data Catalog supports business context and semantic search. This context helps models discover and select the right data asset for any given task.

Scenarios

Real-time sports commentary: Streaming game events (passes, shots, player positions) flow through Apache Flink on Managed Service for Apache Flink, which computes rolling features (possession percentage, shot frequency by zone, player heat maps). These features feed a generative AI model through Amazon Bedrock that produces natural-language commentary and statistical insights in real time. Simultaneously, the features are written to S3 Tables to improve the model’s understanding of game patterns over time.

Streaming personalization: User clickstream data flows through Managed Service for Apache Flink, which computes behavioral features (session duration, category affinity scores, recency-weighted purchase history). These features invoke a personalization model that updates the user’s experience in real time by reranking product recommendations, adjusting content feeds, or triggering targeted offers. The same features feed the lakehouse to retrain the personalization model nightly.

Streaming data flows through Managed Service for Apache Flink, then forks into a real-time inference path and a training path

Figure 1: Streaming feature engineering feeding a real-time inference path and a continuous training path

Pattern 2: Event-driven agent invocation

At 2:47 AM, a pressure sensor on a manufacturing line begins drifting. Within seconds, a streaming pipeline detects the anomaly, assembles full context (device history, maintenance schedule, correlated sensor readings), and invokes an agent that opens a maintenance work order, adjusts the device’s sampling rate, and notifies the on-call engineer. All of this happens before a human sees an alert.

Pattern 1 invokes inference on every window or event. It runs continuously. Pattern 2 adds to this approach: the streaming pipeline continuously analyzes data and invokes an agentic workflow when specific conditions are met or a pattern is detected. The pipeline is the sensor. The agent is the responder. Dynamic rules are the bridge between them.

The key distinction is that the events and triggers are dynamic. They’re defined by rules programmed into the streaming pipeline or traditional ML models for prediction or detection. The pipeline determines when and how the agent is triggered, making the system fluid and adaptive. You can update detection logic without redeploying the agent. You can add new anomaly patterns without changing the response logic.

Streaming telemetry flows into Amazon MSK or Amazon Kinesis Data Streams. Managed Service for Apache Flink runs continuous anomaly-detection logic, such as statistical models, windowed aggregations, threshold-based rules, or ML-based scoring. Critically, when Flink detects an anomaly, it doesn’t only publish a raw alert. It assembles a context package: the anomaly details, relevant historical data, correlated signals from other streams, and metadata the agent needs to act immediately.

This context package is published to a downstream topic and consumed by an Amazon Bedrock AgentCore agent. Because the pipeline has already assembled full context, the agent doesn’t waste time gathering information. It can reason and act immediately. AgentCore Runtime hosts the agent, AgentCore Observability provides tracing and logging, and AgentCore Memory maintains state across invocations (so the agent knows, for example, that this is the third anomaly from this device this week).

The benefit of this pattern over a polling-based or scheduled approach is twofold:

  1. Latency: The agent is invoked within seconds of the anomaly, not at the next polling interval.
  2. Context richness: The pipeline has already done the work of correlating signals and assembling context. A polling-based agent would need to make multiple queries to reconstruct what the pipeline already knows.

The rules that trigger invocation are a powerful abstraction. They can be simple thresholds (“temperature exceeds 95°C”), statistical (“value deviates more than 3σ from the rolling mean”), or ML-based (“anomaly score from an embedded model exceeds 0.85”). You can update these rules dynamically by adding new detection patterns, adjusting sensitivity, or routing different anomaly types to different agents.

Managed Service for Apache Flink detects anomalies and sends a context package to an Amazon Bedrock AgentCore agent that acts on them

Figure 2: Event-driven agent invocation triggered by anomaly detection in the streaming pipeline

Pattern 3: Real-time agent context

A customer messages their bank: “Was that $847 charge at the airport legitimate?” The agent responds in under two seconds with full context (the customer’s recent travel pattern, the merchant’s fraud-risk score, and the transaction details) because all of this was already loaded into the agent’s context layer through streaming CDC. A reactive agent without this synchronization would need to make five separate API calls across three systems, taking 8–12 seconds and risking timeout failures.

This pattern addresses a fundamental question: how proactive should your agent be about gathering context?

A proactive agent has the full context, continuously synchronized with the state of the world. When a user asks a question, the agent already has the relevant knowledge from context. It responds from memory rather than making expensive external calls. A reactive agent starts cold. It knows nothing until it queries for information, making multiple calls across security boundaries, handling authentication, and stitching together data from disparate sources. For latency-sensitive use cases, where a user sends a prompt and expects a fast response, this difference is critical.

Real-time context synchronization uses CDC and streaming pipelines to keep agent memory current. The agent’s knowledge graph becomes a synchronized replica of the distributed systems it needs to reason about.

No agent is purely proactive or purely reactive. The design decision is: what data should be pre-loaded, and what should be fetched on demand? This is a spectrum, and where you land depends on three factors:

  1. Latency sensitivity: If users expect fast, contextually relevant responses, pre-load the data the agent needs most frequently.
  2. Data volume: Synchronizing everything is impractical. An efficient, fast search that still produces accurate results matters more than exhaustive pre-loading. Be selective about what you push.
  3. Data freshness requirements: Some data changes every second (stock prices, session state). Other data changes rarely (customer preferences, account configuration). Load what changes frequently and matters immediately.

Streaming pipelines (Managed Flink reading from Amazon MSK, Kinesis Data Streams, or CDC streams from operational databases) continuously process events and write aggregated results to the agent’s knowledge graph, or the context layer. These stores can take multiple forms depending on your access patterns:

  • AWS Context automatically maps relationships across your existing data into a knowledge graph and supports agentic search so AI agents can access governed data relationships, business rules, and domain knowledge at runtime. Data stewards manage the graph through an intuitive console, reviewing inferred relationships, promoting them to production, and attaching domain-specific knowledge like business definitions and usage rules.
  • Amazon Bedrock AgentCore Memory for structured agent context that persists across sessions.
  • Amazon DynamoDB for low-latency key-value lookups (customer profiles, account state).
  • Amazon OpenSearch Serverless for semantic search over unstructured context (past conversations, documents).
  • Amazon Neptune for relationship-rich data (knowledge graph).
  • Amazon S3 Tables fully managed Apache Iceberg tables in Amazon S3, for interoperability between multiple query engines.

For data that isn’t pre-loaded, the agent falls back to on-demand retrieval. This applies when the data is too large, changes too rarely to justify streaming, or is needed only in edge cases. The Model Context Protocol (MCP) provides a standardized interface for this. MCP servers expose heterogeneous data sources through a uniform protocol. The agent queries MCP when it needs context that isn’t in its synchronized memory.

This same real-time context synchronization pattern serves different consumers:

AI agents access fresh context through a real-time knowledge graph or a context layer, and MCP servers (pull tier), as in the preceding sections.

Human analysts and executives access the same context layer, which can directly query Apache Iceberg tables on S3 Tables through its direct query mode. Amazon Quick chat provides natural-language access to real-time lakehouse data. No intermediate warehouse is required. This is the generative BI expression of the same underlying pattern: streaming data keeps the lakehouse current, and Amazon Quick gives humans conversational access to it.

Training and fine-tuning pipelines access the synchronized lakehouse through Amazon SageMaker Lakehouse, keeping models fresh (as described in Pattern 1).

The underlying principle is the same across consumers: streaming pipelines synchronize distributed data into accessible stores, and each consumer accesses those stores through the interface that fits their needs.

A streaming synchronization layer feeds multiple stores that serve AI agents, human analysts, and training pipelines

Figure 3: Real-time context synchronization serving agents, analysts, and training pipelines from shared stores

Bringing it together

The three patterns in this post form a unified architecture built on a single streaming backbone:

Pattern 1 uses streaming pipelines to build features that simultaneously drive real-time inference and keep training data fresh. Your models improve continuously while serving predictions in real time.

Pattern 2 uses streaming pipelines as intelligent sensors that detect anomalies and invoke agents with full context already assembled. This separates detection logic from response logic for maximum flexibility.

Pattern 3 uses streaming pipelines to synchronize distributed system state into the agent’s context layer, making agents more proactive and serving multiple consumers (agents, humans, and training jobs) from the same pre-loaded data.

The streaming infrastructure you build (Amazon MSK, Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, and Amazon S3 Tables) serves all three patterns simultaneously. A Flink application can compute features for inference (Pattern 1), detect anomalies that trigger agents (Pattern 2), and synchronize state into agent memory (Pattern 3).

To get hands on with the patterns described in this post, refer to Agentic AI-Powered anomaly detection: Spotting anomalies in real-time.

You don’t need to implement all three patterns at once. Start with the one that addresses your most pressing need. But design your streaming infrastructure knowing it will serve multiple patterns. In the agentic AI era, every stream is a potential input to an agent, a model, and a human decision-maker.


About the authors

Mazrim Mehrtens

Mazrim Mehrtens

Mazrim is a Sr. Specialist Solutions Architect for messaging and streaming workloads. Mazrim works with customers to build and support systems that process and analyze terabytes of streaming data in real time, run enterprise Machine Learning pipelines, and create systems to share data across teams seamlessly with varying data toolsets and software stacks.

Ali Alemi

Ali Alemi

Ali is a Principal Streaming Solutions Architect at AWS. Ali advises AWS customers with architectural best practices and helps them design real-time analytics data systems which are reliable, secure, efficient, and cost-effective. Prior to joining AWS, Ali supported several public sector customers and AWS consulting partners in their application modernization journey and migration to the Cloud.