Post Syndicated from Peter Garnepudi original https://aws.amazon.com/blogs/architecture/how-to-run-a-dr-readiness-engagement/

Part 3 of 3

The framework from Part 1 (three business decisions) and Part 2 (progressive validation) works at any pace. You can apply it over months as part of normal operations, or compress it into a structured engagement that moves from undefined targets to a validated recovery commitment in weeks. This post walks through how to run that engagement yourself, with concrete steps, outputs, and a before-and-after example from a real enterprise engagement.

Step 1: Kickoff (define the scope)

Bring three groups into one room: the application owner who knows the technical architecture, the business owner who understands the consequence of downtime, and the platform team that owns the shared services your application depends on. If a dependency is owned by a team not represented in the room, that’s your first action item before proceeding.

Ask these questions in order:

• What does the user need to be able to do for this function to count as “recovered”?
• Which applications, shared services, and accounts are in the path of that user journey?
• Who owns each piece, and have they confirmed their recovery timeline?

The output artifact is a boundary map: a document listing every service and account in the recovery path, with an owner assigned to each and their stated recovery timeline recorded. This single artifact becomes the foundation for every subsequent step.

Step 2: Assessment (surface the gaps)

Run a structured review of each component in the boundary map. For each dependency, ask: does it recover in the target AWS Region, does it recover within the stated recovery time objective (RTO), and does an upstream service’s recovery target cap the RTO you can actually commit to?

• A Well-Architected Framework Review against the Reliability pillar gives you a systematic way to surface single points of failure, missing health checks, and unverified quotas.
• An Operational Readiness Review checks whether the runbook’s assumptions hold: are the resources pre-provisioned, do upstream dependencies recover within the target, and are there any manual steps that depend on a specific person being available?

The output is a prioritized list of dependency gaps with owners assigned and a target fix date for each. “Done” at this step means every gap has an owner, a planned fix, and a committed date. Gaps without owners don’t get fixed.

Step 3: Validation plan (structure the tests)

Build a test calendar that progresses through the levels from Part 2. A practical sequence for a first engagement:

• Week 2: Tabletop exercise on the highest-consequence function. Confirm ownership, escalation path, and expected duration per step.
• Week 4-6: Planned failover after gap remediation. Measure actual recovery time against the committed RTO. Verify data consistency against the recovery point objective (RPO).
• Week 8-10: Surprise drill (if the planned failover passed). Test coordination without advance notice.

Don’t schedule a game day with fault injection until you’ve passed a planned failover cleanly. Each level builds confidence for the next. Attempting a higher level before the lower one passes cleanly wastes effort and produces noisy findings that obscure the real gaps.

Step 4: Execute and learn (close the loop)

Each test produces findings. Document them in a format that maps each finding back to the decision it informs:

• Boundary findings (a shared service discovered mid-test that wasn’t in the recovery scope) → feed back into Decision 1.
• Consequence findings (recovery took longer than the business can tolerate) → feed back into Decision 2, where the business owner either accepts the longer timeline or invests in a faster architecture.
• Dependency findings (an upstream service whose recovery target capped the application’s own, a resource that wasn’t pre-provisioned) → feed back into Decision 3.

The output is an updated recovery commitment that reflects what you actually measured, not what you assumed. This document replaces the prior version and becomes the baseline for the next cycle.

Step 5: Ongoing cadence (prevent drift)

Recovery posture drifts as applications change. A new microservice adds a dependency nobody mapped. A team deploys a configuration change that shifts traffic routing. A quota that covered peak load 6 months ago no longer does. Schedule a quarterly review that checks: has the boundary changed (new shared services, new accounts)? Has the business owner confirmed they still accept the RTO/RPO targets? Have any dependencies changed their recovery behavior?

A trigger for re-engagement outside the quarterly cycle: any architecture change that adds a cross-account dependency or changes the failover path.

What this looks like in practice

In the enterprise engagement referenced throughout this series, the team applied this pattern across its portfolio of applications and accounts. The first exercise missed a contractual recovery commitment of 5 hours, taking more than twice that long. Root causes mapped to all three decisions: the boundary excluded a shared authentication platform (most downstream applications couldn’t authenticate users), the dependency chain included an upstream service whose recovery target capped what the applications calling it could achieve, disaster recovery (DR) resources weren’t pre-provisioned, and DNS failover depended on a manual step that took over 3 hours under coordination pressure.

Over 4 months, the team redefined the boundary to include shared platforms, documented the full dependency chain per application, reconciled each application’s target against its slowest dependency, pre-provisioned the recovery environment, and replaced manual DNS failover with automated routing built on Amazon Application Recovery Controller (ARC) routing controls. ARC routing controls are on/off switches on a data plane that shifts traffic without depending on the Amazon Route 53 control plane, so a cutover doesn’t rely on modifying DNS records mid-event. This follows the Well-Architected guidance to rely on the data plane, not the control plane, during recovery (REL11-BP04), the same static-stability principle behind pre-provisioning. The second exercise passed. The organization operated a live customer-facing system from the recovery Region for the first time. Senior leadership cited the structured engagement as the factor that connected the two different outcomes.

Accelerating the timeline

You can run every step in this post with your own teams. If you want to compress the timeline, AWS Enterprise Support provides specific mechanisms that map to each step: a Technical Account Manager to facilitate the kickoff and maintain engagement momentum between sessions, Well-Architected and Operational Readiness Reviews for the assessment, specialist Solutions Architects for deep dependency analysis, and AWS Countdown for coordinated operational-readiness and engineering support during critical events, including planned failovers and DR game days. These are acceleration options, not prerequisites. The steps and outputs are the same either way.

The AWS Resilience Lifecycle Framework and AWS Resilience Analysis Framework provide the detailed practices behind each decision, and AWS Resilience Hub helps you assess applications against your defined RTO and RPO targets.

---

About the authors

Post Syndicated from Peter Garnepudi original https://aws.amazon.com/blogs/architecture/validating-your-dr-commitment-from-controlled-test-to-real-world-readiness/

Part 2 of 3

In Part 1 of this series, you defined three business decisions that shape your disaster recovery (DR) commitment: the recovery boundary, the business-anchored target, and the dependency-reconciled commitment. Those decisions give you a target on paper. Testing tells you whether it holds under real conditions.

This post covers a progressive approach to DR validation. Each level of testing surfaces a different class of gap, and each gap feeds back into the three decisions from Part 1 so your commitment gets stronger with every cycle.

Why a single test isn’t enough

A scheduled DR exercise with advance notice proves that the failover mechanism works. That’s a necessary starting point. You can confirm that traffic shifts correctly, data remains consistent in the recovery Region, and the application functions as expected after the transition.

A scheduled exercise doesn’t test the assumptions that only surface under pressure. It doesn’t show whether teams who haven’t rehearsed together can coordinate, or whether everyone agrees on who declares the failover and under what conditions. It also can’t confirm that you provisioned and validated your recovery environment before you triggered the failover. These are the conditions where real recoveries stall, and they’re invisible to a test where every team knows it’s coming and watches closely.

In the same engagement from Part 1, several applications had DR instances in a stopped state. The failover automation assumed it could launch instances, create database replicas, and scale container fleets on demand during the exercise. That approach worked only if the team confirmed two things in advance: that the recovery AWS Region had capacity for the requested instance types, and that the account quotas could absorb the full production load. The team didn’t verify these assumptions in advance.

The architecture was correct. The team didn’t verify those assumptions in advance, and it didn’t pre-provision the recovery environment. To close this gap, the team needs to pre-provision the recovery environment and configure auto scaling groups, confirm that its account quotas match production capacity requirements, and remove any runbook step that assumes your team needs to create resources during the event.

Four levels of progressive validation

The AWS Resilience Lifecycle Framework treats testing as a practice you deepen over time. The four levels below are a maturity model you can apply yourself, ordered from lowest-cost awareness to highest-fidelity validation. We’ve drawn it from patterns across the DR exercises we’ve seen. Each surfaces a different class of gap.

Level 1: Tabletop exercise. A facilitated walkthrough of the recovery plan without touching production. The goal is shared awareness and ownership clarity.

Questions to ask during the tabletop:

• Who declares the failover, and under what conditions?
• For each step in the runbook, who owns it and what’s their backup?
• What’s the expected duration of each step, and what’s the total end-to-end timeline?
• Which steps depend on a specific person being available?

In the same case study from Part 1, the first tabletop surfaced three boundary gaps: the shared authentication platform wasn’t in the recovery scope, a networking account owned by a different team had no failover runbook, and the DNS update step listed a generic team alias with no named owner.

Level 2: Planned failover. A scheduled failover with advance notice. The goal is confirming the mechanism works under known conditions.

What to measure:

• Actual recovery time per application compared to the stated recovery time objective (RTO).
• Data consistency validation in the recovery Region to confirm the recovery point objective (RPO).
• Any step that requires manual intervention (potential automation candidate).
• Any dependency that recovered later than expected.

In the same case study, the planned failover revealed the DNS misconfiguration: the shared platform’s Amazon Route 53 records still resolved to the primary Region 45 minutes into the test. Most downstream applications appeared healthy but couldn’t serve users. Actual recovery took more than twice the 5-hour target.

A successful planned failover produces a measured gap (or confirmation) between the RTO target you defined and your actual recovery time. That number is the starting point for every subsequent conversation with the business owner about whether the target is realistic.

Level 3: Surprise drill. A failover triggered without advance notice to the responding team. The goal is testing execution readiness, not capability.

This level finds the gaps that preparation hides: the runbook that only its original author can interpret, the step that depends on a specific person’s tribal knowledge, and the escalation path that nobody has practiced. In the case study, a later surprise drill revealed that only one engineer understood how to validate database consistency in the recovery Region. With that person unavailable, the team lost 90 minutes locating and interpreting the procedure.

Level 4: Game day with fault injection. An exercise where you inject specific dependency failures using AWS Fault Injection Service, which ships purpose-built scenarios such as “AZ Availability: Power Interruption” and “Cross-Region: Connectivity”. The goal is testing resilience under conditions you didn’t fully script.

You discover the dependency you assumed would be healthy, the monitoring gap that delays detection, and the cascade that your tabletop didn’t model. This is the closest approximation to a real event without waiting for one to happen.

What each level reveals

The tabletop finds the step no team owns. The planned failover finds the misconfiguration. The surprise drill finds the coordination failure. The game day finds the unmodeled dependency. Each level surfaces a different class of gap.

Run only the calm, scheduled test and you validate the mechanism while leaving the conditions that cause real recovery failures untested. The point isn’t to skip levels. You build confidence at each level before increasing realism. But stopping at the planned failover, a pattern we often see once the mechanism works, leaves team, process, and dependency gaps invisible until an event exposes them.

Closing the loop

Each test produces specific findings that map directly to the three decisions from Part 1.

• A boundary gap (a shared service discovered mid-test that wasn’t in the recovery scope) feeds back into Decision 1.
• A recovery that exceeds the time the business can tolerate feeds back into Decision 2, where the business owner either accepts the longer timeline or invests in a faster architecture.
• A dependency that recovered slower than the target assumed feeds back into Decision 3 for a revised commitment that reflects what was actually measured.

In the same engagement, the first exercise surfaced all four patterns described across this series: a boundary gap (shared platform not in scope), a dependency gap (an upstream service whose recovery target capped the application’s own), a stability gap (resources not pre-provisioned), and a coordination gap (manual DNS failover). The team mapped each finding to a specific decision and a specific fix. Four months later, the second exercise passed. The organization operated a live customer-facing system from the recovery Region for the first time.

The value of progressive testing is not passing a single exercise. It’s building a practice where each cycle tightens the recovery commitment, and each finding has a clear path back to the decision it informs.

Where to start

If you haven’t run a tabletop yet, start there. Gather the owners of one critical business function and walk the recovery plan together. Ask: who declares the failover, what’s the first action, who owns each step, and what’s the expected duration of each. If the room disagrees with any answer, this indicates an opportunity to clarify and strengthen your recovery plan before touching production.

If you’ve passed a planned failover, consider whether a surprise drill would surface coordination gaps that preparation masks. In this engagement, the surprise drill cost the team 90 minutes that the planned failover never surfaced. Whatever level you’re at, document what each test reveals and feed it back into the three decisions. A posture that improves every test cycle is worth more than one that passes a single exercise and never evolves. In Part 3, we walk through how to structure a full DR readiness engagement: who to involve, what to produce at each step, and how to compress the timeline from months to weeks.

---

About the authors

Post Syndicated from Peter Garnepudi original https://aws.amazon.com/blogs/architecture/three-business-decisions-that-define-your-dr-success/

Part 1 of 3

A disaster recovery (DR) strategy is only as strong as the business decisions behind it. The architecture and failover mechanisms matter, but they execute against targets that you define. In our experience facilitating enterprise DR exercises, the difference between teams that meet their recovery targets and teams that don’t often comes down to three decisions made (or deferred) before the first architecture diagram: what you’re actually recovering, what the business needs from that recovery, and what your dependencies allow you to commit to.

This post helps you validate whether your current approach accounts for each one. It draws on a pattern from a real engagement where revisiting these decisions turned a missed recovery target into a successful test four months later.

This is Part 1 of a three-part series. It walks through those three decisions and shows how to make them before you choose an architecture pattern. Part 2 covers progressive validation testing. Part 3 covers how to run a DR readiness engagement. The approach builds on the AWS Resilience Lifecycle Framework and the AWS Resilience Analysis Framework.

Define the recovery boundary

The first decision is drawing the line around what “recovered” means. You cannot meaningfully define a recovery time objective (RTO) until you specify precisely what you need working for you to consider that objective met.

The Resilience Lifecycle Framework defines a business function as the end-to-end capability a user depends on (completing a transaction, filing a report, accessing an account). A single business function often spans multiple applications, accounts, and teams. When you set a recovery target at the application level without accounting for the shared services that the application calls, you might define the boundary too narrowly to deliver the capabilities your users require.

Here’s a pattern we observed in a DR exercise for a large enterprise customer. The customer tested dozens of applications across many AWS accounts. Each application had its own failover automation, and each one failed over correctly. But a shared authentication platform (running in a separate account, managed by a different team) used Amazon Route 53 simple routing pointed at the primary AWS Region. It had no health checks and no failover configuration. Because the team didn’t include it in the recovery scope, the platform remained in the primary Region while everything else moved. The result: Most of the downstream applications that depended on it couldn’t authenticate users. Although the team had successfully failed over the applications themselves, the business function (user login, transaction, confirmation) did not work. This single boundary gap, not the individual application architectures, was the primary driver of the missed recovery target.

The solution involved redefining the recovery scope rather than changing the architecture. The organization redefined recovery boundaries around business functions (what capabilities must you deliver to your users?) rather than individual applications (is this service running?). That single change pulled the shared authentication platform inside the scope and helped the team discover the dependency. A business impact analysis is the structured starting point for this work.

Anchor the target to business consequence

The second decision is settling what recovery means in terms the business owns. Business stakeholders strengthen RTO and recovery point objective (RPO) when they help set them. In organizations where these targets are defined by engineering alone, they can become disconnected from what the business actually tolerates. A 4-hour RTO means something different for a payments platform that clears interbank settlement than it does for an internal reporting dashboard, even when both share the same database engine and replication setup.

In the same engagement, the business carried a contractual recovery commitment of 5 hours. The business treated it as a firm commitment to their regulator. The engineering team treated it as a stretch goal based on the architecture’s theoretical capability. Neither side had validated whether 5 hours was achievable given the dependency chain. That disconnect only became visible when the first exercise took 9–11 hours. The target had never been tested against the actual recovery path, and the business stakeholder who owned the penalty had never been asked whether the engineering team’s assumptions matched theirs.

RPO is a statement about data loss. For a payments platform clearing interbank settlement, an RPO of 5 minutes might still be too wide if those minutes contain transactions that can’t be reconstructed. The question for the business owner is: Which records must survive a Regional impairment, and can you prove they will? The business owner uses that framing to decide whether asynchronous replication (with its window of in-flight data loss) is acceptable. The alternative is synchronous writes, with the latency cost that comes with them.

RTO is a statement about downtime tolerance. A 4-hour recovery time is a technical figure until someone attaches a consequence to each hour. For regulatory filings, the fourth hour might mean a missed deadline with financial penalties. For a storefront, every minute carries lost orders.

The AWS Resilience Lifecycle Framework recommends defining objectives around business outcomes directly (for example, the percentage of successful orders maintained when a component degrades). A service-level agreement (SLA) is a contract with a financial penalty. A service-level objective is a target you manage toward. Raising an SLA doesn’t make an application more resilient.

The output of this decision is a recovery requirement stated in language the business recognizes and engineering can build against: the function, the consequence of losing it, the data that must survive, and the tolerable downtime before the consequence lands.

Account for the dependency chain

The third decision is confirming that your recovery target accounts for everything in the critical path, not just the components you own. Your application recovers only as fast as its slowest dependency. If a service you call takes 4 hours to recover in the recovery Region, your achievable RTO is 4 hours regardless of your own architecture.

In the same engagement, one customer-facing application had committed to a 1-hour recovery target and had the architecture to meet it on its own. But it called a shared identity service, owned by a different team, to validate every user session. That identity service carried a 4-hour recovery target. No matter how fast the customer-facing application recovered, users couldn’t do anything until the identity service came back. The application’s real, achievable RTO was 4 hours, set entirely by a dependency its own team didn’t own and hadn’t accounted for when they committed to 1 hour.

This is why the dependency chain, not the application in isolation, defines the commitment you can make. Identify and document every service your business function calls, note the recovery target each one holds, and set your own target against the slowest link you depend on. Where another team owns a dependency, its recovery target becomes a commitment you need from them, in writing, not an assumption. The AWS Fault Isolation Boundaries whitepaper helps you reason about which dependencies hold during a Regional impairment, and which introduce risk. The distinction that matters is what your runbook asks a service to do. Reading existing data keeps working during an impairment: existing credentials still authenticate, and existing DNS records still resolve. Changing things is what carries risk, because creating or modifying AWS Identity and Access Management (IAM) roles and Route 53 records runs through control planes that live in a single Region. A recovery path that only reads existing state holds up. One that must create or change resources mid-event might not. That’s the case for settling those changes ahead of time.

Where you can, reduce the chain itself. When you pre-provision and run a dependency in the recovery Region before the event, you reduce how many services need to recover before you can restore your application. The shorter the chain of things that must recover in sequence, the closer your achievable target gets to the one you want to commit to.

Where to start

Start your validation by examining one business function. Assess whether: Is the recovery boundary drawn around the full user journey, including shared services in other accounts? Has a business stakeholder confirmed the consequence-per-hour attached to the current RTO and RPO? Does the target account for every dependency in the path, including global endpoints and any runbook step that assumes resource creation during failover?

If you can answer yes to all three with evidence, your targets are well-founded. If any answer is uncertain, you’ve identified where to focus next. In Part 2, we cover how to validate your answers through progressive testing that closes the distance between a controlled exercise and real-world readiness.

---

About the authors