All posts by Jennifer Moran

Recommended resilience testing in the next generation of AWS Resilience Hub

Post Syndicated from Jennifer Moran original https://aws.amazon.com/blogs/architecture/recommended-resilience-testing-in-the-next-generation-of-aws-resilience-hub/

Organizations that have defined resilience policies and assessed their applications face a common next step: verifying those applications behave as expected when failures occur. Designing fault injection experiments manually requires deep expertise. You need to know which faults to inject, which resources to target, and how to judge whether the outcome counts as success. Many teams do not know where to start, and experiments built by hand can drift out of date as architectures evolve.

Recommended resilience testing in the next generation of AWS Resilience Hub addresses this. It’s a new capability that helps you validate how your services respond to and recover from failure scenarios. The tests are pre-configured and powered by AWS Fault Injection Service (FIS).

Recommended resilience testing uses what AWS Resilience Hub already knows about your service, including its resources, dependencies, and resilience policy. With this information, AWS Resilience Hub automatically creates FIS experiments, injects faults, and evaluates the results against your success criteria and recovery time objective (RTO). It uses that evaluation to determine a pass or fail outcome. What used to be manual experiment design becomes an out-of-the-box test with built-in success evaluation based on your policy. Each test run auto-generates a report, accessible through the Resilience Hub console.

Customers frequently ask which failure modes to test for and what prescriptive testing guidance we can offer. AWS Resilience Hub answers this by providing four tests at launch, selected based on your service’s architecture and resilience policy.

Availability Zone: recovery – Validates your service can detect and recover from an Availability Zone impairment. The test injects the symptoms of a power interruption in a single Availability Zone, affecting compute, networking, storage, and database resources within that AZ. It confirms your service detects the impaired AZ, continues operating in the healthy AZs, and returns to a healthy state within your defined RTO. It can also help surface single-AZ dependencies you might not be aware of.

Dependency validation – Validates how your service responds when its dependencies within the AWS Region are impaired. If you have enabled dependency discovery, the discovered soft dependency with the highest query volume is pre-selected by default. You can adjust the selection to target individual dependencies or specific groups. The dependency validation test blocks dependencies within the Region to see how your service behaves when they are unavailable. It can help you validate that your service remains healthy when soft dependencies are impaired, and surface dependencies you might not have known were hard. You can also include known hard dependencies to understand their impact and confirm your service detects and responds when they are unavailable. We recommend starting with a single soft dependency and expanding from there.

Multi-Region: isolation – Validates your service can operate independently from a Region when connectivity to another Region is unavailable. The test blocks connectivity between the two Regions, including network traffic, cross-Region data replication, and selected dependencies. This test applies to both active-active and active-passive architectures, and is also a useful precursor to the Multi-Region: recovery test.

Multi-Region: recovery – Validates that your service can recover and serve clients within your defined RTO after a Region is impaired. The test introduces failures into dependencies in the selected Region. Through success alarms, it confirms that your service recovers within your defined RTO in the recovery Region and remains healthy for the duration of the test. You perform the recovery action yourself, such as triggering a failover or executing an Amazon Application Recovery Controller (ARC) Region switch plan.

Each test works at three levels. A test template is the pre-configured recommendation, specifying the fault actions, target resource types, and parameters. A test is your configured instance of a template for your service, with your success alarms, stop conditions, and settings. A test run is a single execution that scopes to your current resources and produces a pass/fail outcome with results and a report. You can run the same test quarterly, monthly, or on demand. To get to that pass/fail outcome you can use success criteria in the test.

Choose success criteria that reflect your service’s health and customer experience, such as order success rate or end-to-end availability, rather than the state of individual resources. The impaired resources are expected to be unhealthy. What you are validating is whether your service as a whole absorbed the failure. Combined with your resilience policy, these criteria determine the pass/fail outcome.

For recovery tests (Availability Zone recovery and Multi-Region recovery), the test passes when all success criteria return to a healthy state within the RTO defined in your policy and hold until the test ends. This validates not just that your service recovered, but that it recovered fast enough to meet your stated objectives.

For sustained tests (Dependency validation and Multi-Region isolation), the test passes when all success criteria hold their thresholds from fault injection through the end of the test, proving your service remained healthy throughout the impairment.

You can also select additional alarms for observability only, such as watching the impaired Region degrade. These appear in results and reports without affecting the outcome.

Deep dive: Dependency validation

Of the four recommended resilience tests, dependency validation was one of the most requested capabilities by our customers, and for good reason. Modern applications depend on dozens of services, and understanding which of those dependencies your application can function without is the key to building a resilient application. The dependency validation test determines whether your application continues to work when a given dependency becomes unavailable.

How dependency validation works

The dependency validation test blocks connectivity to your selected dependencies within an AWS Region to observe how your service behaves when they are unavailable. It validates that your service remains healthy when soft dependencies are impaired and surfaces dependencies you might not have known were hard. You can also include known hard dependencies to understand their failure impact.

This is a sustained test. Unlike recovery tests that measure whether you meet your RTO, the dependency validation test passes when all success alarms remain in OK state from the moment faults are injected through the end of the test.

Dependency discovery

You can define dependencies in two ways:

  1. Turn on dependency discovery. This uses DNS query log analysis with a 35-day look-back window and continuous polling to automatically identify the AWS services, internal endpoints, and third-party endpoints your service calls. Dependency discovery surfaces dependencies you might not be aware of, including unexpected cross-Region calls, critical third-party dependencies, and infrequently accessed services that could cause failures during incidents.
  2. Manually enter DNS endpoints directly when configuring the test.

After you define your dependencies, we recommend classifying each one as a soft or hard dependency.

Classification When to use Example What the test validates
Soft Your service degrades but continues operating without this dependency Analytics API, notification service, feature flags, recommendation engine Application handles absence gracefully. Test should pass
Hard Your service fails if this dependency is unavailable Primary database, authentication service, credential exchange Application detects failure and handles it gracefully. Test might fail (this is expected and informative)

Prerequisites

Before you begin, make sure you have at least one service created and assessed in the next generation of Resilience Hub. You would also require an AWS Identity and Access Management (IAM) service role for the next generation of Resilience Hub to assume when running a resilience test. This role can either be created in the next generation of Resilience Hub console or you can specify an existing one.

Configuring the test

From your service dashboard in the next generation of Resilience Hub, choose the Testing tab. You will see four recommended resilience tests. Select Dependency validation, and then choose Edit test configuration.

The configuration page walks you through six sections:

1. Basic settings

Test duration: The length of time fault actions run. Set any value from 1 minute to 12 hours (default: 30 minutes). After the duration, the test takes a few additional minutes to evaluate success before it ends.

Dependency Region: The AWS Region where your service is deployed that you want to test for dependency failures.

2. Dependencies to block

This is where the distinction between soft and hard dependencies directly drives your test.

Default behavior:

The discovered soft dependency with the highest query volume is pre-selected. If no dependencies have been classified as soft, none are pre-selected.

  • Discovered dependencies: A selection box pre-populated with dependencies found in your service in the selected AWS Region. You can choose up to 10 dependencies to test.
  • (Optional) Additional dependencies: Expand this section to include hard dependencies or dependencies not listed in the primary selection. If you didn’t run any dependency discovery, you can use this field to enter a DNS address that will be blocked as a dependency.

Best practice: Start with a single soft dependency and expand from there, progressively building confidence in your application’s failure handling before including hard dependencies.

3. Test actions (fault actions)

The next generation of Resilience Hub uses AWS FIS actions to block connectivity to your selected dependencies, automatically targeting resources discovered in your service. The fault actions inject 100 percent packet loss at the compute layer, dropping traffic to the selected dependencies:

Fault action What it does
aws:ssm:send-command Drops traffic from Amazon Elastic Compute Cloud (Amazon EC2) instances to the selected dependencies
aws:ecs:task-network-packet-loss Drops traffic from Amazon Elastic Container Service (Amazon ECS) tasks (EC2 and AWS Fargate) to the selected dependencies
aws:eks:pod-network-packet-loss Drops traffic from Amazon Elastic Kubernetes Service (Amazon EKS) pods to the selected dependencies

When the test starts, it matches actions to resources discovered in your service. If no resources match an action’s target type, that action is skipped. This means the test automatically adapts to your compute environment, whether you are running on Amazon EC2, Amazon ECS, Amazon EKS, or a combination.

Additional setup required: The setup for these actions depends on your compute type. You need an AWS Systems Manager Agent container in your Amazon ECS task definition, a Kubernetes service account for Amazon EKS pods, or AWS Systems Manager Agent installed on your Amazon EC2 instances.

4. Test alarms

Alarms determine whether the test passes or fails. Select alarms discovered from your service that reflect service performance and customer experience. For example, order success rate for an ecommerce service or transactions processed for a payments system.

  • Success alarms: The next generation of Resilience Hub uses these to validate whether your service passed or failed the test. Choose alarms that measure your service’s health across the AWS Region.
  • (Optional) Additional alarms: Tracked during the test run and shown in the report, but don’t affect the test outcome. Use additional alarms for observability to monitor how these alarms behave during the dependency impairment.

How to pass: The test passes if all success alarms remain in OK state until the test actions end.

Success criteria and additional alarms share a limit of 5 per configuration update.

5. Stop conditions, reports, and logs

Stop conditions: Amazon CloudWatch alarms that act as guardrails. If breached, they stop the test automatically to protect your workloads. Choose up to five alarms. They must be in the account where the test runs and the AWS Region where the faults are injected.

(Optional) Reports: If you configure an Amazon Simple Storage Service (Amazon S3) bucket as a report destination on your service, the next generation of Resilience Hub generates a report after each test run that includes the test configuration, alarm status, timeline, and results.

Logs: Optionally send test logs to an Amazon S3 bucket or Amazon CloudWatch Logs for deeper analysis. The next generation of Resilience Hub doesn’t charge for sending logs, though ingestion and storage charges apply.

6. Test permissions

The test requires an IAM role to grant access to AWS services and resources. For single account setups, you need one execution role. For multi-account setups, you need an orchestrator role in the account running the test and a target role in each account containing targeted resources. You can create the role directly from AWS Management Console with required permissions pre-configured or use an existing role. Once saved, the role is used by default for future runs.

Note: Updating test settings might change required IAM permissions. Required permissions vary based on the trust policy, fault actions, and test options selected. Always review the IAM role instructions to confirm correct permissions.

Running the test: What success and failure look like

The value of the dependency validation test becomes clear when you compare results across dependency classifications. We walk through two test runs on the same service with the same success alarms: one blocking soft dependencies, one blocking hard dependencies.

Test run: Soft dependencies blocked — PASSED ✅

Dependencies blocked:

  • Payment Gateway.
  • Notification Service.

Success alarms:

  • canary-heartbeat-canary-failure-us-east-2: Monitors service health check availability.
  • canary-order-flow-canary-failure-us-east-2: Monitors end-to-end order processing.
Soft dependency test results: both success alarms in OK state, solid green timeline throughout test duration

Figure 1: Soft dependency test run with both success alarms holding in OK state for the full test duration

What the results show: Both alarms remained in OK state (solid green) from the moment test actions started through the end of the test. The application continued functioning normally without Payment Gateway and Notification Service. The order flow kept processing. The heartbeat kept responding. The service handled the absence of its soft dependencies gracefully, exactly as designed.

What this proves: These dependencies are correctly classified as soft. Your application can tolerate the loss of these dependencies, suppressing notifications without impacting the customer-facing order flow.

Test run: Hard dependencies blocked — FAILED ❌

Dependencies blocked:

  • Amazon Cognito.
  • AWS Security Token Service (AWS STS).

Success alarms:

  • canary-heartbeat-canary-failure-us-east-2: Monitors service health check availability.
  • canary-order-flow-canary-failure-us-east-2: Monitors end-to-end order processing.
Hard dependency test results: order-flow alarm transitions to red immediately after test actions start, heartbeat remains green

Figure 2: Hard dependency test run with the order-flow alarm entering alarm state while the heartbeat alarm stays healthy

What the results show: The heartbeat canary stayed green throughout. The application did not become unavailable. But the order-flow canary transitioned to alarm state (red) immediately after faults were injected and remained there until the test ended. Without Amazon Cognito (authentication) and AWS STS (credential exchange), the application couldn’t complete the business-critical order flow.

What this proves: Amazon Cognito and AWS STS are correctly classified as hard dependencies. The test confirms that their absence impacts critical functionality, but importantly, it also shows the failure was contained. The application did not cascade into a complete failure. It continued responding (heartbeat passed) while the specific business flow that required authentication failed gracefully.

Interpreting results: A decision framework

Test results aren’t simply pass or fail. A passing test can still surface a misclassified dependency, and a failing test can confirm that a dependency is genuinely critical. What matters is reading the outcome against how you classified the dependency, so you know whether your architecture behaved as intended and what to do about it.

The following decision framework helps you interpret the results of each test, understand what they mean, and decide what to do next.

Scenario Expected Outcome What it means What to do next
Soft dependency blocked → test passes Yes Graceful degradation is working. Classification is accurate. No action required. Your architecture handles this failure correctly.
Soft dependency blocked → test fails No The dependency might be misclassified as soft when it’s actually hard. Or your degradation logic has a gap. Review application behavior. Reclassify as hard, or fix the degradation path.
Hard dependency blocked → test fails Yes Confirms the classification. Critical functionality is impacted without this dependency. Verify the failure is graceful (no cascading failures, no data corruption, no hung connections). Consider whether you can add retry/fallback to reclassify as soft in the future.
Hard dependency blocked → test passes No The dependency might be misclassified. It could actually be soft. This could also indicate the dependency isn’t being called as intended. Investigate whether your application is more resilient than expected, or whether the dependency is not being exercised. Reclassify if appropriate or fix the application logic.

Summary

Recommended resilience testing in the next generation of AWS Resilience Hub removes the barrier between knowing your architecture and proving it holds under failure. It provides four AWS recommended tests built on AWS FIS, covering Availability Zone recovery, dependency validation, Multi-Region isolation, and Multi-Region recovery. You can move from policy definition and failure mode assessment to evidence-backed validation without building experiments from scratch.

You can start with the dependency validation test to understand how your application handles the loss of its dependencies. Expand to AZ recovery and multi-Region tests as your program matures. Each test run captures a point-in-time snapshot of your current resources, produces a pass/fail outcome evaluated against your resilience policy, and feeds results into your compliance and reporting workflows over time.

Recommended resilience testing is available today in all AWS commercial Regions where the next generation of AWS Resilience Hub is supported. To learn more and get started, visit the AWS Resilience Hub User Guide and the AWS Resilience Hub product page.

 


About the authors

Jennifer Moran

Jennifer Moran

Jennifer is a Principal Resilience Specialist Solutions Architect at AWS, where she helps customers architect resilient cloud solutions that improve their resilience posture and operational excellence. She is the co-author of Engineering Resilient Systems on AWS (O’Reilly Media). Her interests include fault-tolerant design, scalable architectures, and high-availability systems.

Imran Sadiq

Imran Sadiq

Imran is a Senior Solutions Architect at Amazon Web Services, working with financial services customers across New Zealand. As a resilience specialist, he is passionate about helping organizations prepare for, respond to, and recover from the unexpected. With deep expertise in resilience testing, disaster recovery, and cloud-native resilience patterns, Imran partners with customers to build resilient systems.

Streamlining access to powerful disaster recovery capabilities of AWS

Post Syndicated from Jennifer Moran original https://aws.amazon.com/blogs/architecture/streamlining-access-to-powerful-disaster-recovery-capabilities-of-aws/

Learn how you can use AWS services like AWS Backup and AWS Elastic Disaster Recovery (AWS DRS), along with AWS Resilience Competency Partner solutions like Arpio to implement powerful and comprehensive Disaster Recovery solutions.

Resilience is the ability of your application to keep running even when “bad stuff” happens. A critical part of your resilience strategy is Disaster Recovery (DR). DR is what protects you against less frequent, but bigger faults like natural disasters, technical faults, and bad actors. To maintain critical business continuity, disaster recovery requires recovering your workload to a new site, such as a different AWS Region or AWS account.

AWS provides powerful tools for all aspects of resilience. However, achieving a comprehensive Disaster Recovery solution for your cloud workloads using native AWS services requires planning and engineering effort. This is because, as the Shared Responsibility Model for Resiliency states, resilience is a shared responsibility between AWS and the customer. This blog post will help you understand your responsibilities and show you how to reduce the work required to access the powerful DR capabilities of AWS.

In this blog post, we take a building blocks approach. Starting with the tools like AWS Backup to protect your data, we then add protection for Amazon Elastic Compute Cloud (Amazon EC2) compute using AWS Elastic Disaster Recovery (AWS DRS). Finally, we show how to use the full capabilities of AWS to restore your entire workload—data, infrastructure, networking, and configuration, using Arpio disaster recovery automation.

Your recovery site

For DR, your recovery site is usually going to be a different AWS Region (cross-Region) or a different AWS account (cross-account) than where your workload runs.

Cross-Region backup and recovery are essential for disaster recovery. This helps to keep your workloads protected if an event causes your source Region to be unable to run your workload. AWS Regions are strong fault isolation boundaries, so the event in your source is highly unlikely to affect your recovery Region.

Cross-account backup is a critical security measure to enable recovery from malware and ransomware. By storing copies of your data in a separate clean room recovery account with distinct credentials, you create an isolated environment that can’t be accessed, even if the source account is compromised.

Protecting your data

We start with your data—your data is the foundation of your workload.

Each AWS data storage resource offers the ability to back up or replicate your data. For example, Amazon Elastic Block Store (Amazon EBS) snapshots, Amazon Relational Database Service (Amazon RDS) for Db2) Backups, and Amazon Simple Storage Service (Amazon S3) replication, offer data protection for Amazon EBS volumes, Amazon RDS Databases instances, and Amazon S3 Buckets respectively. Figure 1, for example, illustrates the several methods and destinations of backup and replication for Amazon RDS.

AWS Backup and replication for Amazon RDS

Figure 1. AWS Backup and replication for Amazon RDS

AWS Backup takes this further, tying together many of these disparate backup technologies, giving a single plane of glass to configure data backup plans across resources. AWS Backup also added backup capabilities for AWS resources that previously didn’t have them such as Amazon Elastic File System (Amazon EFS) and Amazon FSx. It also provides the ability to back up your data to a different AWS Region or AWS account. It even enabled cross-Region backup for services like Amazon DynamoDB, which previously didn’t have that capability.

AWS Backup is a powerful tool for protecting your data. With your data protected, you will then need additional automation to get to a fully recovered workload. If you want to build this yourself, AWS offers the tools to do this. In this prior blog post on Backup and Restore, we go more into detail about adding automation using Amazon EventBridge and AWS Lambda functions for automated recovery. For more information, see figures 6 and 7.

With its ability to create vaults for secure storage, define policies for governance, and set schedules for automation, AWS Backup centralizes and streamlines the backup process. Instead of managing backups service by service, you can enforce consistent protection across resources, reduce manual effort, and streamline recovery when it matters most.

Protecting your Amazon EC2 compute

As important as data is, only restoring your data isn’t a complete solution to recovering from disaster. You must also restore your compute resources.

For static Amazon EC2 instances, you can create snapshots of your instances as Amazon Machine Images (AMIs), or use AWS Backup to manage this for you. By static instances, we mean those you create directly and maintain, as opposed to those created by Amazon EC2 Auto Scaling. Such a strategy can deliver a Recovery Points Objective (RPO) and Recovery Time Objective (RTO) of minutes to hours. The size (and growth in size) of your EC2 instances determines the time to back them up. Their size and launch time determines the time to restore them.

If you need real-time RPO (near-zero data loss) and RTO (recovery) in minutes or less, then AWS DRS is the solution here. AWS DRS provides a nearly continuous block-level replication, recovery orchestration, and automated server conversion capabilities. With these, you to achieve a crash-consistent recovery point objective of seconds, and a recovery time objective typically ranging between 5–20 minutes. You can also use AWS DRS to configure your recovery Amazon Virtual Private Cloud (Amazon VPC). So, with the right settings, you can get your EC2 networking to look like your primary environment.

Protecting everything in your entire workload

Restoring data and static EC2 instances is only part of the disaster recovery solution that you need. Modern workloads often rely on a broader range of compute services, including EC2 Auto Scaling, AWS Lambda, Amazon ECS, and Amazon Elastic Kubernetes Service (Amazon EKS). For ECS and EKS, you can run on EC2 instances or go serverless with AWS Fargate. You will need a solution that can restore either of these.

The challenge with these services is making sure that they are recreated with the right configuration and metadata. For example, EC2 instance types and volume sizes, EC2 user data, or AWS Lambda function code, and not everything here is stateless. Both ECS and EKS can rely on persistent Amazon EBS volumes or Amazon Elastic File System (Amazon EFS). In those cases, recovery requires restoring the data and reattaching volumes restored from backup to the correct ECS tasks or EKS pods.

You can build automation to do all of this, or you can rely on an AWS Resilience Software Competency Partner solution to take care of this for you. Arpio is a software as a service (SaaS) product focused on discovering and backing up everything it takes to run your workload on AWS, and recovering it cross-Region and cross-account as a fully functional workload.

Figure 2 illustrates how AWS tools (left) establish a powerful foundation for robust workload recovery. Beyond these foundational building blocks, full recovery requires additional resources (right), including AWS compute options as discussed. Furthermore, complex networking (potentially spanning VPCs and accounts), infrastructure, and IAM principals are critical. Arpio uses and extends AWS Backup, AWS DRS, and other AWS service capabilities to back up and restore a functional AWS workload, including all its necessary components. This unburdens you from the undifferentiated heavy lifting of building your own automation. You still have responsibilities in the shared responsibility model, but Arpio takes on most of the work of getting you backed up and recovered.

AWS tools on the left provide a powerful foundation for full recovery on the right

Figure 2. AWS tools on the left provide a powerful foundation for full recovery on the right

Even with data, compute, networking, infrastructure, and IAM principals restored, there is another requirement to achieve full recovery: translation of your configuration. For example, an application that accesses the Amazon RDS database requires configuration information about the DB endpoint and credentials. When restoring your RDS instance into your recovery environment, it will have a new endpoint. Arpio addresses this using a two-fold strategy. First Arpio will find all references to the early database endpoint name and translate them to the new database endpoint name. Next, Arpio will also create an Amazon Route 53 private hosted zone in the recovered VPC, mapping the early endpoint to the new one using a CNAME record. This way, applications still using the early name still connect to the newly recovered database. Arpio also securely backs up the credentials in your recovery account, for every database backup taken, ready to be recovered for the point in time that you recover your database from. Figure 3 shows how your recovered application can seamlessly access your restored database.

Arpio automation ensures your applications can access your restored database in the recovery environment

Figure 3. Arpio automation ensures your applications can access your restored database in the recovery environment

Figure 4 shows a sample AWS workload protected by Arpio. In the standby state, you can see how Arpio is coordinating multiple AWS services. When a disaster or ransomware event occurs, you can launch a recovery. This will create a fully recovered workload as seen in the recovery stage on the right.

Sample AWS workload protected by Arpio

Figure 4. Sample AWS workload protected by Arpio

Arpio does all of this in your accounts, on your behalf. To enable this, Arpio applies AWS Well-Architected Tool (AWS WA Tool) best practices for security, using only IAM roles with least-privilege permissions. For example, the IAM role used to access your source AWS account is incapable of changing or mutating your source workload and is explicitly denied from reading or exfiltrating any data.

With its ability to back up over 140 AWS resources and restore them as fully functioning AWS workloads in a cross-Region cross-account recovery environment, Arpio builds on top of the powerful AWS tooling to streamline your complete workload recovery.

Conclusion

Disaster recovery (DR) is essential for a robust resilience strategy. By using the powerful tools offered by AWS and complementing them with AWS Resilience Competency Partner solutions like Arpio, organizations can significantly streamline access to comprehensive and powerful disaster recovery capabilities for their AWS workloads.


About the authors