Tag Archives: AWS Resilience Hub (ARH)

Recommended resilience testing in the next generation of AWS Resilience Hub

Post Syndicated from Jennifer Moran original https://aws.amazon.com/blogs/architecture/recommended-resilience-testing-in-the-next-generation-of-aws-resilience-hub/

Organizations that have defined resilience policies and assessed their applications face a common next step: verifying those applications behave as expected when failures occur. Designing fault injection experiments manually requires deep expertise. You need to know which faults to inject, which resources to target, and how to judge whether the outcome counts as success. Many teams do not know where to start, and experiments built by hand can drift out of date as architectures evolve.

Recommended resilience testing in the next generation of AWS Resilience Hub addresses this. It’s a new capability that helps you validate how your services respond to and recover from failure scenarios. The tests are pre-configured and powered by AWS Fault Injection Service (FIS).

Recommended resilience testing uses what AWS Resilience Hub already knows about your service, including its resources, dependencies, and resilience policy. With this information, AWS Resilience Hub automatically creates FIS experiments, injects faults, and evaluates the results against your success criteria and recovery time objective (RTO). It uses that evaluation to determine a pass or fail outcome. What used to be manual experiment design becomes an out-of-the-box test with built-in success evaluation based on your policy. Each test run auto-generates a report, accessible through the Resilience Hub console.

Customers frequently ask which failure modes to test for and what prescriptive testing guidance we can offer. AWS Resilience Hub answers this by providing four tests at launch, selected based on your service’s architecture and resilience policy.

Availability Zone: recovery – Validates your service can detect and recover from an Availability Zone impairment. The test injects the symptoms of a power interruption in a single Availability Zone, affecting compute, networking, storage, and database resources within that AZ. It confirms your service detects the impaired AZ, continues operating in the healthy AZs, and returns to a healthy state within your defined RTO. It can also help surface single-AZ dependencies you might not be aware of.

Dependency validation – Validates how your service responds when its dependencies within the AWS Region are impaired. If you have enabled dependency discovery, the discovered soft dependency with the highest query volume is pre-selected by default. You can adjust the selection to target individual dependencies or specific groups. The dependency validation test blocks dependencies within the Region to see how your service behaves when they are unavailable. It can help you validate that your service remains healthy when soft dependencies are impaired, and surface dependencies you might not have known were hard. You can also include known hard dependencies to understand their impact and confirm your service detects and responds when they are unavailable. We recommend starting with a single soft dependency and expanding from there.

Multi-Region: isolation – Validates your service can operate independently from a Region when connectivity to another Region is unavailable. The test blocks connectivity between the two Regions, including network traffic, cross-Region data replication, and selected dependencies. This test applies to both active-active and active-passive architectures, and is also a useful precursor to the Multi-Region: recovery test.

Multi-Region: recovery – Validates that your service can recover and serve clients within your defined RTO after a Region is impaired. The test introduces failures into dependencies in the selected Region. Through success alarms, it confirms that your service recovers within your defined RTO in the recovery Region and remains healthy for the duration of the test. You perform the recovery action yourself, such as triggering a failover or executing an Amazon Application Recovery Controller (ARC) Region switch plan.

Each test works at three levels. A test template is the pre-configured recommendation, specifying the fault actions, target resource types, and parameters. A test is your configured instance of a template for your service, with your success alarms, stop conditions, and settings. A test run is a single execution that scopes to your current resources and produces a pass/fail outcome with results and a report. You can run the same test quarterly, monthly, or on demand. To get to that pass/fail outcome you can use success criteria in the test.

Choose success criteria that reflect your service’s health and customer experience, such as order success rate or end-to-end availability, rather than the state of individual resources. The impaired resources are expected to be unhealthy. What you are validating is whether your service as a whole absorbed the failure. Combined with your resilience policy, these criteria determine the pass/fail outcome.

For recovery tests (Availability Zone recovery and Multi-Region recovery), the test passes when all success criteria return to a healthy state within the RTO defined in your policy and hold until the test ends. This validates not just that your service recovered, but that it recovered fast enough to meet your stated objectives.

For sustained tests (Dependency validation and Multi-Region isolation), the test passes when all success criteria hold their thresholds from fault injection through the end of the test, proving your service remained healthy throughout the impairment.

You can also select additional alarms for observability only, such as watching the impaired Region degrade. These appear in results and reports without affecting the outcome.

Deep dive: Dependency validation

Of the four recommended resilience tests, dependency validation was one of the most requested capabilities by our customers, and for good reason. Modern applications depend on dozens of services, and understanding which of those dependencies your application can function without is the key to building a resilient application. The dependency validation test determines whether your application continues to work when a given dependency becomes unavailable.

How dependency validation works

The dependency validation test blocks connectivity to your selected dependencies within an AWS Region to observe how your service behaves when they are unavailable. It validates that your service remains healthy when soft dependencies are impaired and surfaces dependencies you might not have known were hard. You can also include known hard dependencies to understand their failure impact.

This is a sustained test. Unlike recovery tests that measure whether you meet your RTO, the dependency validation test passes when all success alarms remain in OK state from the moment faults are injected through the end of the test.

Dependency discovery

You can define dependencies in two ways:

  1. Turn on dependency discovery. This uses DNS query log analysis with a 35-day look-back window and continuous polling to automatically identify the AWS services, internal endpoints, and third-party endpoints your service calls. Dependency discovery surfaces dependencies you might not be aware of, including unexpected cross-Region calls, critical third-party dependencies, and infrequently accessed services that could cause failures during incidents.
  2. Manually enter DNS endpoints directly when configuring the test.

After you define your dependencies, we recommend classifying each one as a soft or hard dependency.

Classification When to use Example What the test validates
Soft Your service degrades but continues operating without this dependency Analytics API, notification service, feature flags, recommendation engine Application handles absence gracefully. Test should pass
Hard Your service fails if this dependency is unavailable Primary database, authentication service, credential exchange Application detects failure and handles it gracefully. Test might fail (this is expected and informative)

Prerequisites

Before you begin, make sure you have at least one service created and assessed in the next generation of Resilience Hub. You would also require an AWS Identity and Access Management (IAM) service role for the next generation of Resilience Hub to assume when running a resilience test. This role can either be created in the next generation of Resilience Hub console or you can specify an existing one.

Configuring the test

From your service dashboard in the next generation of Resilience Hub, choose the Testing tab. You will see four recommended resilience tests. Select Dependency validation, and then choose Edit test configuration.

The configuration page walks you through six sections:

1. Basic settings

Test duration: The length of time fault actions run. Set any value from 1 minute to 12 hours (default: 30 minutes). After the duration, the test takes a few additional minutes to evaluate success before it ends.

Dependency Region: The AWS Region where your service is deployed that you want to test for dependency failures.

2. Dependencies to block

This is where the distinction between soft and hard dependencies directly drives your test.

Default behavior:

The discovered soft dependency with the highest query volume is pre-selected. If no dependencies have been classified as soft, none are pre-selected.

  • Discovered dependencies: A selection box pre-populated with dependencies found in your service in the selected AWS Region. You can choose up to 10 dependencies to test.
  • (Optional) Additional dependencies: Expand this section to include hard dependencies or dependencies not listed in the primary selection. If you didn’t run any dependency discovery, you can use this field to enter a DNS address that will be blocked as a dependency.

Best practice: Start with a single soft dependency and expand from there, progressively building confidence in your application’s failure handling before including hard dependencies.

3. Test actions (fault actions)

The next generation of Resilience Hub uses AWS FIS actions to block connectivity to your selected dependencies, automatically targeting resources discovered in your service. The fault actions inject 100 percent packet loss at the compute layer, dropping traffic to the selected dependencies:

Fault action What it does
aws:ssm:send-command Drops traffic from Amazon Elastic Compute Cloud (Amazon EC2) instances to the selected dependencies
aws:ecs:task-network-packet-loss Drops traffic from Amazon Elastic Container Service (Amazon ECS) tasks (EC2 and AWS Fargate) to the selected dependencies
aws:eks:pod-network-packet-loss Drops traffic from Amazon Elastic Kubernetes Service (Amazon EKS) pods to the selected dependencies

When the test starts, it matches actions to resources discovered in your service. If no resources match an action’s target type, that action is skipped. This means the test automatically adapts to your compute environment, whether you are running on Amazon EC2, Amazon ECS, Amazon EKS, or a combination.

Additional setup required: The setup for these actions depends on your compute type. You need an AWS Systems Manager Agent container in your Amazon ECS task definition, a Kubernetes service account for Amazon EKS pods, or AWS Systems Manager Agent installed on your Amazon EC2 instances.

4. Test alarms

Alarms determine whether the test passes or fails. Select alarms discovered from your service that reflect service performance and customer experience. For example, order success rate for an ecommerce service or transactions processed for a payments system.

  • Success alarms: The next generation of Resilience Hub uses these to validate whether your service passed or failed the test. Choose alarms that measure your service’s health across the AWS Region.
  • (Optional) Additional alarms: Tracked during the test run and shown in the report, but don’t affect the test outcome. Use additional alarms for observability to monitor how these alarms behave during the dependency impairment.

How to pass: The test passes if all success alarms remain in OK state until the test actions end.

Success criteria and additional alarms share a limit of 5 per configuration update.

5. Stop conditions, reports, and logs

Stop conditions: Amazon CloudWatch alarms that act as guardrails. If breached, they stop the test automatically to protect your workloads. Choose up to five alarms. They must be in the account where the test runs and the AWS Region where the faults are injected.

(Optional) Reports: If you configure an Amazon Simple Storage Service (Amazon S3) bucket as a report destination on your service, the next generation of Resilience Hub generates a report after each test run that includes the test configuration, alarm status, timeline, and results.

Logs: Optionally send test logs to an Amazon S3 bucket or Amazon CloudWatch Logs for deeper analysis. The next generation of Resilience Hub doesn’t charge for sending logs, though ingestion and storage charges apply.

6. Test permissions

The test requires an IAM role to grant access to AWS services and resources. For single account setups, you need one execution role. For multi-account setups, you need an orchestrator role in the account running the test and a target role in each account containing targeted resources. You can create the role directly from AWS Management Console with required permissions pre-configured or use an existing role. Once saved, the role is used by default for future runs.

Note: Updating test settings might change required IAM permissions. Required permissions vary based on the trust policy, fault actions, and test options selected. Always review the IAM role instructions to confirm correct permissions.

Running the test: What success and failure look like

The value of the dependency validation test becomes clear when you compare results across dependency classifications. We walk through two test runs on the same service with the same success alarms: one blocking soft dependencies, one blocking hard dependencies.

Test run: Soft dependencies blocked — PASSED ✅

Dependencies blocked:

  • Payment Gateway.
  • Notification Service.

Success alarms:

  • canary-heartbeat-canary-failure-us-east-2: Monitors service health check availability.
  • canary-order-flow-canary-failure-us-east-2: Monitors end-to-end order processing.
Soft dependency test results: both success alarms in OK state, solid green timeline throughout test duration

Figure 1: Soft dependency test run with both success alarms holding in OK state for the full test duration

What the results show: Both alarms remained in OK state (solid green) from the moment test actions started through the end of the test. The application continued functioning normally without Payment Gateway and Notification Service. The order flow kept processing. The heartbeat kept responding. The service handled the absence of its soft dependencies gracefully, exactly as designed.

What this proves: These dependencies are correctly classified as soft. Your application can tolerate the loss of these dependencies, suppressing notifications without impacting the customer-facing order flow.

Test run: Hard dependencies blocked — FAILED ❌

Dependencies blocked:

  • Amazon Cognito.
  • AWS Security Token Service (AWS STS).

Success alarms:

  • canary-heartbeat-canary-failure-us-east-2: Monitors service health check availability.
  • canary-order-flow-canary-failure-us-east-2: Monitors end-to-end order processing.
Hard dependency test results: order-flow alarm transitions to red immediately after test actions start, heartbeat remains green

Figure 2: Hard dependency test run with the order-flow alarm entering alarm state while the heartbeat alarm stays healthy

What the results show: The heartbeat canary stayed green throughout. The application did not become unavailable. But the order-flow canary transitioned to alarm state (red) immediately after faults were injected and remained there until the test ended. Without Amazon Cognito (authentication) and AWS STS (credential exchange), the application couldn’t complete the business-critical order flow.

What this proves: Amazon Cognito and AWS STS are correctly classified as hard dependencies. The test confirms that their absence impacts critical functionality, but importantly, it also shows the failure was contained. The application did not cascade into a complete failure. It continued responding (heartbeat passed) while the specific business flow that required authentication failed gracefully.

Interpreting results: A decision framework

Test results aren’t simply pass or fail. A passing test can still surface a misclassified dependency, and a failing test can confirm that a dependency is genuinely critical. What matters is reading the outcome against how you classified the dependency, so you know whether your architecture behaved as intended and what to do about it.

The following decision framework helps you interpret the results of each test, understand what they mean, and decide what to do next.

Scenario Expected Outcome What it means What to do next
Soft dependency blocked → test passes Yes Graceful degradation is working. Classification is accurate. No action required. Your architecture handles this failure correctly.
Soft dependency blocked → test fails No The dependency might be misclassified as soft when it’s actually hard. Or your degradation logic has a gap. Review application behavior. Reclassify as hard, or fix the degradation path.
Hard dependency blocked → test fails Yes Confirms the classification. Critical functionality is impacted without this dependency. Verify the failure is graceful (no cascading failures, no data corruption, no hung connections). Consider whether you can add retry/fallback to reclassify as soft in the future.
Hard dependency blocked → test passes No The dependency might be misclassified. It could actually be soft. This could also indicate the dependency isn’t being called as intended. Investigate whether your application is more resilient than expected, or whether the dependency is not being exercised. Reclassify if appropriate or fix the application logic.

Summary

Recommended resilience testing in the next generation of AWS Resilience Hub removes the barrier between knowing your architecture and proving it holds under failure. It provides four AWS recommended tests built on AWS FIS, covering Availability Zone recovery, dependency validation, Multi-Region isolation, and Multi-Region recovery. You can move from policy definition and failure mode assessment to evidence-backed validation without building experiments from scratch.

You can start with the dependency validation test to understand how your application handles the loss of its dependencies. Expand to AZ recovery and multi-Region tests as your program matures. Each test run captures a point-in-time snapshot of your current resources, produces a pass/fail outcome evaluated against your resilience policy, and feeds results into your compliance and reporting workflows over time.

Recommended resilience testing is available today in all AWS commercial Regions where the next generation of AWS Resilience Hub is supported. To learn more and get started, visit the AWS Resilience Hub User Guide and the AWS Resilience Hub product page.

 


About the authors

Jennifer Moran

Jennifer Moran

Jennifer is a Principal Resilience Specialist Solutions Architect at AWS, where she helps customers architect resilient cloud solutions that improve their resilience posture and operational excellence. She is the co-author of Engineering Resilient Systems on AWS (O’Reilly Media). Her interests include fault-tolerant design, scalable architectures, and high-availability systems.

Imran Sadiq

Imran Sadiq

Imran is a Senior Solutions Architect at Amazon Web Services, working with financial services customers across New Zealand. As a resilience specialist, he is passionate about helping organizations prepare for, respond to, and recover from the unexpected. With deep expertise in resilience testing, disaster recovery, and cloud-native resilience patterns, Imran partners with customers to build resilient systems.

Scope workloads and share resilience policies with AWS Resilience Hub

Post Syndicated from Sinisa Mikasinovic original https://aws.amazon.com/blogs/architecture/scope-workloads-and-share-resilience-policies-with-aws-resilience-hub/

A resilience plan is useful only while it reflects the application you run. After a migration or deployment, a service can acquire a new dependency while its documented recovery objectives stay the same. In a shared Kubernetes cluster, even deciding which resources belong to that service can take investigation.

On September 18, 2026, the Next generation of AWS Resilience Hub introduced three capabilities to help you keep workload scope and resilience requirements aligned with your running environment:

  • Amazon Elastic Kubernetes Service (Amazon EKS) label support for service input sources selects Kubernetes resources using the labels that already describe your workloads.
  • Generative AI-powered Dependency Insights analyzes a service’s discovered dependencies and highlights the patterns worth reviewing.
  • Resilience policy sharing through AWS Organizations lets a central team define resilience requirements once and make them available to member accounts.

In this post, you follow a team reviewing the resilience of a sample workload named checkout on Amazon EKS, representing an online store’s checkout service. The team starts by defining which Kubernetes resources belong to checkout, using labels to distinguish it from inventory in the same namespace. This establishes the resource boundary for the resilience review.

With that boundary defined, the team examines discovered dependencies and uses Dependency Insights to prioritize findings for investigation. A cross-Region dependency becomes a concrete question about whether checkout’s recovery design accounts for the services it requires to operate. The service owner then works with a central resilience team to validate the recovery requirements and apply a shared policy. Maintaining those requirements centrally gives teams a common definition to adopt across accounts, while policy events help the central team track adoption.

You perform these steps using the console or the AWS Command Line Interface (AWS CLI). The workload generates synthetic traffic to demonstrate dependency discovery.

About the application model: In the Next generation of Resilience Hub, the primary unit of assessment is a service. Services can be grouped into systems, and user journeys describe paths through them. See the key changes and the overview of the new hierarchy in the Next generation of Resilience Hub.

Prerequisites

To follow the examples, you need:

  • Access to the Next generation of AWS Resilience Hub in a supported AWS Region.
  • An Amazon EKS cluster with sample, labeled workloads and permission to configure access to the cluster.
  • If you follow the CLI examples, install or update the AWS CLI using the installation and update instructions. Your installation must support the resiliencehubv2 commands. These examples use Bash.
  • An AWS Identity and Access Management (IAM) role that Resilience Hub can assume, with the AWSResilienceHubV2AssessmentExecutionPolicy managed policy attached. The console can create this role when you create a service.
  • Kubernetes RBAC permissions and an EKS access entry that grant this role access to Kubernetes resources. The permissions guide provides the manifest and commands.
  • For policy sharing, an organization with all features enabled and access to its management account.

The examples use two accounts: a workload account containing the EKS cluster and Resilience Hub service, and a management account that owns the shared policy.

For the CLI examples, configure profiles named workload and management for the respective accounts. Replace the example cluster ARN and other placeholders with your own values, and set the Region:

export AWS_REGION=us-east-1

Each command uses --profile to select the account. Follow either the console steps or their CLI equivalents for each operation.

Amazon EKS label support for service input sources

A service input source tells Resilience Hub where to discover a service’s resources. Input sources can be AWS CloudFormation stacks, Terraform state files, resource tags, or Amazon EKS clusters. For an EKS input source, you select the namespaces to include. With this launch, you can also use Kubernetes label selectors to filter resources within those namespaces.

A namespace can contain workloads owned by different teams. With labels, you can select checkout’s resources without including inventory or reorganizing the namespace for an assessment. You use the same labels your teams already maintain.

How it works

Specify the cluster and one or more namespaces, then add label selectors to filter resources within each namespace. A selector contains a key, an operator, and values. Selectors within a namespace use AND logic: an object must match all of them. Without a selector, Resilience Hub discovers every supported object in the selected namespaces.

Checkout and inventory share the production namespace. Their resources carry app=checkout and app=inventory, respectively. Selecting production with app In checkout includes checkout’s Deployment and ReplicaSet and excludes inventory.

A label selector chooses checkout resources from a namespace shared with inventory

Figure 1: Selecting checkout resources with a Kubernetes label

The resource list also includes supporting AWS infrastructure, such as the cluster, node group, virtual private cloud (VPC), and subnets. Labels select Kubernetes objects. They don’t make that shared infrastructure exclusive to checkout.

How to use it

In the Resilience Hub console:

  1. Choose Create service and enter a name. Under Permission, choose Create default role to create an IAM role with the required AWS permissions, or choose Use another role to select an existing role.
  2. Under Service resource discovery, choose Amazon EKS cluster and enter the cluster ARN.
  3. Choose Configure namespaces, enter production, and add a label selector with key app, operator In, and value checkout.
  4. Choose Save, select Enable dependency discovery for this service, then choose Create service. Resource discovery starts automatically.
Configure namespaces dialog selecting production with app In checkout

Figure 2: Scoping the checkout service to one namespace and one label

If you prefer to use the AWS CLI instead of the console, run the following commands to create the service and add its EKS input source.

Skip this block if you already created the service using the preceding console steps.

The console displays selector operators as In, NotIn, Exists, and NotExists. In the API, use IN, NOT_IN, EXISTS, and DOES_NOT_EXIST, respectively.

In the following command, replace ResilienceHubRole with your service’s IAM role name.

# Create the checkout service with dependency discovery enabled and save its ARN
SERVICE_ARN=$(aws resiliencehubv2 create-service \
  --profile workload \
  --name checkout \
  --regions "$AWS_REGION" \
  --permission-model '{"invokerRoleName": "ResilienceHubRole"}' \
  --dependency-discovery ENABLED \
  --query 'service.serviceArn' --output text)

# Add the EKS input source, selecting resources labeled app=checkout in production
aws resiliencehubv2 create-input-source \
  --profile workload \
  --service-arn "$SERVICE_ARN" \
  --resource-configuration '{
    "eks": {
      "clusterArn": "arn:aws:eks:us-east-1:111122223333:cluster/resilience-demo",
      "namespaces": ["production"],
      "labelSelector": {
        "matchExpressions": [
          {"key": "app", "operator": "IN", "values": ["checkout"]}
        ]
      }
    }
  }'

After resource discovery completes, you can use the following command to list the discovered resource types. If you created the service on the console, first set SERVICE_ARN to that service’s ARN.

# List discovered resource types after resource discovery completes
aws resiliencehubv2 list-resources \
  --profile workload \
  --service-arn "$SERVICE_ARN" \
  --query 'serviceResources[].resource.resourceType'

Generative AI-powered Dependency Insights

Dependency Insights uses generative AI to summarize discovered dependencies and highlight patterns that warrant resilience review. It provides an overview and categorized findings.

The Resources list describes the service’s components. The Dependencies list shows endpoints discovered through DNS activity, including AWS service endpoints in other Regions and external APIs. When that list grows to hundreds of entries, Dependency Insights helps you prioritize what to investigate.

How it works

Dependency Insights builds on dependency discovery, an optional, separately priced feature. Dependency discovery analyzes DNS activity through Amazon Route 53 Resolver and does not require installing an agent in your workload. It examines up to 35 days of available history and continues monitoring for changes. The dependency list shows each endpoint’s name, location, and when it was first and last seen.

Dependency Insights analyzes this list, filters out common service activity that typically does not indicate resilience risk, and highlights patterns such as these:

Pattern What to investigate
Cross-Region dependencies Whether a call to another Region is intentional, and how it affects recovery
New dependencies Whether an endpoint first observed within the last seven days changes the documented architecture
Third-party dependencies How the service behaves when an external provider is unavailable
Uneven usage Whether activity reflects a scheduled job or a dependency used only at certain times
Discovered dependency data feeds a generated overview and findings that a service owner reviews

Figure 3: Turning dependency data into a focused review

Insights prioritize review. They don’t prescribe fixes or list every dependency. Use the full dependency list when you need detail. A new workload’s insights reflect only the history it has generated.

How to use it

In the Resilience Hub console, wait for initial dependency discovery to complete before generating insights:

  1. Open the service. On the Assessment tab, scroll to the Dependencies section to review discovered dependencies.
  2. Choose Generate insights.
  3. Read the overview and compare each finding with the intended architecture.

You can regenerate insights once every 24 hours. This limit applies whether you generate them through the console or the AWS CLI.

Generated Insights overview and categorized findings for the checkout service

Figure 4: Dependency Insights for the checkout service

If you prefer the AWS CLI, use the following commands instead of the console steps. Set SERVICE_ARN to your service’s ARN.

Enable dependency discovery only if it is not already enabled. After initial discovery completes, review the dependency list and generate insights. If you already generated insights on the console, skip start-dependency-insights and use get-dependency-insights to retrieve the existing result.

Generation can complete asynchronously. Run get-dependency-insights again if the returned status is IN_PROGRESS.

# Enable dependency discovery if it is not already enabled
aws resiliencehubv2 update-service \
  --profile workload \
  --service-arn "$SERVICE_ARN" \
  --dependency-discovery ENABLED

# Review the dependency list
aws resiliencehubv2 list-dependencies \
  --profile workload \
  --service-arn "$SERVICE_ARN" \
  --query 'dependencySummaries[].[dependencyName, location, dependencyId]' \
  --output table

# Start generating dependency insights
aws resiliencehubv2 start-dependency-insights \
  --profile workload \
  --service-arn "$SERVICE_ARN"

# Retrieve insights; if status is IN_PROGRESS, run this command again later
aws resiliencehubv2 get-dependency-insights \
  --profile workload \
  --service-arn "$SERVICE_ARN"

The checkout service produced five insights across 29 dependencies. The following response shows the generated overview and all five findings:

{
    "overview": "5 insights generated for service 'checkout' across 29 dependencies, surfacing cross-region, third-party, and uneven usage patterns.",
    "insights": [
        {
            "category": "CROSS_REGION",
            "description": "DynamoDB in us-west-2 is reached cross-region with hard criticality, coupling this service's availability to a remote region. This dependency carries medium DNS query volume with consistent usage."
        },
        {
            "category": "CROSS_REGION",
            "description": "PagerDuty (api.pagerduty.com) is reached cross-region to us-west-2, creating an availability dependency on that remote region."
        },
        {
            "category": "THIRD_PARTY",
            "description": "Three third-party dependencies — example.com, example.net, and example.org — are present at medium DNS query volume with consistent usage patterns."
        },
        {
            "category": "THIRD_PARTY",
            "description": "GitHub is a third-party dependency with low DNS query volume."
        },
        {
            "category": "UNEVEN_USAGE",
            "description": "Two in-region dependencies show highly uneven daily usage: an S3 bucket (amzn-s3-demo-bucket) and an ECR registry (602401143452.dkr.ecr.us-east-1.amazonaws.com)."
        }
    ],
    "status": "COMPLETED",
    "createdAt": "2026-10-08T14:24:27.372000-05:00"
}

The first finding identifies a cross-Region Amazon DynamoDB dependency. If checkout requires that dependency to accept orders, a disruption in the remote Region could prevent it from operating even when its Amazon EKS resources remain healthy. The service owner should verify which operations require the table and review how those operations behave when it is unavailable. That review determines whether the dependency fits the recovery design or whether the architecture needs to change. DNS activity alone cannot answer that question.

The uneven-usage finding points to container image infrastructure. Image downloads during pod startup can produce bursts of DNS activity. This is a pattern to investigate, rather than evidence that your application has an outage.

Classify a dependency as hard if the service fails without it, or soft if the service degrades but keeps running. In the sample environment, we had classified the DynamoDB dependency as hard using the following command. The comment describes an illustrative failure assumption, which requires validation through testing:

# Classify the selected dependency as hard and record the reason
aws resiliencehubv2 update-dependency \
  --profile workload \
  --service-arn "$SERVICE_ARN" \
  --dependency-id "<dependency-id>" \
  --criticality HARD \
  --comment "Checkout writes orders to this table and cannot accept orders without it."

Resilience policy sharing through AWS Organizations

After reviewing checkout’s dependencies, the service owner can validate its recovery requirements with the business and the central site reliability engineering (SRE) team. Dependency findings inform that discussion. They don’t automatically determine the policy targets.

The central SRE team can codify the agreed requirements into a resilience policy, defining availability and recovery targets for critical services. The management account or a delegated administrator can share that policy across the organization so service owners can apply the centrally maintained requirements without creating separate copies.

The policy owner can review attachment and detachment events to track adoption. If a service stops using the policy, the SRE team can engage its owner to understand why and determine whether different requirements are appropriate. Application owners remain responsible for meeting the targets.

How it works

A policy can combine four target types: availability, recovery from an Availability Zone disruption, recovery from a Regional disruption, and data recovery. The recovery time objective (RTO) specifies the targeted recovery time. The recovery point objective (RPO) specifies the tolerable period of data loss.

When sharing is on, the policy appears in every member account alongside the account’s own policies, and its ARN shows which account owns it. Each service owner chooses whether to apply it. If a requirement must apply regardless of that choice, associate the policy with a system containing the relevant services instead, as described in Applying resilience policies across accounts.

The owning account can review a timeline of the services that start or stop using the policy. Turning sharing off stops evaluating member services against the policy and removes it from the services that used it, so coordinate changes with the teams involved.

A centrally owned resilience policy is shared with service owners in two member accounts

Figure 5: One policy definition, shared with member accounts

How to use it

The organization integration is a one-time setup in the management account:

  1. In the Resilience Hub console, open AWS Organizations settings.
  2. Select Enable trusted access and create a service-linked role, then choose Create integration.

To confirm that the service-linked role exists in a member account:

# Confirm the service-linked role exists in the workload account
aws iam get-role \
  --profile workload \
  --role-name AWSServiceRoleForResilienceHub \
  --query 'Role.Arn'

Next, create the policy in the management account with sharing turned on. Policy names can contain letters, numbers, hyphens, and underscores, but no spaces. The following targets are examples. Agree on real values with the business and service owners. This example uses an availability target of 99.9 percent.

# Create an organization-shared resilience policy and save its ARN
POLICY_ARN=$(aws resiliencehubv2 create-policy \
  --profile management \
  --name tier-1-critical \
  --availability-slo '{"target": 99.9}' \
  --multi-az '{"rtoInMinutes": 5, "rpoInMinutes": 1, "disasterRecoveryApproach": "ACTIVE_ACTIVE"}' \
  --multi-region '{"rtoInMinutes": 15, "rpoInMinutes": 5, "disasterRecoveryApproach": "WARM_STANDBY"}' \
  --data-recovery '{"timeBetweenBackupsInMinutes": 60}' \
  --sharing-enabled \
  --query 'policy.policyArn' --output text)

In the workload account, find the shared policy and apply it to the service. On the console, choose Change policy on the service and select the shared policy. With the AWS CLI:

# List policies available in the workload account, including shared policies
aws resiliencehubv2 list-policies \
  --profile workload \
  --query 'policySummaries[].[name, policyArn, sharingEnabled]' \
  --output table

# Apply the shared resilience policy to the checkout service
aws resiliencehubv2 update-service \
  --profile workload \
  --service-arn "$SERVICE_ARN" \
  --policy-arn "<shared-policy-arn>"
Lifecycle panel showing Policy associated and four targets pending assessment

Figure 6: Policy targets are defined. Assessment of those targets is still pending

Back in the management account, the policy owner can confirm adoption:

# Review service attachment events for the shared policy
aws resiliencehubv2 list-policy-events \
  --profile management \
  --policy-arn "$POLICY_ARN" \
  --event-types POLICY_ATTACHED_TO_SERVICE

An attachment event confirms adoption, not compliance. After we applied the shared policy, the console changed each target from Not defined in policy to Pending failure mode assessment. A failure mode assessment and appropriate resilience testing are needed to determine whether the service meets those targets.

Things to know

  • Shared clusters. Labels decide which Kubernetes objects belong to a service. On a cluster that several workloads share, review the discovered dependencies to confirm they belong to the service, as described in the guidance for shared EKS clusters.
  • What DNS-based discovery can’t see. Connections made directly by IP address aren’t discovered, AWS Lambda functions need VPC connectivity to be covered, and very infrequent calls might not appear in initial results. See Coverage and known limitations.
  • Timing. Results reflect the history available to the service. Give a new workload time to build history before drawing conclusions from its insights. You can regenerate insights once every 24 hours.

Cleanup

If you created resources to try these capabilities, remove them when you finish. Turning sharing off also removes the policy from every member service that uses it, so only do this for a demonstration policy:

# Stop sharing the demo policy; this also removes it from member services
aws resiliencehubv2 update-policy --profile management --policy-arn "$POLICY_ARN" --no-sharing-enabled
# Delete the demo policy
aws resiliencehubv2 delete-policy --profile management --policy-arn "$POLICY_ARN"
# Delete the demo Resilience Hub service
aws resiliencehubv2 delete-service --profile workload --service-arn "$SERVICE_ARN"

Delete the EKS cluster as well if you created it only for this walkthrough.

Conclusion

Start with a service whose architecture and ownership you know. Use EKS labels to check its Kubernetes resource boundary, then review Dependency Insights against its intended behavior. If several teams maintain the same requirements separately, consider a shared policy.

You can use the capabilities together or independently. Label support applies to EKS workloads. Dependency Insights and policy sharing also support services beyond EKS. Insights focus the investigation. Your architecture review determines what to change.

Further reading


About the author

Architecting AI-powered resilience framework on AWS

Post Syndicated from Medha Shree original https://aws.amazon.com/blogs/architecture/architecting-ai-powered-resilience-framework-on-aws/

When your production system goes down, you often discover the hard way that your resilience testing missed critical dependencies. Building an AI-powered resilience framework on AWS helps you find those weaknesses before your customers do.

Your systems don’t fail because your infrastructure isn’t resilient. They fail because resilience is assumed, not proven. Every deployment introduces new dependencies, every configuration change creates untested paths, and every gap between design intent and runtime behavior is a risk waiting to surface. In a world where customers expect always-on availability, the cost of discovering these weaknesses in production isn’t just technical. It’s measured in revenue lost, trust eroded, and events that were entirely preventable.

In this post, you’ll learn how to architect and implement a five-layer AI-powered resilience framework that automatically discovers dependencies, generates targeted experiments, and integrates with your existing Continuous Integration/Continuous Deployment (CI/CD) pipelines. First, we’ll explore the key challenges in resilience testing. Then, we’ll walk through the five-layer architecture that solves these challenges. Finally, we’ll show you how to implement this, with phased rollout guidance for pilot, expansion, and organization-wide deployment.

Traditional resilience testing can take weeks because the information needed is naturally spread across architecture diagrams, runbooks, code repositories, and team knowledge that evolves with every deployment. Designing meaningful chaos experiments on top of that requires specialized expertise that most teams don’t have on hand. Here’s how this AI-powered framework discovers your infrastructure dependencies in hours and generates targeted experiments without requiring specialized knowledge.

Combining AWS Resilience Hub, AWS Fault Injection Service, Amazon Bedrock AgentCore, and AWS Systems Manager work together to discover infrastructure dependencies in hours, tailor experiments to your specific architecture, and identify weaknesses before they affect customers.

With the next generation of AWS Resilience Hub now providing native dependency discovery and generative AI-powered failure mode analysis, this framework extends those capabilities by automating experiment generation, embedding resilience testing into your CI/CD pipelines, and creating a continuous validation loop through custom AI agents hosted on Amazon Bedrock AgentCore.

Key concepts

Before diving into the architecture, here are the key terms used throughout this post:

  • Chaos engineering — The discipline of experimenting on a system to build confidence in its ability to withstand turbulent conditions in production.
  • Mean time to resolution (MTTR) — The average time to restore service after a failure.
  • Recovery Time Objective (RTO) — The maximum acceptable downtime for your application.
  • Recovery Point Objective (RPO) — The maximum acceptable data loss, measured in time.
  • Shift-left — Testing earlier in the development cycle to catch issues before they reach production.
  • Circuit breaker — An automated mechanism that detects service failures and helps prevent cascading outages by temporarily blocking requests to failing services.
  • Canary deployment — A technique where changes roll out to a small percentage of users first to validate functionality before full deployment.

Who this is for

This post targets cloud architects, DevOps engineers, and Site Reliability Engineering (SRE) teams responsible for system reliability. You should understand AWS services, distributed systems architecture, and CI/CD pipelines. While chaos engineering experience helps, it’s not required. This framework reduces the expertise barrier that traditionally prevented adoption.

Resilience testing challenges

When infrastructure changes happen quickly, documentation tends to lag behind. A routing change to the payment service might not get reflected in the architecture diagrams, leaving single points of failure (like a single-Availability Zone authentication dependency) undocumented and untested. The payment service calls a legacy authentication API that only runs in one Availability Zone, a critical single point of failure. But documentation doesn’t reflect this because someone made a “quick fix” three weeks ago and forgot to update the diagrams.

Distributed systems contain hundreds of interconnected components. Tracking every dependency manually becomes impractical when you deploy changes continuously. Documentation created last month already misses dozens of new dependencies.

Resilience testing can be challenging without dedicated specialists to design meaningful experiments. Resilience testing is most effective when it’s tailored to your actual architecture and runs continuously. Generic fault injection and one-off test runs can leave gaps, especially as your system changes over time.

The expertise barrier stops many organizations entirely. Effective chaos engineering demands understanding distributed systems architecture, failure mode analysis, experiment scope management, and safe experiment design. Without this specialized knowledge, you either avoid resilience testing or run superficial tests that miss critical vulnerabilities.

According to the 2024 IBM Security Services Benchmark Report, organizations with mature response capabilities reduce their MTTR by approximately 50% and achieve cost savings of up to 58% per event when compared to organizations with less mature capabilities. Yet Gartner research shows that 68% of organizations cite increasing system complexity as their reason for adopting chaos engineering, while 50% admit they weren’t prepared when failures occurred.

This framework automates discovery. Automated discovery reduces infrastructure mapping from weeks to hours, typically completing initial assessment in 2–4 hours for single-account environments with thousands of resources. Subsequent runs process only the changes tracked by AWS Config, so your architecture map stays current without manual effort. Agents hosted on AgentCore Runtime analyze your AWS CloudFormation templates, code repositories, and runtime behavior to identify every connection, including hidden dependencies that manual audits miss. Experiment templates analyze your specific architecture and produce targeted tests that validate your actual failure modes, removing the need for specialized chaos engineering expertise. Continuous integration into your CI/CD pipelines catches regressions before they reach production, shifting resilience from a one-time project into an ongoing practice embedded in your development workflow.

Solution overview

The framework addresses these gaps by automatically discovering infrastructure dependencies and continuously validating resilience as the system changes. AI agents, hosted on AgentCore Runtime for secure, scalable execution, discover system dependencies automatically, analyze architectural patterns, and create targeted experiments based on your actual risk profiles. Testing scales across your application portfolio while lowering the expertise barrier.

Five-layer AI-powered resilience architecture with AWS Resilience Hub as the central orchestration hub

Figure 1. Five-layer AI-powered resilience architecture

Five-layer architecture diagram with AWS Resilience Hub — labeled “Next-Gen” with native dependency discovery and generative AI failure mode analysis — as the central orchestration hub. Layers 1–4 (Discovery, Test Generation, Experimentation, Gap Analysis) sit across the top, each connecting down to the hub. Layer 5 (Continuous Validation) sits below with CI/CD, drift detection, and dashboards. Three dashed feedback loops overlay the diagram: Gap Analysis feeds “Architecture updates” back to Discovery, Continuous Validation sends “Experiment learnings” up to Test Generation, and SSM Docs feeds “Validated recovery procedures” into FIS Templates within the Test Generation layer.

Each layer builds on the previous one, creating a comprehensive validation strategy. This architecture aligns with the AWS Well-Architected Reliability Pillar, specifically the Test reliability best practice area. Discovery maps your infrastructure. Test generation creates relevant experiments from that map. Experimentation executes those tests safely. Gap analysis identifies what needs fixing. Continuous validation helps verify that improvements persist as your systems evolve.

Now that you understand the overall strategy, let’s examine each layer in detail, starting with how the discovery layer automatically maps your infrastructure.

Discovery layer

The discovery layer forms the foundation by automatically identifying infrastructure components and their dependencies. The next generation of AWS Resilience Hub provides native dependency discovery that identifies AWS services, internal endpoints, and third-party endpoints your applications rely on. A custom agent deployed on Amazon Bedrock AgentCore (with read permissions to AWS APIs) extends this native discovery with code-level analysis, scanning your code repositories for hard-coded dependencies, connection strings, timeout configurations, and retry logic that infrastructure-level discovery alone cannot detect. A custom agent deployed on Amazon Bedrock AgentCore (with read permissions to AWS APIs) handles infrastructure discovery. Amazon Bedrock AgentCore Runtime provides dedicated MicroVM session isolation, supports long-running discovery sessions up to eight hours, and handles scaling and security without requiring you to manage infrastructure. The runtime’s built-in observability (traces, logs, and metrics) integrates natively with your existing Amazon CloudWatch dashboards without additional instrumentation.

The AgentCore-hosted agent queries services including Amazon Elastic Compute Cloud (Amazon EC2), Amazon Relational Database Service (Amazon RDS), AWS Lambda, Amazon DynamoDB, and Amazon Simple Storage Service (Amazon S3) to build comprehensive inventory. It analyzes AWS CloudFormation templates and Terraform configurations to understand your intended architecture, and accesses code repositories to identify hard-coded dependencies, connection strings, and timeout configurations in your applications. AWS Config provides configuration data and tracks changes over time. This discovery completes in 2–4 hours for environments with thousands of resources.

AI-powered discovery workflow showing infrastructure-level and code-level discovery feeding into a combined dependency map

Figure 2. AI-powered discovery workflow

Workflow diagram organized in two swim lanes. Lane 1 (blue) shows AWS Resilience Hub native infrastructure-level discovery querying AWS service APIs (Amazon EC2, Amazon RDS, AWS Lambda, Amazon S3, Elastic Load Balancing, Amazon DynamoDB) for service topology, endpoints, and Multi-AZ configuration. Lane 2 (yellow) shows an agent on Amazon Bedrock AgentCore performing code-level discovery — scanning repositories (CodeCommit, GitHub, GitLab via IAM permissions) and analyzing CloudFormation/Terraform templates for connection strings, timeouts, and circuit breakers. Both lanes feed into a combined dependency map, which flows into the AWS Resilience Hub assessment engine. Outputs include single points of failure, dependency maps (infrastructure + code), baseline resilience scores, configuration drift, and resilience gaps. AWS Config monitors configurations in parallel. A dashed feedback arrow loops from the assessment engine back to the AgentCore agent labeled “Architecture updates & learnings.” Initial mapping completes in 2–4 hours for typical single-account environments; subsequent runs process only changes tracked by AWS Config.

Test generation layer

While the next generation of Resilience Hub includes a generative AI-powered failure mode assessment that identifies potential weaknesses through static analysis, this test generation layer converts those recommendations into executable AWS Fault Injection Service experiment templates, complete with safety guardrails, progressive scope expansion, and business impact scoring tailored to your specific architecture.

Building on the discovered infrastructure, the test generation layer creates targeted chaos experiments for your specific architecture. The agent hosted on AgentCore Runtime uses Amazon Bedrock foundation models to analyze your infrastructure context, combined with the RTO, RPO, and availability targets you define in AWS Resilience Hub, to identify single points of failure and produce hypothesis-driven test scenarios aligned with your business requirements. Each hypothesis is scored by potential business impact, prioritizing experiments for customer-facing systems and components where architectural patterns indicate high-availability intent. Each experiment includes business impact scoring based on your application tier definitions in AWS Resilience Hub, architectural patterns (such as internet-facing load balancers and Amazon API Gateway endpoints), dependency analysis, and AWS resource tags. This makes sure experiments prioritize your customer-facing systems and highest-impact components. Using the code repository analysis from the discovery layer, the system detects when your applications use Amazon RDS Multi-AZ but lack proper connection retry handling, and designs database failover tests that validate your actual recovery mechanisms rather than generic network disruption tests.

For your production environments, implement a manual approval workflow where your infrastructure teams review experiment templates before execution. AWS Step Functions orchestrates approval gates. Step Functions is a workflow management service that coordinates multiple AWS services into serverless workflows.

Experimentation layer

After creating targeted experiments, the experimentation layer runs chaos tests with multi-layered safety guardrails on your infrastructure. AWS Fault Injection Service executes chaos tests with built-in safety mechanisms. Experiments start with minimal scope (affecting only 1% of your resources) and expand progressively based on your risk tolerance and validation results (for example, 1% → 5% → 10% → 25%). This follows progressive deployment strategies recommended in the AWS Well-Architected Reliability Pillar, similar to canary deployments where changes roll out incrementally to limit blast radius. Amazon CloudWatch alarms serve as stop conditions that halt experiments before they violate your Service Level Agreements (SLAs), which are contracts defining expected uptime and performance. Set alarm thresholds well below your SLA limits. If your SLA allows 1% error rate, configure stop conditions to trigger at 0.1%.

Gap analysis layer

After your experiments complete, the gap analysis layer processes results to identify weaknesses and prioritize remediation. AWS Resilience Hub correlates experiment outcomes with your resilience policies, categorizing gaps across architectural, operational, data protection, and testing dimensions. Each gap receives a priority score based on severity (how badly this violates your resilience policy), likelihood (how often this failure mode occurs), and business impact (the cost if this failure occurs in your environment).

Continuous validation layer

The continuous validation layer integrates resilience testing into your development workflow. The right approach depends on your deployment velocity and testing goals.

For most teams, a lightweight policy-as-code check (using tools like Open Policy Agent to validate Infrastructure as Code and Dockerfiles) runs in seconds and fits naturally in your CI/CD pipeline for every commit. This catches basic configuration issues, like missing health checks or single-AZ deployments, before code reaches staging.

Full resilience assessments are better suited as a pre-production gate, triggered on significant architectural changes rather than every commit. For routine deployments, lightweight resilience regression tests (validating a focused set of critical failure scenarios like database failover, Availability Zone loss, and circuit breaker activation) run automatically to catch unintended resilience degradation from code or configuration changes. This two-tiered approach gives you comprehensive safety validation for major changes and continuous regression coverage for everyday deployments, without slowing your pipeline. Your new code and infrastructure changes trigger automated resilience assessments that identify potential weaknesses during development rather than after deployment, embedding this shift-left strategy directly into your CI/CD workflow.

The policy-as-code check adds seconds to each pipeline run. Full resilience assessments add approximately 2–3 minutes per experiment.

AWS Config drift detection identifies manual changes that bypass your deployment pipelines, helping keep your architecture aligned with tested configurations.

CI/CD pipeline with two-tiered resilience testing gates for routine deployments and architectural changes

Figure 3. Resilience testing in the CI/CD pipeline

Flowchart showing a CI/CD pipeline with two-tiered resilience testing. The top row shows the standard pipeline flow: Developer Commits Code → Build & Unit Tests → Deploy to Test Environment → Resilience Regression Tests (green hexagon gate, runs 3–5 critical scenarios in ~2–3 minutes on every deployment) → Integration Tests → Deploy to Staging. From staging, if an architectural change is detected, the flow drops to a second row: Full Resilience Assessment (orange hexagon gate, runs 15–20 experiments over ~15–45 minutes using AWS Resilience Hub, FIS, and Bedrock) → Manual Approval Gate → Deploy to Production → Continuous Monitoring. If no architectural change occurred, staging skips directly to the approval gate. Both resilience gates have failure paths (red) that block deployment and loop back to the developer. A legend and a side panel summarize the two tiers.

Continuous improvement through feedback

Experiment results feed back into the discovery and test generation layers, creating a continuous improvement cycle. When experiments reveal undocumented dependencies, the discovery layer updates your architecture map. When remediation actions successfully resolve failure patterns, Systems Manager automation documents capture these procedures for future use. The Bedrock agent analyzes experiment outcomes to refine hypothesis generation, deprioritizing consistently passing scenarios and focusing on emerging risk areas as your architecture evolves.

Key benefits

Now that you’ve seen how each layer works together, let’s examine the concrete benefits this architecture brings to your organization.

Faster infrastructure discovery: Manual infrastructure discovery requires significant effort across distributed teams, including cataloging resources, tracing dependencies, and validating configurations. Automated discovery reduces this from weeks to hours by programmatically querying cloud service APIs, analyzing infrastructure-as-code templates, and mapping dependencies. After implementation, the framework scales across your application portfolio without proportionally increasing staffing.

Removes expertise barrier: Without chaos engineering specialists, you can implement resilience testing using automated scenarios. Start with the AWS Fault Injection Service Scenarios Library for common failure patterns, then expand with scenarios specific to your architecture and customize based on your application’s specific failure modes.

Proactive risk identification: Automated discovery reveals critical single points of failure that manual audits consistently miss, including hard-coded endpoints, missing circuit breakers, and absent health checks. The system identifies vulnerabilities across your infrastructure and prioritizes them by business impact, so your teams can focus remediation on the highest-risk items first.

Faster recovery through automated remediation: Automated remediation reduces your mean time to resolution by removing manual intervention for common failure patterns in your environments. AWS Systems Manager automation documents codify recovery procedures discovered during chaos experiments. When Amazon CloudWatch alarms detect failure patterns in your systems, AWS Systems Manager automatically executes remediation actions, handling issues faster than manual response.

Continuous resilience validation: Integrating resilience assessments into your CI/CD pipelines catches regressions before production deployment, maintaining resilience as your system evolves rather than treating it as a one-time validation.

Framework components

AWS Resilience Hub serves as the central orchestration layer, defining your resilience policies, running assessments, and tracking improvements. Define RTO and RPO targets for each application tier based on your business impact analysis.

Amazon Bedrock delivers the AI capabilities that power discovery and test creation. AgentCore Runtime provides the managed hosting layer, handling session isolation, scaling, identity management, and observability, so your agent runs securely in production without infrastructure overhead.

Deploy a custom agent on Amazon Bedrock AgentCore, a framework-agnostic managed runtime that supports agents built with Strands, LangChain, or custom Python. The agent uses Amazon Bedrock to analyze your infrastructure context against architectural patterns, AWS documentation, and best practices. AgentCore Runtime’s built-in tool gateway provides controlled, secure access to your AWS APIs during discovery.

AWS Fault Injection Service runs controlled chaos experiments with built-in safety mechanisms on your infrastructure. Pre-built actions cover common failure scenarios: terminating Amazon EC2 instances, injecting network latency, throttling API calls, failing over Amazon RDS databases, and disrupting Availability Zone connectivity in your environments.

AWS Systems Manager extends your resilience framework beyond the default AWS Fault Injection Service actions. You can create custom automation documents that codify recovery procedures and transform manual runbooks into automated self-healing responses. When you build custom actions, you take on responsibility for proper rollback procedures and service state restoration. Design these with the same rigor you’d apply to your production runbooks.

AWS Config continuously monitors your resource configurations and tracks changes. AWS Config rules validate that your resources comply with resilience policies. For example, they verify your Amazon RDS instances use Multi-AZ deployment and confirm your Auto Scaling groups span multiple Availability Zones.

Prerequisites

To implement this framework, you’ll need:

  1. AWS account with administrative access.
  2. AWS Identity and Access Management (IAM) permissions for: AWS Resilience Hub, AWS Fault Injection Service, Amazon Bedrock AgentCore, AWS Systems Manager, and AWS Config. For each service, follow the principle of least privilege. Refer to the respective service documentation for minimum required permissions.
  3. The Amazon Bedrock AgentCore Starter Toolkitcreates broad dev/test permissions by default. Scope these down to least-privilege before production deployment.
  4. AWS Command Line Interface (AWS CLI) installed and configured.
  5. Basic understanding of AWS CloudFormation or Terraform.
  6. Non-critical application available for testing.
  7. Estimated time: 4–6 hours for pilot implementation (with a team of 2–3 engineers who have working knowledge of your AWS environment).
  8. Cost awareness: This implementation creates billable AWS resources including AWS Resilience Hub, AWS Fault Injection Service, Amazon Bedrock AgentCore, AWS Systems Manager, AWS Config, and Amazon CloudWatch. Follow the cleanup procedures after testing to avoid ongoing charges.

Getting started

A phased rollout builds confidence before expanding scope if you’re new to chaos engineering.

Pilot phase (1–2 weeks, 2–3 engineers)

  1. Select a non-critical application with well-understood architecture from your portfolio.
  2. Enable AWS Config across the regions where your application runs.
  3. Package your discovery agent code using Strands, LangChain, or custom Python.
  4. Deploy the packaged agent on Amazon Bedrock AgentCore using the Amazon Bedrock AgentCore Starter Toolkit. AgentCore Runtime handles the compute, session management, and security so you can focus on the agent logic and discovery scope. The runtime maintains stateful working context (including tool state and memory) across the multi-step infrastructure discovery workflow.
  5. Run a baseline resilience assessment in AWS Resilience Hub to identify initial architectural gaps. Resilience Hub evaluates your architecture against Well-Architected best practices and establishes your starting resilience posture. The Bedrock agent you deploy in the next step builds on this baseline, discovering undocumented dependencies and generating targeted experiments that go beyond standard recommendations.

Verify: Check the AWS Resilience Hub console for a completed assessment report showing baseline resilience scores and identified gaps. You should see a resilience score for each disruption type (AZ, Region, Application) and a list of recommended actions.

Expansion phase (4–6 weeks, cross-functional team)

  1. Expand to 3–5 applications across different tiers after validating safety with your pilot.
  2. Configure automated test creation to develop experiments specific to each of your application’s architectures.
  3. Run controlled chaos experiments starting with 1% scope during your low-traffic periods.

Verify: Review the AWS Fault Injection Service console for experiment status “Completed” and check Amazon CloudWatch metrics to confirm the 1% scope was applied without triggering stop conditions.

  1. Analyze results to identify common patterns across your applications.

Enterprise scale (8–12 weeks, dedicated resilience team)

  1. Expand resilience assessments in your CI/CD pipelines to comprehensive, multi-account validation.
  2. Configure centralized reporting across organizational units.
  3. Set up cross-account experiment coordination.
  4. Distribute shared experiment templates across organizational units using AWS Organizations.
  5. Deploy distributed worker pools for parallel testing across your multiple applications.
  6. Implement executive dashboards using Amazon QuickSight to track resilience trends across your portfolio.

When to move to enterprise patterns

Consider adopting the enterprise deployment patterns in the next section when you meet any of these criteria:

  • You manage dozens of applications across multiple AWS accounts.
  • Multiple business units require differentiated resilience policies based on varying risk tolerances, compliance requirements, or customer SLAs.
  • Compliance requirements demand centralized audit trails across accounts.
  • Your testing cadence exceeds what a single-account setup can handle.

Design considerations for enterprise deployment

When you’re ready to scale beyond initial pilots, consider these enterprise deployment patterns that handle the unique challenges of managing resilience testing across large organizations.

Enterprise scalability patterns showing multi-account structure with tiered resilience policies and distributed worker pools

Figure 4. Enterprise scalability patterns

Diagram showing multi-account structure with native AWS Resilience Hub and AWS Organizations integration. A central management account connects to Production and Non-Production organizational units with distributed application accounts. The framework extends this native integration with cross-account FIS coordination (coordinated experiments across OUs) and shared FIS experiment templates distributed via Organizations. Priority-based scheduling shows Tier 1 Mission-Critical (weekly assessments, 100+ applications), Tier 2 Business-Critical (monthly assessments, 500+ applications), and Tier 3 Non-Critical (quarterly assessments, 1000+ applications). Distributed worker pools operate across multiple AWS regions (us-east-1, us-west-2, eu-west-1, ap-southeast-1) with centralized monitoring via Amazon EventBridge, Amazon QuickSight dashboards, Amazon CloudWatch logs, and Amazon SNS notifications.

Multi-account architecture

The next generation of AWS Resilience Hub supports modular resilience policies that you can assign at the system, user journey, or service level. Choose this multi-account strategy if you manage more than 100 applications across different business units. When you have thousands of applications, assessing your workloads simultaneously from a single account becomes impractical. Implement a hub-and-spoke model (a centralized architecture pattern where a central “hub” account manages shared services while “spoke” accounts contain individual workloads) with centralized resilience testing infrastructure and distributed application ownership. Deploy AWS Resilience Hub and AWS Fault Injection Service in your central management account. Your production accounts contain application workloads and local AWS Config recorders. The next generation of AWS Resilience Hub natively integrates with AWS Organizations, enabling central teams to define resilience policies and monitor posture across all accounts and regions from a single dashboard. This framework extends native multi-account visibility with cross-account experiment coordination and shared experiment templates across organizational units. For guidance on structuring your multi-account environment, see Best practices for a multi-account environment and the Organizing Your AWS Environment Using Multiple Accounts whitepaper.

Tiered resilience policies

Not every workload justifies the same resilience investment. Implement tiered resilience based on your business impact analysis. For example, mission-critical applications might target stricter recovery objectives (such as RTO < 15 minutes, RPO < 5 minutes, 99.99% availability) with comprehensive quarterly chaos experiments, while business-critical applications might set moderate targets (such as RTO < 1 hour, RPO < 15 minutes, 99.9% availability) with monthly validation, and non-critical applications might accept longer recovery windows with quarterly assessments. This tiered strategy optimizes costs by focusing resilience investments on your highest-impact workloads.

Security and compliance

Under the AWS Shared Responsibility Model, AWS is responsible for security of the cloud (infrastructure), while you are responsible for security in the cloud (your configurations, data, and access management). The controls described below are your responsibility to configure and maintain. Encrypt your data at rest using AWS Key Management Service (AWS KMS) with customer-managed keys. Implement separate encryption keys per environment to limit the scope of unauthorized access in your infrastructure. Enforce Transport Layer Security 1.3 (TLS 1.3), a cryptographic protocol that secures data transmission, for data in transit. AWS CloudTrail delivers complete audit trails of your resilience operations. Automated compliance monitoring through AWS Security Hub (which continuously evaluates resources against standards including CIS AWS Foundations Benchmark, PCI DSS, and AWS Foundational Security Best Practices), combined with AWS Config conformance packs, streamlines evidence collection for Service Organization Control 2 Type II (SOC 2 Type II), International Organization for Standardization 27001 (ISO 27001), Payment Card Industry Data Security Standard (PCI DSS), and Digital Operational Resilience Act (DORA), an EU regulation requiring financial institutions to test operational resilience.

AI agent security: The Amazon Bedrock AgentCore-hosted agent operates with scoped IAM roles following least-privilege principles. AgentCore Runtime’s MicroVM session isolation makes sure that each discovery session runs in a dedicated, ephemeral environment with no cross-session data leakage. The agent’s infrastructure access is read-only during discovery and cannot modify resources. Amazon Bedrock interactions occur within your AWS account boundary, and no customer data is used for training purposes. For additional guardrails, you can configure Amazon Bedrock Guardrails to filter agent outputs and enforce responsible AI policies.

Note: This framework supports your compliance efforts but does not guarantee compliance with any regulatory framework. Compliance is a shared responsibility. Consult your legal and compliance teams and qualified auditors to validate that your implementation meets your specific regulatory obligations.

Addressing common concerns

Progressive scope expansion and automated stop conditions help you verify experiments reveal weaknesses without causing outages in your environments. Starting with 1% of your resources limits potential impact to statistically insignificant traffic. Organizations have validated this strategy using progressive scope expansion and automated stop conditions.

Automated scenarios remove the expertise barrier. The AI-powered analysis examines your specific architecture to develop targeted experiments rather than demanding you design tests manually. AWS CloudTrail provides comprehensive audit trails of your chaos experiments, which can help support due diligence documentation for resilience testing. This evidence can contribute to your compliance documentation for SOC 2, ISO 27001, and other frameworks relevant to your organization. For financial services, the framework supports DORA scenario testing requirements.

Clean up

To avoid ongoing charges, delete the resources you created during implementation:

  1. Delete your AWS Fault Injection Service experiment templates. Warning: This permanently removes experiment history and results. Consider exporting experiment data before deletion if you need to retain this information for compliance or analysis purposes.
  2. Remove your Amazon Bedrock AgentCore agent deployment, runtime endpoints, and associated configurations.
  3. Delete your AWS Systems Manager automation documents. Warning: This removes your automation runbooks permanently. Back up any custom runbooks you may want to reuse in future implementations.
  4. Remove your Amazon CloudWatch alarms created for stop conditions.
  5. Delete any AWS Step Functions state machines created for approval workflows.
  6. Remove any Open Policy Agent configurations deployed for IaC validation.
  7. Delete Amazon QuickSight dashboards created for resilience tracking (if applicable). Warning: This removes resilience trend data and operational insights. Export dashboard data or save analysis snapshots before deletion.
  8. Remove Amazon EventBridge rules and Amazon SNS topics created for notifications (if applicable).

Your AWS Resilience Hub and AWS Config continue incurring minimal costs. Consider retaining them for ongoing resilience validation.

Conclusion

In this post, I showed you how to build a five-layer AI-powered resilience framework that automatically discovers dependencies, generates targeted experiments, and integrates with your CI/CD pipelines. Building on the next generation of AWS Resilience Hub’s native dependency discovery and generative AI-powered failure mode analysis, this framework adds automated experiment generation through Amazon Bedrock AgentCore, controlled execution via AWS Fault Injection Service, and continuous CI/CD validation through AWS Systems Manager, creating an end-to-end resilience pipeline that goes from discovery to prevention.

The next frontier is shifting even earlier, scanning your Infrastructure as Code and application code for resilience anti-patterns before a single resource is deployed. When your CI/CD pipeline can flag a missing circuit breaker or a single-AZ dependency at the pull request stage, prevention becomes truly proactive.

The progressive strategy (starting with your single application pilot, expanding to multiple applications, then scaling organization-wide) builds confidence while demonstrating value at each phase. Organizations often realize positive return on investment through prevented events and reduced MTTR.

The framework makes resilience testing accessible to you, removing the expertise barrier that traditionally prevented adoption. Start with the pilot phase outlined earlier and expand to your mission-critical systems as confidence builds.

Ready to get started? Pick a non-critical application from your portfolio, deploy the discovery agent, and run your first assessment this week. Then share what you found. We’d love to hear about the hidden dependencies your team uncovered.

Have you implemented chaos engineering in your organization? What challenges did you face? Share your experience in the comments below.

Next steps

For the latest Resilience Hub capabilities including native dependency discovery and generative AI-powered failure mode analysis, see Introducing the next generation of AWS Resilience Hub and the next generation documentation.

Start with the resources most relevant to where you are in your resilience journey:

If you’re just getting started:

  1. For more information about resilience policies and assessment capabilities, see AWS Resilience Hub documentation.
  2. For hands-on experience with chaos engineering, see AWS Fault Injection Service Workshop.

If you’re ready to build:

  1. For information about creating fault injection experiments using natural language through Amazon Bedrock, see Chaos engineering made clear: Generate AWS FIS experiments using natural language through Amazon Bedrock.
  2. For information about assessing application resilience with AWS Resilience Hub and AWS CodePipeline, see Continually assessing application resilience with AWS Resilience Hub and AWS CodePipeline.
  3. For additional fault injection experiment templates, see AWS Fault Injection Service Template Library.

If you’re scaling to production: 6. For information about hosting production AI agents at scale, see Amazon Bedrock AgentCore 7. For sample agent code and deployment templates, see the Amazon Bedrock AgentCore Starter Toolkit.

Systems will face failures. Discover and fix weaknesses before your customers experience them by shifting your resilience testing from reactive response to proactive prevention.

If you have questions or need guidance implementing the framework, contact AWS or reach out to your AWS Solutions Architect.


About the author

AWS Weekly Roundup: Claude Opus 4.8 on AWS, Aurora MySQL with Kiro Powers, and more (June 1, 2026)

Post Syndicated from Micah Walter original https://aws.amazon.com/blogs/aws/aws-weekly-roundup-claude-opus-4-8-on-aws-aurora-mysql-with-kiro-powers-and-more-june-1-2026/

In my last Week in Review post, I shared what I’d been hearing from customers in the AI-Driven Development Lifecycle (AI-DLC) workshops I’ve been delivering. Last week I was back at it, this time in Denver for a two-day AI-DLC workshop, where I helped facilitate 17 teams to deliver nearly 20 separate use cases in just two days. The pace of acceleration that AI-DLC unlocks—especially when paired with tools like Claude Code on Amazon Bedrock—is fundamentally changing how businesses operate. Traditional roles within software development teams are collapsing into smaller, AI-augmented squads, and the paradigm shift is beginning to take place right in front of us. To learn more about how to utilize various AI tools, visit the GitHub repository of AI-DLC workflow.

This shift is also reshaping how AWS account teams (solutions architects, customer solutions managers, and technical account managers) collaborate with customers. It’s becoming less about handing off advisory design documents and more about building alongside them in real time. It’s a genuinely exciting moment to be in the middle of the change, and this week’s headline launch — Anthropic’s most capable model yet, now on AWS — is going to push that pace even further.

Now, let’s get into this week’s AWS news…

Headlines
Claude Opus 4.8 on AWS — Anthropic’s most capable generally available model is now accessible through both Amazon Bedrock and the Claude Platform on AWS. Opus 4.8 is built for agentic coding, knowledge work, and extended autonomous task execution — it sustains longer autonomous sessions with deeper reasoning, recovers from errors, and synthesizes information across lengthy documents. For coding workloads, it reads codebases like an engineer, plans before it edits, and holds context across long sessions. On Amazon Bedrock, you get AWS-managed features like Guardrails, Knowledge Bases, and data residency; on the Claude Platform on AWS, you get Anthropic’s native APIs unified with AWS billing. To learn more, visit the deep-dive blog post.

Last week’s launches
Here are some launches and updates from this past week that caught my attention:

  • Introducing the next generation of AWS Resilience Hub — A reimagined Resilience Hub gives SREs and developers a unified framework to define resilience standards, evaluate applications against them, and demonstrate compliance across an entire portfolio. It introduces modular resilience policies (covering service-level objectives (SLOs), multi-AZ/Region DR, and data recovery), business-oriented application modeling, generative AI-powered assessments aligned with the Well-Architected and Resilience Analysis Frameworks, and automatic dependency discovery via DNS query log analysis. Integration with AWS Organizations enables organization-wide resilience management from a single delegated administrator account.
  • Introducing the next generation of Amazon OpenSearch Serverless for building agentic AI applications — Amazon OpenSearch Serverless is now a fully managed search and vector engine purpose-built for agentic AI applications. It scales from zero to thousands of requests per second—roughly 20x faster than the prior generation—delivers up to 60% cost savings versus peak-provisioned clusters, and adds GPU acceleration plus new SEARCH and VECTORSEARCH collection types. Native integrations with Vercel, Kiro, Claude Code, and Cursor through OpenSearch Agent Skills make it straightforward to plug into your agent stack.
  • New assessment capabilities in AWS Transform — AWS Transform expands with new tools to help you build migration business cases and evaluate TCO before moving workloads to AWS. You can ingest data from RVTools exports, CMDB data, the AWS Transform discovery tool, and third-party discovery tools, then run what-if scenarios across region, utilization, and service mapping for EC2, FSx, S3, SQL Server on EC2, and virtual desktops. The release also adds Agentic Readiness Analysis (ARA) and Modernization Analysis (MODA), which scan code repositories in 5 to 30 minutes per repo to surface severity-tagged findings with file-level evidence and AWS-mapped remediation guidance.
  • Amazon Aurora MySQL with Kiro Powers — Aurora MySQL now integrates with Kiro Powers, drawing from a curated repository of pre-packaged MCP servers, steering files, and hooks validated by Kiro partners. Developers can execute both data plane tasks (queries, schema management) and control plane tasks (cluster management) in natural language, with dynamic guidance for Aurora MySQL Serverless scaling, RDS-to-Aurora migration, and replication setup. The companion Database Blog post explains how the agent produces the API calls, SQL, and configuration for you to review and run — available via one-click install from the Kiro IDE or webpage.
  • Amazon WorkSpaces Applications now supports Windows Desktop OS — You can now bring your own Windows Desktop licenses to Amazon WorkSpaces Applications and stream full Windows desktops and applications from AWS-hosted dedicated hardware. BYOL eliminates OS fees (you pay only for compute and streaming infrastructure), supports eligible Microsoft 365 Apps for enterprise, and gives users a matching experience between local and remote environments — same workflows, shortcuts, and navigation in both.

For a full list of AWS announcements, be sure to keep an eye on the What’s New with AWS page.

Other AWS news
Here are some additional posts and resources that you might find interesting:

For a full list of AWS blog posts, be sure to keep an eye on the AWS Blogs page.

Learn more about AWS, browse and join upcoming AWS-led in-person and virtual events, startup events, and developer-focused events as well as AWS Summits and AWS Community Days. Join the AWS Builder Center to connect with builders, share solutions, and access content that supports your development.

That’s all for this week. Check back next Monday for another Weekly Roundup!

-Micah

Introducing the next generation of AWS Resilience Hub for generative AI-based SRE resilience journey

Post Syndicated from Channy Yun (윤석찬) original https://aws.amazon.com/blogs/aws/introducing-the-next-generation-of-aws-resilience-hub-for-generative-ai-based-sre-resilience-journey/

Today, we’re announcing the next generation of AWS Resilience Hub with a significantly expanded experience that brings together a new application model, dependency discovery assessment, generative AI-powered failure mode analysis, modular resilience policies, and organization-wide reporting.

Organizations running hundreds of applications share a common challenge: availability is a top concern, yet there is no consistent way to set resilience goals, measure progress, or prove compliance across a portfolio. Teams set different standards, use different tools, and struggle to exchange information about whether applications actually meet expectations.

The next generation of AWS Resilience Hub changes this by giving Site Reliability Engineers (SREs) and development teams a structured way to align on resilience policy expectations, help application teams achieve them, and demonstrate compliance through testing. With integration into AWS Organizations, teams can now evaluate resilience at scale, identify failure modes, discover hidden dependencies, and report on progress across the enterprise.

The next generation of Resilience Hub walks you through your resilience journey and to help you there are the following concepts built into it.

  • Resilience policy: You can define your resilience expectations through modular, composable requirements. Rather than choosing a single rigid policy type, you construct policies by selecting the requirements that matter to your application, such as service level objective (SLO), multi-AZ and multi-Region disaster recovery, and data recovery requirements.
  • Business-level understanding: You can use new application modeling through critical end-user paths that map directly to business outcomes. Systems represent a business application, user journeys describe critical business paths, and services are the deployable units comprising AWS resources, code, and observability. Resilience Hub automatically discovers and maps them into a topology showing how resources connect.
  • AI failure mode assessments: You can run generative AI-powered assessments that analyze your services against your defined resilience policies, AWS Well-Architected best practices, and the AWS Resilience Analysis Framework. These assessments identify potential failure modes and provide actionable recommendations.
  • Dependency discovery assessment: You can automatically discover AWS services, internal endpoints, and third-party endpoints that your services depend on. This dependency assessment uses DNS query log analysis to identify dependencies you may not know about—including unexpected cross-region calls or critical third-party dependencies.

The next generation of AWS Resilience Hub in action
To get started, you configure a resilience policy, set up your first system and service, run a failure mode assessment, review the results, and implement the findings.

Before you begin, you should set up the invoker IAM role, which grants Resilience Hub read-only access to your AWS resources, cross-account roles (if not using AWS Organizations), or service-linked roles (SLRs) with AWS Organizations. Resilience Hub also integrates with AWS Organizations to enable organization-wide resilience management from a single delegated administrator account. This eliminates the need to log in to individual accounts to assess resilience posture across your enterprise. To learn more, visit For prerequisite details in the AWS Resilience Hub User Guide.

To configure a resilience policy, choose Create policy in the Policies menu through the AWS Resilience Hub console. Enter a policy name, description, and choose resilience requirements. For example, you can create a reusable policy for multi-Region disaster recovery used in financial applications—including 99.95% availability SLO, 15-minutes RTO, 5-minutes RPO for multi-Region disaster recovery, and disaster recovery approach that aligns with your RTO and RPO requirements.

If you choose data recovery requirements, you can define the data recovery time objective for restoring from backups for each service associated with this policy.

To create your first system representing your business application, choose Create a system in the Systems menu. Optionally, you can enable AWS Organizations account access for this system.

Now you can create a service that represents a deployable unit, like one of your microservices, and associate it with your system, and tell Resilience Hub where to find your resources. Enter a service name, for example, stock-exchange-service, choose your resilience policy and invoker AWS IAM role name. You can choose service Regions, service resources such as your resource tags, AWS CloudFormation stack, Terraform state file location, or Amazon EKS cluster and namespace.

When you enable dependency discovery for this service, AWS examines your VPC query logs for the VPCs associated with the resources in your service. You can disable this feature anytime from the dependency discovery settings in the service details page.

Now, you can run your first assessment with the service creation complete and a policy applied. Choose Run failure mode assessment in your service page and wait for the assessment to complete.

During the assessment, Resilience Hub assumes your invoker role, reads resources from your configured input sources, identifies parent-child relationships, queries the application topology service to map connections between resources, and builds a topology showing data flow, containment, and permissions.

By choosing Service topology, you can see service resources grouped by service functions in the graph, table, or JSON format.

By choosing Failure mode guidance, you can add assertions used to guide the agents while performing the failure mode assessment. Assertions are either generated by the agent or added by users. You can update them to improve assessment accuracy.

Once the assessment is complete, you can review findings and recommendations in the Assessment tab of your service page. Each finding tells you what the failure mode is, why it matters for your architecture, how to fix it, and which policy requirement it relates to.

You can choose Mark as resolved to implement the recommendation or Mark as irrelevant if the finding doesn’t apply to your use case.

If you’re an existing Resilience Hub customer, Resilience Hub provides migration APIs to simplify the transition of your previous applications. These APIs convert your previous assessment policies to new resilience policies, map your previous applications to the new model, such as multiple related applications to one system with multiple services.

For more information about new features, visit the AWS Resilience Hub User Guide.

Now available
The next generation of AWS Resilience Hub is now generally available in AWS commercial Regions where Resilience Hub is available. For Regional availability and the future roadmap, visit the AWS Capabilities by Region.

Resilience Hub uses a new service-based pricing model. Pricing includes two failure mode assessments per month for services, and optionally automated dependency assessment. You can try AWS Resilience Hub free. For pricing details, visit the AWS Resilience Hub pricing page.

Give the new AWS Resilience Hub a try in the Resilience Hub console and send feedback to AWS re:Post for Resilience Hub or through your usual AWS Support contacts.

— Channy