Tag Archives: Technical How-to

Securing your Amazon S3 buckets: Identifying and remediating over-permissioned access

2026-08-07 Hetal Kolekar

Post Syndicated from Hetal Kolekar original https://aws.amazon.com/blogs/security/securing-your-amazon-s3-buckets-identifying-and-remediating-over-permissioned-access/

Misconfigured Amazon Simple Storage Service (Amazon S3) buckets can expose your data to unauthorized access. Without proactive review, S3 bucket policies or Access Control Lists (ACLs) configured with broad access may go unnoticed in your environment. In this post, you learn how to identify and fix over-permissioned S3 buckets across your AWS environment, along with best practice recommendations and automation opportunities to help you prevent security gaps. This post provides a workflow framework and methodology recommendations for your security team to adapt. The focus of this post is on the what and why rather than a prescriptive implementation. You will need to customize the approach based on your organization’s requirements and existing security tooling.

This solution is intended for security engineers, cloud architects, and DevOps teams managing single- or multiple-account AWS environments with Amazon S3 workloads that require access management.

Prerequisites

Before you begin, make sure you have the following in place:

Ensure you have an AWS account with permissions to create AWS Lambda functions, AWS Identity and Access Management (IAM) roles, and Amazon Simple Notification Service (Amazon SNS) topics.
You will also need the AWS Command Line Interface (AWS CLI) or AWS SDK installed locally.
For multi-account environments, AWS Organizations should already be configured.
Basic familiarity with IAM policies and Python will help you customize the solution to your needs.

Solution overview

This solution uses a five-phase workflow diagram to detect, remediate, and continuously monitor over-permissioned S3 buckets across your AWS accounts. The following workflow diagram illustrates the high-level end-to-end process for identifying and remediating over-permissioned S3 buckets across your Amazon Web Services (AWS) environment.

Figure 1: Amazon S3 over-permissive access – Detection, remediation, monitoring and cleanup workflow

The diagram in Figure 1 consists of five phases:

Setup and prerequisites – Configure AWS Organizations or multi-account access, designate a central security account, deploy AWS Config across all accounts, and enable AWS Security Hub with a central administrator.
Detection and identification – Deploy AWS Config rules (such as s3-bucket-public-read-prohibited and s3-bucket-public-write-prohibited) and run an audit Lambda function that scans each S3 bucket. The function checks three areas: Public Access Block configuration, bucket policy status, and bucket ACL grants. Buckets with issues are added to a risky buckets list. The function then generates a report in CSV and JSON format, uploads it to an output S3 bucket, and sends an SNS alert.
Remediation – Address findings using one or more approaches – Apply restrictive bucket policies to deny public read/write access and restrict access to specific IAM principals; deploy a remediation Lambda function to automatically update bucket policies and disable public access settings; or use CloudFormation StackSets to deploy standardized policies across multiple accounts.
Continuous monitoring – Schedule the audit Lambda function for recurring scans (daily or weekly) using Amazon EventBridge. Use EventBridge to detect policy changes, configure automated notifications for new violations, enable IAM Access Analyzer for S3 to identify external access, and run regular compliance scans.
Resource cleanup – Review and delete resources created during the audit that are no longer needed, including Lambda functions and IAM roles, EventBridge rules, SNS topics and subscriptions, audit output S3 buckets, AWS Config rules, and Security Hub (if enabled only for this audit).

Cost considerations

This section covers the AWS services used in this solution and their associated costs so you can estimate spend before deployment. The primary cost drivers are AWS Config and Security Hub, which scale with the number of accounts and resources you monitor. Lambda, Amazon EventBridge, Amazon SNS, and Amazon S3 typically add minimal costs for most environments. Start with a pilot in one or two accounts to validate costs before scaling.

AWS Config – Charges per configuration item recorded and per rule evaluation. Costs scale with the number of accounts and resources tracked.
Security Hub – Charges per account per AWS Region for security checks and finding ingestion.
Lambda – Charges per request and per GB-second of compute time.
EventBridge – Scheduled rules are free. Custom event bus usage might incur charges.
Amazon SNS – Charges per notification delivered.
Amazon S3 – Storage costs for audit report output files. Minimal for most environments.
AWS IAM Access Analyzer – Check the AWS IAM Access Analyzer pricing page to understand which features have costs associated with them.

Check the service pricing pages for current rates. Use the AWS Pricing Calculator to estimate costs for your specific environment before enabling services across all accounts. Consider starting with a pilot in one or two accounts to validate costs before scaling.

Detect and report over-permissioned buckets

This section walks you through setting up the audit environment, deploying the Lambda-based scanner, and generating reports of over-permissioned S3 buckets across your accounts. Follow these steps to identify over-permissioned S3 buckets in your multi-account environment, starting with preparing your environment for an Amazon S3 audit.

To set up the multi-account audit environment:

Set up AWS Organizations or multi-account access. Set up centralized management of your AWS accounts using AWS Organizations or configure cross-account IAM roles.
Choose a central security account. Choose one account as your security/audit account. This account will run the audit Lambda function and collect results from member accounts.
Create an Amazon SNS topic for alerts. Subscribe your security team to receive notifications when over-permissioned buckets are detected. Note the topic Amazon Resource Name (ARN) from the output—you will need it when creating the Lambda execution role (step 6) and the Lambda function (step 9). Confirm the email subscription before testing; Amazon SNS doesn’t deliver alerts until the subscription is confirmed. Learn more in the Amazon SNS Developer Guide.
(Optional): Create an S3 bucket for audit reports. If you plan to use Script v2 for historical reporting and trend analysis, create a dedicated bucket now. Skip this step if you only need real-time alerts using Script v1.
Plan cross-account IAM roles. The central security account needs permission to scan member accounts. Design cross-account roles that:
1. Grant minimum Amazon S3 read permissions (list buckets, read policies, ACLs, public access configurations).
2. Include an external ID condition to mitigate the confused deputy problem.
3. Can be deployed consistently using AWS CloudFormation StackSets.
4. See the IAM documentation on creating cross-account roles, The confused deputy problem, and IAM security best practices for additional guidance on role configuration and trust policies.
  Note: The specific trust policy and permissions policy for your cross-account roles will depend on organizational requirements. Work with your IAM administrators to grant minimum necessary access for the audit function.
Create the Lambda execution role. Create an IAM role for your Lambda function with the permissions it needs to scan buckets, publish alerts, and write logs. Apply the principle of least privilege—grant only the minimum Amazon S3 read permissions required for the audit (such as, listing buckets, reading bucket policies, ACLs, and public access block configurations), Amazon SNS publish permission for the alert topic created in step 3, Amazon S3 write permission for the output bucket created in step 4 (Script v2), and Amazon CloudWatch Logs permissions. For multi-account scanning, also include sts:AssumeRolepermission for the cross-account role ARNs created in step 5. The AWS Lambda execution role documentation has instructions on creating and configuring execution roles.
To deploy the S3 audit solution Deploy the audit components
1. Enable AWS Config in member accounts. AWS Config provides compliance monitoring and can detect when S3 buckets are created or modified with public access settings. This will enable the Lambda-based audit to receive real-time detection between scheduled scans. The AWS Config Developer Guide has setup instructions. Deploy pre-defined AWS Config rules to identify overly permissive settings. These managed rules provide automated compliance checking. When AWS Config detects violations, it sends findings to Security Hub (configured in step 8) for centralized visibility alongside the Lambda audit results.
  - s3-bucket-public-read-prohibited
  - s3-bucket-public-write-prohibited
  - Create AWS Config rules for specific permission patterns. For the full list of available rules, see the AWS Config managed rules reference
Enable Security Hub for centralized visibility. Enable AWS Security Hub in member accounts and configure the central security account as the administrator. Security Hub aggregates findings from AWS Config rules (step 7), IAM Access Analyzer (enabled later), and can receive custom findings from your Lambda audit function, providing a single dashboard for Amazon S3 security issues across your organization. See the Security Hub User Guide for setup details.
Deploy the audit Lambda function. Deploy a Python Lambda function using the Boto3 library to list S3 buckets, check their policies, ACLs, and IAM permissions, and identify over-permissioned buckets. See the example scripts that follow.

Important: These code examples aren’t production ready. Adapt them to meet your organization’s requirements and test them in a non-production environment before deployment.

Choose your approach:

Script v1 – Best for immediate SNS alerts when issues are detected.
Script v2 – Best for historical reports, trend analysis using BI tools.
Both scripts – Best for different schedules and ongoing needs.

Audit Lambda function – Example script v1 (Scan and alert)

The following is an example of a Lambda function script for reference purposes. Review, adapt, and test before use in your environment, it scans all S3 buckets in the current account and checks for:

Public Access block configuration gaps
Bucket policies that allow public access
ACL grants to AllUsers

Note: Replace placeholder values with actual values before deployment:

<REGION>– Your AWS Region (for example, us-east-1)

<ACCOUNT_ID>– Your 12-digit AWS account ID

<TOPIC_NAME>– The name of your SNS topic created in step 3

import boto3
import json

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    sns = boto3.client('sns')
    risky_buckets = []
    errors = []

    try:
        buckets = s3.list_buckets()['Buckets']
    except Exception as e:
        return {'statusCode': 500, 'body': f'Failed to list buckets: {str(e)}'}

    for bucket in buckets:
        bucket_name = bucket['Name']
        issues = []

        try:
            # Check Public Access Block — all four settings should be enabled
            try:
                pab = s3.get_public_access_block(Bucket=bucket_name)
                config = pab['PublicAccessBlockConfiguration']
                if not all([
                    config.get('BlockPublicAcls'),      # Block new public ACLs
                    config.get('BlockPublicPolicy'),     # Block new public bucket policies
                    config.get('IgnorePublicAcls'),      # Ignore existing public ACLs
                    config.get('RestrictPublicBuckets')   # Restrict access to public buckets
                ]):
                    issues.append('Public Access Block not fully enabled')
            except s3.exceptions.NoSuchPublicAccessBlockConfiguration:
                issues.append('No Public Access Block configured')

            # Check bucket policy — flag if policy status is public
            try:
                policy_status = s3.get_bucket_policy_status(Bucket=bucket_name)
                if policy_status['PolicyStatus']['IsPublic']:
                    issues.append('Bucket policy allows public access')
            except s3.exceptions.NoSuchBucketPolicy:
                pass  # No bucket policy is acceptable

            # Check bucket ACL
            acl = s3.get_bucket_acl(Bucket=bucket_name)
            for grant in acl.get('Grants', []):
                grantee = grant.get('Grantee', {})
                uri = grantee.get('URI', '')
                # 'AllUsers' = anonymous public access
                # 'AuthenticatedUsers' = any AWS account (still overly permissive)
                if grantee.get('Type') == 'Group' and ('AllUsers' in uri or 'AuthenticatedUsers' in uri):
                    issues.append('Bucket ACL grants public access')
                    break

            if issues:
                risky_buckets.append({'bucket': bucket_name, 'issues': issues})

        except Exception as e:
            errors.append(f'{bucket_name}: {str(e)}')

    # Send alert if risky buckets found
    if risky_buckets:
        message = f'Found {len(risky_buckets)} buckets with public access:\n\n'
        for item in risky_buckets:
            message += f"  {item['bucket']}: {', '.join(item['issues'])}\n"

        sns.publish(
            TopicArn='arn:aws:sns:<REGION>:<ACCOUNT_ID>:<TOPIC_NAME>',
            Subject='S3 Public Access Alert',
            Message=message
        )

    return {
        'statusCode': 200,
        'body': json.dumps({
            'risky_buckets': risky_buckets,
            'errors': errors,
            'total_checked': len(buckets)
        })
    }

Multi-account scanning: This script scans the current account only. To scan across member accounts, see the Multi-account extension section later in this post.

Audit Lambda function – Example script v2 (CSV and JSON report)

The following is an example Lambda function script for reference purposes. Before deploying any script, review error handling, logging, output structure, and permissions. This script generates CSV and JSON output files and uploads them to an S3 bucket for reporting and business intelligence (BI) dashboard integration.

You can deploy both functions with different EventBridge schedules, for example, Script v1 daily for alerts and Script v2 weekly for reports.

Note: Before you deploy this script, replace <OUTPUT_BUCKET_NAME> with the S3 bucket you created for audit reports in step 4.

import boto3
import csv
import json
import os

def lambda_handler(event, context):
    s3 = boto3.client('s3')
    buckets = s3.list_buckets()['Buckets']

    full_access_buckets = []
    for bucket in buckets:
        bucket_name = bucket['Name']
        try:
            bucket_policy = s3.get_bucket_policy(Bucket=bucket_name)['Policy']
            policy = json.loads(bucket_policy)
            for statement in policy['Statement']:
                if (statement['Effect'] == 'Allow'
                    and statement['Principal'] == '*'
                    and 'Action' in statement
                    and 's3:*' in statement['Action']):
                    full_access_buckets.append({'BucketName': bucket_name})
                    break
        except s3.exceptions.ClientError as e:
            if e.response['Error']['Code'] != 'NoSuchBucketPolicy':
                print(f'Error checking bucket policy for {bucket_name}: {e}')

    # Output CSV
    csv_output = os.path.join('/tmp', 'full_access_buckets.csv')
    with open(csv_output, 'w', newline='') as csvfile:
        writer = csv.DictWriter(csvfile, fieldnames=['BucketName'])
        writer.writeheader()
        writer.writerows(full_access_buckets)

    # Output JSON
    json_output = os.path.join('/tmp', 'full_access_buckets.json')
    with open(json_output, 'w') as jsonfile:
        json.dump(full_access_buckets, jsonfile, indent=2)

    # Upload to Amazon S3
    output_bucket = '<OUTPUT_BUCKET_NAME>'
    s3.upload_file(csv_output, output_bucket, 'full_access_buckets.csv')
    s3.upload_file(json_output, output_bucket, 'full_access_buckets.json')

    return {
        'statusCode': 200,
        'body': json.dumps(f'CSV and JSON files uploaded to {output_bucket}')
    }

Important: If this function runs on a schedule, consider implementing a file naming strategy with timestamps to prevent overwriting previous reports or establish a lifecycle policy to manage retention. Include the output bucket in your cleanup procedures when the auditing process is no longer needed.

What if no over-permissioned buckets are found?

If the audit scan returns zero risky buckets, document the clean baseline for future comparison and move to the verification and monitoring phase to so new buckets or policy changes don’t introduce risk over time.

Multi-account extension

The preceding example scripts scan buckets in the current account only. To scan across member accounts in your organization, add the following AssumeRole logic. This function assumes the cross-account IAM role you created during setup, then returns an Amazon S3 client with temporary credentials for each member account.

Note: Before you deploy, configure the following Lambda environment variables:

<MEMBER_ACCOUNTS> – Comma-separated list of 12-digit account IDs to scan (for example, 111111111111,222222222222)

<CROSS_ACCOUNT_ROLE_NAME> – The IAM role name created in each member account (for example, S3AuditRole)

<EXTERNAL_ID> – The external ID configured in the trust policy (for example, s3-audit-external-id)

import boto3
import os

def get_member_s3_clients():
    """
    Assumes the cross-account audit role in each member account
    and returns a list of (account_id, s3_client) tuples.
    """
    sts = boto3.client('sts')
    member_accounts = os.environ.get('<MEMBER_ACCOUNTS>', '').split(',')
    cross_account_role_name = os.environ.get('<CROSS_ACCOUNT_ROLE_NAME>')
    external_id = os.environ.get('<EXTERNAL_ID>')

    clients = []
    for account_id in member_accounts:
        account_id = account_id.strip()
        if not account_id:
            continue

        try:
            assumed_role = sts.assume_role(
                RoleArn=f'arn:aws:iam::{account_id}:role/{cross_account_role_name}',
                RoleSessionName='S3AuditSession',
                ExternalId=external_id
            )

            # Create S3 client with assumed credentials
            s3_client = boto3.client(
                's3',
                aws_access_key_id=assumed_role['Credentials']['AccessKeyId'],
                aws_secret_access_key=assumed_role['Credentials']['SecretAccessKey'],
                aws_session_token=assumed_role['Credentials']['SessionToken']
            )
            clients.append((account_id, s3_client))

        except Exception as e:
            print(f'Failed to assume role in account {account_id}: {e}')

    return clients

To scan each member account, replace the single-account s3.list_buckets() call with a loop over member accounts:

def lambda_handler(event, context):
    all_risky_buckets = []
    all_errors = []

    # Scan each member account
    for account_id, s3_client in get_member_s3_clients():
        try:
            buckets = s3_client.list_buckets()['Buckets']
            for bucket in buckets:
                # ... same scanning logic as the single-account scripts ...
                # Use s3_client instead of s3 for each API call
                pass
        except Exception as e:
            all_errors.append(f'Account {account_id}: {e}')

    # ... same alerting/reporting logic ...

The Lambda execution role in the central security account needs sts:AssumeRole permission for the cross-account role ARNs. Add this to the execution role policy you created in step 5.

Remediate elevated access

This section describes how to fix over-permissioned buckets using account-level controls, bucket policies, and optional automation. Any elevated access that you find needs to be remediated.

Enable Amazon S3 Block Public Access (account level)

Before applying individual bucket policies, enable Amazon S3 Block Public Access at the account level. This prevents buckets in the account from being made public, regardless of individual bucket policies or ACLs. See theS3 Block Public Access documentation for configuration details. See the following example AWS CLI command; replace <ACCOUNT_ID> with the ID of the account you’re using to manage resource access:

aws s3control put-public-access-block \
  --account-id <ACCOUNT_ID> \
  --public-access-block-configuration \
BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

For multi-account environments, deploy this setting across member accounts using AWS CloudFormation StackSets or AWS Organizations service control policies (SCPs).

Important: Before enabling account-level S3 Block Public Access, check whether any workloads need public bucket access (for example, static website hosting, public dataset sharing). Coordinate with your application teams to identify any exceptions.

Remediate using bucket policies

Implement bucket policies that restrict access to specific IAM users, roles, or accounts. When crafting policies, apply the principle of least privilege and include only the actions and principals required for your use case.

Example S3 bucket policy: deny public read/write access. Modify the resource ARN, actions, and conditions to match your requirements:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": [
        "s3:PutObject", "s3:PutObjectAcl",
        "s3:GetObject", "s3:GetObjectAcl",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>/*",
      "Condition": {
        "StringEquals": {
          "s3:x-amz-acl": ["public-read", "public-read-write"]
        }
      }
    }
  ]
}

Example S3 bucket policy: restrict access to specific IAM principals. Replace <ACCOUNT_ID>, <USERNAME>, and <ROLE_NAME>:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowObjectAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<ACCOUNT_ID>:user/<USERNAME>",
          "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"
        ]
      },
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>/*"
    },
    {
      "Sid": "AllowBucketAccess",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::<ACCOUNT_ID>:user/<USERNAME>",
          "arn:aws:iam::<ACCOUNT_ID>:role/<ROLE_NAME>"
        ]
      },
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": "arn:aws:s3:::<BUCKET_NAME>"
    }
  ]
}

See the Amazon S3 bucket policy documentation for additional examples and guidance.

Automate remediation with Lambda or CloudFormation StackSets (optional):

You can also remediate using Lambda or CloudFormation Stacksets:

Create Lambda functions to automatically update bucket policies or disable public access settings for flagged buckets
Use CloudFormation StackSets to deploy standardized bucket policies and S3 Block Public Access settings across multiple accounts

Verify your remediation

This section explains how to confirm that your fixes are effective before moving to ongoing monitoring. After applying remediation, verify the fix is effective before setting up ongoing monitoring:

Re-run the audit Lambda function – Confirm the previously flagged buckets no longer appear in the risky buckets list.
Check Security Hub compliance – Verify the compliance status has changed from FAILED to PASSED for Amazon S3-related controls.
Validate with IAM Access Analyzer – Review findings for the remediated S3 buckets. Active findings should resolve automatically after public access is removed.
Test application functionality – Confirm that legitimate workloads continue to function correctly.

Document the verification results for your auditing needs. If any S3 buckets still show issues, investigate whether the policy was applied correctly or if there are conflicting permissions.

Automation opportunities

This section covers optional strategies to automate ongoing detection and maintain your security posture without manual intervention.

(Optional) Schedule recurring scans with Amazon EventBridge
- Regular security scans help identify new issues arising from configuration changes or newly created S3 buckets. When new security risks are detected, Amazon SNS sends an alert and automatically initiates the remediation phase (Workflow 2 in Figure 1). To avoid repeated alerts, you can configure the audit Lambda function to run on a schedule and compare current results with the previous baseline to generate notifications when new findings are discovered.
- For ongoing monitoring, you can schedule the audit Lambda function to run on a recurring basis using EventBridge. Create a scheduled rule with a cron expression (for example, daily at 6:00 AM UTC or weekly on Mondays), add the Lambda function as the target, and grant EventBridge permission to invoke it. See Amazon EventBridge scheduling documentation for instructions on creating scheduled rules and configuring targets.
Enable IAM Access Analyzer for Amazon S3
- IAM Access Analyzer monitors bucket policies, ACLs, and access points to identify buckets accessible from outside your account or organization. Create an analyzer scoped to your organization or individual account, then review findings to identify unintended external access. Findings automatically flow into Security Hub when both services are enabled, giving you a dashboard view for Amazon S3 security findings. See the IAM Access Analyzer documentation for setup and usage instructions.
Automate notifications for policy drift
- Recurring scans might surface new findings from policy drift or newly created buckets. When new risks are detected, Amazon SNS alert triggers and the remediation cycle repeat (as shown in Workflow 2 in Figure 1) sends email notifications. Configure the audit Lambda function to compare current scan results against the previous baseline and alert on new findings for ongoing reviews.

Clean up

This section lists the resources created during this walkthrough that you should review and remove when they are no longer needed. If the following services were not previously active in your account, leaving them enabled might result in additional ongoing charges. See the Cost considerations section for details. Review and remove unused resources to optimize costs.

Delete or disable the following script-generated resources if they’re not required after outputs are generated. Focus first on Lambda functions and EventBridge rules if you’re not running recurring scans. If you enabled AWS Config or Security Hub specifically for this audit, evaluate whether you need them for other compliance requirements before disabling.

Lambda – Functions, IAM roles, and policies created for auditing
Amazon EventBridge – Scheduled rules created for recurring audit triggers
Amazon SNS – Topics and subscriptions created for notifications
Amazon S3 – Buckets containing script-generated audit output files
AWS Config – Rules and recorders if no longer needed for compliance
Security Hub – Disable if enabled solely for this audit
IAM Access Analyzer – Delete the analyzer if no longer needed for ongoing monitoring

Note: Be careful when deleting data and consider temporarily disabling services first to check for dependencies. Only delete resources generated as part of your audit outputs. Verify you have retained any necessary results before proceeding. Verify resources are not used by other workloads before deletion.

Best practices

This section provides recommendations to maintain secure Amazon S3 configurations long-term. To learn more about maintaining secure Amazon S3 configurations, review the AWS documentation links provided in the conclusion. The following recommendations aren’t exhaustive. Adapt and extend them based on your organization’s evolving security requirements and AWS best practices guidance. After you’ve fixed existing issues, these practices help you maintain secure Amazon S3 configurations.

Start with account-level controls – Enable S3 Block Public Access at the account level. This prevents buckets from becoming public even if someone misconfigures an individual bucket policy. For multi-account environments, enforce this through AWS Organizations SCPs.
Automate detection – Use IAM Access Analyzer to detect external access. Schedule your audit Lambda function with EventBridge to catch new issues weekly or daily, depending on your change frequency. Compare scan results against previous baselines to identify drift.
Standardize across accounts – Use CloudFormation StackSets to deploy the same secure configuration to all accounts in your organization, reducing the chance of configuration drift. Use StackSets for IAM roles, AWS Config rules, and S3 Block Public Access settings.

Additional security measures

Regularly review and rotate cross-account IAM role credentials and external IDs
Implement Amazon S3 server-side encryption (SSE-S3 or SSE-KMS) for data at rest
Enable S3 access logging and AWS CloudTrail data events for audit trails

Conclusion

This section summarizes what you accomplished and suggests next steps to maintain your S3 security posture. By implementing the detection, remediation, and monitoring workflow outlined in this post, you can proactively identify and secure over-permissioned S3 buckets across your AWS environment. To maintain your ongoing security posture, enable IAM Access Analyzer for continuous monitoring and schedule recurring audits with EventBridge. To learn more about Amazon S3 security best practices, see Security best practices for Amazon S3

For more information:

If you have feedback about this post, submit comments in the Comments section below.

Automate certificates with ACME support in AWS Certificate Manager

2026-08-07 Anthony Harvey

Post Syndicated from Anthony Harvey original https://aws.amazon.com/blogs/security/automate-certificates-with-acme-support-in-aws-certificate-manager/

Customers tell us that managing TLS certificates at scale is one of their biggest operational concerns. The Certification Authority Browser Forum (CA/Browser Forum) has mandated a phased reduction in maximum certificate validity for public certificates. By March 2027, the maximum validity drops to 100 days. By March 2029, it lasts for 47 days. For an organization managing 1,000 certificates, the final transition means roughly 30 renewal events every day. Renewal and rotations of renewed certificates at that cadence isn’t something manual processes or ticket-driven workflows can sustain at scale.

We recently announced Automated Certificate Management Environment (ACME) protocol support in AWS Certificate Manager (ACM). With this launch, you can use the ACME clients your teams already know, including popular open source tools like certbot, cert-manager, acme.sh, and win-acme, to automate public certificate issuance and renewal for your infrastructure. Customers that are using third-party certificate authorities (CAs) can point their existing ACME-compatible clients at ACM instead of their current CA, with minimal reconfiguration. This applies whether it’s running on Amazon Web Services (AWS), on premises, or in a hybrid environment. Certificates created through ACME are registered in ACM, giving you a unified view of your entire certificate inventory.

This post covers how the feature works, how to get started, and the controls and best practices to help you manage certificate issuance at scale.

Background

ACME is an open source protocol that automates the process of verifying domain ownership and issuing certificates and has become a standard mechanism for certificate automation. While ACM has long provided managed certificate issuance and renewal for AWS-integrated services such as Elastic Load Balancing (ELB), Amazon CloudFront, and Amazon API Gateway, many customers also need to automate certificates for their own infrastructure, including servers they manage in their data centers, Kubernetes clusters, Internet of Things (IoT) fleets, and hybrid environments. Until now, those customers had to turn to external providers. This launch brings the ACM automation model to that same infrastructure, using the standard ACME protocol with AWS managed certificate endpoints.

How it works

The feature introduces a new centrally provisioned and managed resource type: the ACME endpoint. Each endpoint is an AWS resource with a unique ACME directory URL and AWS Identity and Access Management (IAM)-based access controls. You create and manage endpoints through the ACM API or AWS Management Console, and point your existing ACME clients at the endpoint URL. Certificates issued through your endpoint are automatically registered with ACM, appearing in your certificate inventory alongside certificates created by the RequestCertificate and ImportCertificate API calls.

The architecture separates into two planes. In the control plane, PKI administrators use ACM APIs to create ACME endpoints, pre-approve the domains an endpoint is allowed to issue for, and generate external account binding (EAB) credentials. In the data plane, ACME clients register with an endpoint using EAB credentials and request certificates for domains the administrator has already validated. This architecture is how we provide customers the ability to scale. Instead of each client proving domain ownership on every request, a principal with appropriate ACM permissions (typically your PKI administrator) validates domains once at the endpoint level, and then application owners don’t need DNS credentials to get a certificate.

Adding to the data plane, EABs control client access to the endpoints. Each EAB is bound to an IAM role that controls what certificate operations the ACME client can perform, and credentials you generate in ACM are distributed to authorized ACME clients. An ACME client authorized for one endpoint can’t use a different endpoint. This creates security boundaries between environments. For example, a client authorized for your development endpoint can’t obtain certificates from your production endpoint.

Figure 1 shows the ACME request flow through ACM. An ACME client authenticates to an ACME endpoint using EAB credentials. The endpoint routes certificate orders to Amazon Trust Services for issuance. Issued certificates are registered in ACM inventory, where Amazon EventBridge and AWS CloudTrail provide expiration alerting and audit logging.

Figure 1: An ACME architecture and workflow

Getting started

Getting started with the new ACME feature in ACM is straightforward. Use the following steps to create your first ACME-generated certificate.

Prerequisites

An AWS account with permissions to create and manage ACM resources
An ACME client installed on your infrastructure (for example, Certbot, cert-manager, acme.sh, or others)
AWS Command Line Interface (AWS CLI) installed on your device (see this blog post for the console equivalent)
Amazon Route 53 hosted zone for your domain, or the ability to create a CNAME record with your DNS provider

Step 1: Create an ACME endpoint

Before you can use ACME clients with ACM, you need to create an ACME endpoint. This endpoint provides the URL that your ACME clients will use to request certificates.

Run the following command from the AWS CLI to create an ACME endpoint:

aws acm create-acme-endpoint \
  --authorization-behavior PRE_APPROVED \
  --certificate-authority '{"PublicCertificateAuthority":{"AllowedKeyAlgorithms":["EC_prime256v1"]}}

Note the endpoint Amazon Resource Name (ARN) from the response.

{"AcmeEndpointArn": "arn:aws:acm:us-east-1:123456789012:acme-endpoint/11111111-2222-3333-4444-555555555555"}

Run the following command to retrieve the endpoint URL, replacing the ARN with your endpoint ARN:

aws acm describe-acme-endpoint \
--acme-endpoint-arn arn:aws:acm:us-east-1:123456789012:acme-endpoint/11111111-2222-3333-4444-555555555555

Save the output of the ACME EndpointUrl:

{
    "AcmeEndpoint": {
        "AcmeEndpointArn": "arn:aws:acm:us-east-1:123456789012:acme-endpoint/11111111-2222-3333-4444-555555555555",
        "EndpointUrl": "https://acm-acme-enroll.<region>.api.aws/6666666-7777-8888-9999-000000000000/directory",
        "Status": "ACTIVE",
        "AuthorizationBehavior": "PRE_APPROVED",
        "Contact": "REQUIRED",
        "CertificateAuthority": {
            "PublicCertificateAuthority": {
                "AllowedKeyAlgorithms": [
                    "EC_prime256v1"
                ]
            }
        },
        "CreatedAt": "2026-07-14T18:23:58.876000-04:00",
        "UpdatedAt": "2026-07-14T18:23:58.876000-04:00"
    }
}

Step 2: Pre-approve a domain

Before ACME clients can request a certificate, the administrator validates the domain using DNS once at the endpoint level. Use DomainScope to control exactly which certificate patterns are allowed:

Enabling only ExactDomain restricts clients to that specific name,
Subdomains enabled allows names like api.example.com,
Wildcards enabled allows *.example.com.

Leave a scope disabled to block that pattern outright, even if an otherwise-valid ACME request asks for it. For a production endpoint, consider enabling only ExactDomain and Subdomains and leaving Wildcards disabled for a stricter posture.

aws acm create-acme-domain-validation \
--acme-endpoint-arn arn:aws:acm:us-east-1:123456789012:acme-endpoint/11111111-2222-3333-4444-555555555555 \
--domain-name example.com \
--prevalidation-options '{"DnsPrevalidation":{"DomainScope":{"ExactDomain":"ENABLED","Subdomains":"ENABLED","Wildcards":"DISABLED"},"HostedZoneId":"Z1234567890ABC"}}'

If your domain is hosted in Route 53, specifying HostedZoneId lets ACM create the required CNAME record automatically. If your domain is hosted elsewhere, omit it and create the provided CNAME record manually with your DNS provider. Validation typically completes within a few seconds after the record is in place.

You will receive the following response back:

{
    "AcmeDomainValidationArn": "arn:aws:acm:us-east-1:123456789012:acme-endpoint/1111111-2222-3333-4444-555555555555/acme-domain-validation/6666666-8888-9999-0000-11111111111"
}

Step 3: Generate EAB credentials

EAB credentials authenticate your ACME clients to your endpoint. Generate a unique set of credentials for each client or environment to maintain security boundaries.

Run the following command to generate your EAB credentials, adjusting your expiration to fit your organization’s risk profile:

aws acm create-acme-external-account-binding \
    --acme-endpoint-arn arn:aws:acm:region:111122223333:acme-endpoint/00000000-0000-0000-0000-000000000000 \
    --role-arn arn:aws:iam::111122223333:role/AcmeIssuanceRole \
    --expiration '{"Value": 7, "Type": "DAYS"}'

Note the response from a successful invocation of the command

{
    "ExternalAccountBinding": {
        "AcmeExternalAccountBindingArn": "arn:aws:acm:region:111122223333:acme-endpoint/00000000-0000-0000-0000-000000000000/acme-external-account-binding/1234567-1234-1234-1234-123456789012",
        "AcmeEndpointArn": "arn:aws:acm:region:111122223333:acme-endpoint/00000000-0000-0000-0000-000000000000",
        "RoleArn": "arn:aws:iam::123456789012:role/service-role/AcmAcmeIssuanceRole-XXXXXXXX",
        "ExpiresAt": "2026-07-21T18:47:50.641000-04:00"
    }
}

Run the following command to retrieve the credentials. You’ll need these values for your ACME client configuration the next step.

aws acm get-acme-external-account-binding-credentials \
    --acme-external-account-binding-arn arn:aws:acm:region:111122223333:acme-endpoint/00000000-0000-0000-0000-000000000000/acme-external-account-binding/22222222-2222-2222-2222-222222222222

Save the KeyId and MacKey for the next step.

{
    "KeyId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "MacKey": "xxxxxxxx-xxxxxxxxxx-xxxxxxxxxxxxxxx"
}

Step 4: Configure your ACME client

With your endpoint URL and EAB credentials ready, you can now configure your preferred ACME client. The following examples show configuration for two popular clients. As a reminder, the server information was retrieved in step 1, part 4 as the EndpointUrl.

acme.sh:

acme.sh --issue --server https://acm-acme-enroll.us-east-1.api.aws/123457-1234-1234-123456789012/directory \
    --eab-kid <KeyId> --eab-hmac-key <MacKey> \
    --email <EMAIL> \
    -d <DOMAIN> \
    --dns --yes-I-know-dns-manual-mode-enough-go-ahead-please

Certbot:

certbot certonly --standalone --non-interactive --agree-tos \
  --email <EMAIL> \
  --server https://acm-acme-enroll.us-east-1.api.aws/1234567-1234-1234-123456789012/directory \
  --eab-kid <KeyId> \
  --eab-hmac-key <MacKey> \
  -d <DOMAIN>

After the initial registration, your ACME client handles renewals.

Enterprise controls

Other ACME alternatives can provide certificates but don’t give the same amount of control and governance for customers that need to scale their certificate environment. The following controls are available to help reduce risk across your organization.

Domain validation

Customers managing large numbers of domains told us they need a way to prevent unauthorized certificate issuance across their domain space. Domain validation gives you this control. For each domain you validate, you enable the certificate patterns it should be allowed to issue, whether it’s ExactDomain, Subdomains, or Wildcards. For example, if you validate internal.example.com and enable only Wildcards, an ACME client can request *.internal.example.com but a request for internal.example.com itself or api.internal.example.com is rejected. This enforcement happens at the endpoint level, before requests reach the ACM certificate authority, and you can validate multiple domains under a single endpoint, each with its own scope.

Centralized certificate visibility

Certificates issued through your ACME endpoints are registered with ACM. You can use the aws acm list-certificates command to see all your issued certificates.

IAM authorization, CloudTrail audit logging and observability

Endpoint management operations are authorized through IAM and logged to CloudTrail. You can use IAM policies to control which principals can create endpoints, generate EAB credentials, and manage domain constraints.

Best practices

For customers implementing ACME certificates for the first time, consider the following best practices for your organizations.

Segment endpoints along organizational or environment boundaries

The endpoint serves as a useful method of isolation for larger organizations. A large enterprise can create one endpoint per organizational boundary (business unit, subsidiary, or environment) instead of a single shared endpoint company-wide. Each endpoint has its own pre-approved domains and its own set of EABs, so a compromised credential in one business unit has no path to certificates in another.

However, weigh this against your operational overhead as well. A reasonable starting point is one endpoint per environment (dev, staging, andprod) within a business unit, expanding to per-business-unit endpoints only where compliance or organizational requirements call for it.

Manage EAB credentials securely

Anyone holding a validKeyIdandMacKeyfor an endpoint can obtain certificates for any domain pre-approved on that endpoint, so these credentials deserve the same handling you’d give an access key.

Avoid hard coding theMacKeywhere possible by using a secret store such as AWS Secrets Manager. Distribute it only to the ACME clients that you authorize to use the endpoint.
Set the expiration of the EAB to an acceptable level. While EAB supports long-lived credentials, not all scenarios require an indefinitely long EAB.
When creating the role for each EAB, adhere to concept of least privilege. Creating a role per EAB, rather than sharing a role across all bindings, can help reduce risk in your AWS environment.
Audit CreateAcmeExternalAccountBinding and GetAcmeExternalAccountBindingCredentials calls in CloudTrail separately. Because retrieving the actual key material is a distinct API call from creating the binding, alerting on retrieval events is a stronger signal of real credential distribution than binding creation alone.

Automate how EABs are associated with clients at runtime

Generate a unique set of EAB credentials for each client or environment rather than sharing one binding across multiple ACME clients. As you begin to scale with multiple endpoints, usesome of the following patterns to reduce operational toil.

Name each EAB and its bound IAM role after the client it belongs to (team, application, environment), so the binding’s purpose is obvious from DescribeAcmeExternalAccountBinding output alone, without cross-referencing a spreadsheet.
Store each client’s KeyId and MacKey under a secrets path scoped to that client (for example, a Secrets Manager path per team and environment), and let the client’s provisioning pipeline retrieve its own credentials.
In Kubernetes, use one ClusterIssuer or namespace-scoped Issuer per EAB rather than one shared issuer across teams. This keeps the client-to-EAB association explicit in cluster config, and lets you revoke one team’s access without touching anyone else’s.
For ephemeral infrastructure (build agents, autoscaled fleets), provision EAB credentials as part of your infrastructure-as-code or continuous integration and deployment (CI/CD) pipeline instead of a one-time manual handoff, so credential lifecycle tracks infrastructure lifecycle.

Monitor your deployment of ACME

ACME’s power is through automation, and organizations should monitor their ACME usage for anomalies.

Alarm on issuance failures, not just successes. At 45-day certificate validity, a silent renewal failure gives you far less runway to react than the months of time you might be used to with longer-lived certificates.
Test renewal automation before you depend on it. Force a manual renewal against a non-production endpoint and confirm your client, monitoring, and on-call runbooks behave as expected, before the CA/Browser Forum’s shortened validity windows turn a failed renewal into a disruptive event for your organization.

Availability and pricing

ACME support in AWS Certificate Manager is available today in all commercial AWS Regions and will be available in AWS GovCloud (US), the China Regions, and the AWS European Sovereign Cloud partitions at a later date. See the ACM pricing page for more information on ACME pricing.

Conclusion

The phased reduction in certificate validity can’t easily be solved without automation. ACME support in ACM gives you that automation through a standard protocol and standard tooling, while keeping the visibility and governance controls your security teams rely on from ACM.

To get started, see the AWS Certificate Manager documentation or follow the getting started guide.

If you have feedback about this post, submit comments in the Comments section below.

Scaling Autonomous Operations with AWS DevOps Agent and ServiceNow

2026-08-06 Jack Hwang

Post Syndicated from Jack Hwang original https://aws.amazon.com/blogs/devops/scaling-autonomous-operations-with-aws-devops-agent-and-servicenow/

This post is co-written with Govind Menon, Head of MCP Product at ServiceNow.

Introduction

Enterprise teams managing applications on AWS often rely on ServiceNow as their IT service management (ITSM) system for incident tracking, change management, and configuration management. When incidents occur, engineers must context-switch between AWS, third party observability tools and ServiceNow, manually correlating data across those sources before updating ServiceNow incident records. This fragmented workflow delays resolution, increases mean time to resolution (MTTR), and introduces the risk of missed signals.

AWS DevOps Agent is a frontier agent that resolves and proactively helps prevent incidents, continuously improving reliability and performance of applications in AWS, and hybrid environments. In this post, we demonstrate how to integrate AWS DevOps Agent with ServiceNow using the Model Context Protocol (MCP) and ServiceNow Action Fabric, enabling autonomous incident investigation and resolution workflows that are governed by ServiceNow and that execute and record authorized actions directly on the application.

By the end of this post, you will be able to:

Configure AWS DevOps Agent as an MCP client connecting to ServiceNow MCP Server created in the MCP Server Console
Authenticate securely via OAuth 2.0 between AWS DevOps Agent and ServiceNow
Enable dynamic discovery of ServiceNow tools exposed through Action Fabric and governed through the ServiceNow MCP Server Console
Automate root cause analysis directly within ServiceNow incidents

Integrating ServiceNow MCP Server with AWS DevOps Agent

The integration between ServiceNow MCP Server and AWS DevOps Agent connects ITSM workflows with automated incident response through the Model Context Protocol (MCP), an open standard for AI agent-to-tool communication.

ServiceNow MCP Server Console lets you create a ServiceNow MCP Server and configure the tools it exposes, capabilities such as incident management, CMDB queries, and change requests as discoverable tools. The console governs what the agent can see and do through tool-level scoping, access control lists, and role masking. It is the access channel for ServiceNow Action Fabric, the application’s governed action layer: ServiceNow does not merely store the agent’s output, it controls and executes the actions the agent is authorized to perform.

AWS DevOps Agent acts as an MCP client that dynamically discovers available ServiceNow tools at runtime. You can create tools based on existing capabilities, such as ServiceNow NowAssist Skills.

When a ServiceNow incident triggers AWS DevOps Agent, the following happens:

Correlates telemetry from Amazon CloudWatch, deployment data, and code changes
Discovers available ServiceNow tools through the ServiceNow MCP Server
Queries ServiceNow for related incidents, change records, and CMDB context
Identifies root cause by correlating AWS telemetry with ServiceNow operational data
Writes findings, root cause analysis, and mitigation plans directly into the ServiceNow incident
Executes governed actions on the application (for example, creating a change request) through the tools the ServiceNow MCP Server Console exposes, where authorized

Security is built into every interaction. Communication uses OAuth 2.0 authentication with scoped

Permissions. The ServiceNow MCP Server Console governs which tools the agent can access and what actions it can perform, with every invocation authenticated, authorized at the tool and skill level, and recorded in an auditable trail that ServiceNow AI Control Tower can observe.

AWS DevOps Agent connecting to ServiceNow via MCP Server with OAuth 2.0

Figure 1: Integration architecture showing AWS DevOps Agent connecting to ServiceNow via MCP Server

Prerequisites

Before you begin, make sure you have access to and understanding of the following:

An AWS account with permissions to create AWS Identity and Access Management (IAM) roles:
Created AWS DevOps Agent Space role and Web app role
Access to AWS DevOps Agent
A ServiceNow instance with admin access
ServiceNow MCP Server configured and accessible ServiceNow MCP Server configured and accessible on an AI Native subscription (Foundation, Advanced, or Prime) or via the standalone MCP add-on

Step 1: Configure the ServiceNow MCP Server and its Tools in the MCP Server Console

As first step, configure the ServiceNow instance to expose capabilities through the MCP Server:

Navigate to the MCP Server Console in the ServiceNow Instance
Create a new MCP Server (or select the MCP server provisioned).

MCP Server Console in a ServiceNow instance

Figure 2: MCP Server Console in ServiceNow Instance

Add Tools for the capabilities the agent needs (for example, incident read and update, CMDB query, change request creation), and scope each with ACLs and role masking so the agent can perform only authorized actions.

Tool selection in the ServiceNow MCP Server Console

Figure 3: Tool selection in ServiceNow MCP Server

Configure inbound authentication for the MCP Server.

Create Inbound Integration dialog with OAuth Client Credentials grant

Figure 4: Create Inbound Integration – OAuth Client Credentials grant

Step 2: Create and configure a DevOps Agent Space

Create an AWS DevOps Agent Space in your AWS account to define the scope of resources the agent will monitor and investigate:

Access the AWS DevOps Agent console
Choose Create Agent Space and provide a name and description, and configure the required IAM roles (automated or manual setup)

Create Agent Space workflow in the AWS DevOps Agent console

Figure 5: Creating an Agent Space in the AWS DevOps Agent console

Agent Space name and IAM role configuration

Figure 6: Agent Space Name and IAM role configuration

Confirm creation of AWS DevOps Agent Space.

Step 3: Register ServiceNow MCP Server in the AWS DevOps Agent console

Navigate to Capability Providers in the AWS DevOps Agent console. Under MCP Server, select Add source, then Register New MCP Server.
Enter your ServiceNow MCP Server endpoint URL:https://<instance>.service-now.com/sncapps/mcp-server/mcp/<server_label>

Figure 7: Entering the ServiceNow MCP Server endpoint URL

Select OAuth Client Credentials as the authorization flow. Enter the Client ID, Client Secret, and Exchange URL (https://<instance>.service-now.com/oauth_token.do) from Step 1.

OAuth Client Credentials form with Client ID, Client Secret, and Exchange URL

Figure 8: OAuth Client Credentials configuration for the ServiceNow MCP Server

Submit the registration. AWS DevOps Agent validates the connection and discovers available tools. Select the tools to add to your Agent Space.

Selecting discovered ServiceNow MCP tools to add to the Agent Space

Figure 9: Selecting ServiceNow MCP tools to add to the Agent Space

Confirm the MCP Server is associated and tools are connected.

Putting It All Together: End-to-End Test

Once the setup is complete, we need to make sure the connection is working.

Navigate to Operator Access in the AWS DevOps Agent Space.
Open a new chat window, and type “Can you show me all the incident in the past week from ServiceNow”
Make sure the Agent calls the ServiceNow tools and shows the right results.

Testing the ServiceNow MCP connection by querying recent incidents

Figure 10: Test the ServiceNow MCP connection from AWS DevOps Agent

You can also configure your environment so that the creation of an incident in ServiceNow automatically triggers the AWS DevOps Agent. To set up this integration, follow the AWS documentation to establish the connection between AWS DevOps Agent and your ServiceNow instance. Then, create a Business Rule in ServiceNow. This enables incident creation to seamlessly trigger the DevOps Agent without manual intervention.

Once this setup is complete, here’s how the workflow comes together: when an incident is created, the DevOps Agent automatically investigates and adds relevant context such as root cause analysis, related changes, and affected resources directly back into the incident record. This means that by the time your Operations or SRE team picks up the incident, they already have the context they need to begin resolution, significantly reducing triage time and accelerating mean time to recovery (MTTR).

ServiceNow console showing investigation kick off

Figure 11: AWS DevOps Agent initiating an automated investigation on the ServiceNow incident

ServiceNow console investigation complete

Figure 12: AWS DevOps Agent mitigation plan posted to the ServiceNow incident

Clean up

To avoid incurring ongoing costs, clean up your resources when you are done using the integration. For details on pricing, visit the AWS DevOps Agent pricing page.

When you are done using the integration, clean up your resources:

Delete your Agent Space from the AWS DevOps Agent console
Remove the ServiceNow MCP Server connection from your settings
Delete the IAM roles created for the Agent Space
(Optional) Disable the MCP Server configuration in your ServiceNow instance

Conclusion

For organizations running workloads on AWS and managing operations through ServiceNow, incident response has long meant toggling between systems and racing to document findings before context fades. The integration between AWS DevOps Agent and ServiceNow through MCP and Action Fabric alleviates that gap. The agent investigates autonomously, correlates telemetry with operational context, and documents root cause and mitigation directly in the incident record, compressing resolution times from hours to minutes.

And because the connection is built on MCP, an open protocol for agent-to-tool communication, what you configure today continues to expand as your ServiceNow workflows evolve. New tools exposed through Action Fabric are discovered and available to the agent immediately. To get started, visit the AWS DevOps Agent product page and ServiceNow MCP Server Console page.

From 2 weeks to 2 minutes: Amazon Cognito launches Provisioned limits for self-service rate limit management

2026-08-05 Kiran Dongara

Post Syndicated from Kiran Dongara original https://aws.amazon.com/blogs/security/from-2-weeks-to-2-minutes-amazon-cognito-launches-provisioned-limits-for-self-service-rate-limit-management/

Imagine preparing for your biggest sales event of the year, and you want to ensure your customer identity management service can handle the elevated traffic for carrying out application activities. For security teams, business leaders, and technologists managing identity infrastructure at scale, this scenario has been all too familiar. Whether you’re a CISO evaluating security controls, a CFO managing cloud costs, traditional support ticket processes for adjusting Amazon Cognito rate limits meant waiting 10–14 days for capacity increases, requiring teams to plan weeks in advance or rush to escalate.

Today, we’re announcing provisioned limits for Amazon Cognito, a capability that transforms how you manage authentication rate limits. This introduces a feature in the AWS Management Console for Amazon Cognito for on-demand capacity adjustments, working alongside the existing account-level maximum limits in AWS Service Quotas. Together, they give you self-service control over your authentication infrastructure so you can scale up for Black Friday (or similar sales events), scale down after tax season, and optimize costs with unprecedented precision. What once took up to 2 weeks now happens in minutes.

In this post, you’ll learn how provisioned limits work, the relationship between account-level maximums and provisioned capacity, the cost implications and optimization strategies, and step-by-step implementation guidance. This capability fundamentally changes how you approach authentication capacity planning.

Manual processes that can’t keep pace

Identity management services open the front door to your application. When users can’t sign in, everything else stops. Amazon Cognito offers extremely flexible limit management: customers can request adjustments as granular as 1 request per second (RPS) for as short as 1 day. As customer demand for faster, self-service adjustments grew, we identified opportunities to address the following challenges:

Support tickets required for each rate limit change
10–14 day approval timelines for standard review and processing
Advance planning needed weeks ahead of anticipated traffic spikes

For businesses with seasonal traffic, like tax preparation services that see 90% of annual authentication volume in March and April, or ecommerce platforms preparing for Black Friday, these factors meant teams had to plan capacity decisions well in advance with limited ability to adjust in the moment.

Provisioned limits and account-level max limits

Starting July 6, 2026, Amazon Cognito introduced provisioned limits in the Amazon Cognito console. At the account level (per AWS Region, per account), you’ll find a Provisioned limits tab next to the User Pools tab. This gives you direct control over your authentication rate limits through two complementary mechanisms:

Provisioned limits (Amazon Cognito console) – Adjust your provisioned capacity up or down on-demand. Changes take effect immediately. You’re billed for the capacity you provision above the default limit, regardless of how much you use.
Account-level max limit (Service Quotas console) – Set your account’s ceiling, the maximum RPS your account is allowed to provision. Raising this ceiling doesn’t incur additional charges. Approximately 90% of requests are automatically approved within minutes. For larger limit increases (depending on the API category and Region), manual approval through AWS Support might still be required.

The following experience shows the end-to-end workflow of adjusting your provisioned limits and requesting a higher account-level max.

Figure 1: Provisioned limit experience workflow

How they work together

Let’s use the UserCreation API as an example. The default limit is 50 RPS, and initially the provisioned limit is also 50 RPS—meaning billed capacity is 0 (no additional charge). The applied account-level max limit is also 50 RPS. So you have three values: default (50), provisioned limit (50), and account-level max (50).

Start by going to the Amazon Cognito console and choosing User pools from the navigation pane.

Figure 2: UserCreation with default values
Choose Edit provisioned limit, to go to the Edit provisioned limit page with an input field for New provisioned limit. However, because the account-level max limit is 50 RPS, you can’t set the provisioned limit above 50. For example, if you want to provision 55 RPS, the console won’t allow it because 55 exceeds the current account max of 50.

Figure 3: Editing the provisioned limit constraint constraint to more than the account-level max
To set a higher limit, choose Request an increase. This takes you to the Service Quotas console, where you can choose Request increase at account level to request a higher account-level max, for example, 55 RPS. Most requests are automatically approved within minutes. At any time, you can check the status of the request using the Request history tab.

Figure 4: Service Quotas page where the account-level max increase is requested and auto-approved
After receiving approval, return to the Amazon Cognito console to edit your provisioned limit up to 55 RPS. Your billed capacity becomes 5 RPS (55 minus the 50 default).

Figure 5: Provisioned after increasing the provisioned limit to 55 RPS, with billed capacity of 5 RPS.

This two-part model gives you precise control over both cost and capacity. Raising the account-level max in Service Quotas doesn’t incur additional charges—it only sets your ceiling. You are billed for what you provision in the Amazon Cognito console above the default, so you benefit from right-sizing your provisioned capacity to match expected demand. Raise your account max ahead of time to prepare for future scaling needs without incurring any cost. When the time comes, increase your provisioned limit to what you need, and scale back down after the event. You stop being charged for the extra capacity as soon as you reduce the provisioned limit. This applies equally to seasonal spikes, planned load tests, or unexpected viral growth—all self-service, all within minutes.

How the provisioned limits experience works

The provisioned limits experience introduces three key concepts that work together:

Default limit – The baseline rate included at no additional cost (for example, 50 RPS for UserCreation).
Provisioned limit – The capacity you actively request and reserve in the Amazon Cognito console. Because this capacity is reserved specifically for your account, it’s the chargeable dimension. You’re billed for any provisioned capacity above the default, regardless of how much you consume. For example, if the default is 50 RPS and you provision 80 RPS, you’re billed for 30 RPS even if your actual usage is only 60 RPS. If your provisioned capacity is 50 RPS (the default), your billed capacity is 0.
Applied account-level max limit – The ceiling managed through Service Quotas. This determines how high you can set your provisioned limit. Raising this ceiling doesn’t incur charges, it only unlocks the ability to provision higher capacity. Importantly, the Provisioned limits page displays each API category with its adjustability status. For example, UserCreation is marked Adjustable (shown earlier in Figure 5) and can be modified. However, UserList is marked Not adjustable, meaning the account-level max limit can’t be adjusted for that category. You can still see its default and provisioned limit on the page, but you can’t modify them. For adjustable categories, you will see the default limit, current provisioned limit, and billed capacity at a glance.

Figure 6: Provisioned limits overview with non-adjustable API categories.

Multi-tenant SaaS considerations

For software as a service (SaaS) providers managing multiple tenants with varying throughput requirements, the UpdateProvisionedLimit API enables programmatic management of provisioned capacity. Teams using dedicated user pools per tenant, for example, can integrate this into their infrastructure-as-code pipelines to adjust provisioned limits per tenant tier. With provisioned limits, SaaS vendors can tier their capacity management per tenant, for example, provisioning higher capacity for enterprise-tier tenants and lower capacity for free-tier tenants, and adjust each tenant’s provisioned capacity independently through the API based on their service tier and demand patterns.

Conclusion

With provisioned limits, whether you’re preparing for peak shopping season, tax filing deadlines, or any other scaling event, you can now adjust provisioned limits to respond to your organization’s needs on demand. The separation between the account-level max (in Service Quotas) and the provisioned limit (in the Amazon Cognito console) gives you full control to plan ahead, respond to demand changes in minutes, and optimize costs on your own terms.

Get started

The provisioned limits experience for Amazon Cognito user pools launched on July 6, 2026, and is available across all AWS Regions where Amazon Cognito is supported. To get started:

Review your current authentication traffic patterns using Amazon CloudWatch metrics to understand your baseline
Set up CloudWatch alarms at 70% and 85% of your current rate limits
Ensure your team has appropriate AWS Identity and Access Management ( IAM) permissions for both Service Quotas and the Provisioned limits tab in the Amazon Cognito console
Raise your account-level max in Service Quotas based on your demand expectations
Use the Amazon Cognito console Provisioned limits tab to adjust capacity up or down as needed

Resources:

Need help? Contact your AWS account team or visit AWS Support for assistance with planning your rate limit strategy.

If you have feedback about this post, submit comments in the Comments section below.

Accelerate CloudFormation development with the IaC MCP Server

2026-08-04 Shuto Yukawa

Post Syndicated from Shuto Yukawa original https://aws.amazon.com/blogs/devops/accelerate-cloudformation-development-with-the-iac-mcp-server/

Organizations adopt Infrastructure as Code (IaC) to manage cloud environments reliably, repeatably, and at scale. As teams grow and infrastructure complexity increases, IaC becomes the backbone of consistent deployments, compliance enforcement, and operational agility. The developer’s experience around IaC, however, remains fragmented — engineers routinely context-switch between documentation portals, linting tools, deployment consoles, and logging systems just to complete a single deploy cycle. This friction compounds across teams: slower iteration means delayed feature releases, longer incident recovery times, and increased operational risk. When a deployment fails, diagnosing the root cause across disconnected interfaces can take longer than writing the template itself — turning a feedback loop that could take hours of manual investigation into a more streamlined process.

The AWS Infrastructure as Code (IaC) MCP Server brings AWS CloudFormation documentation search, template validation, and deployment troubleshooting into your AI assistant, so you can move through a full AWS CloudFormation development cycle without leaving the chat interface. Developing AWS CloudFormation templates often means switching between documentation pages, linters, the deployment console, and AWS CloudTrail Logs. Each context switch adds friction to the inner development loop — the tight cycle of writing, validating, deploying, and fixing infrastructure code. This fragmented workflow increases time-to-deployment, delays feedback, and reduces developer productivity, particularly for teams managing complex, multi-resource stacks at scale.

The AWS Infrastructure as Code (IaC) Model Context Protocol (MCP) Server unifies these capabilities in one place. This post demonstrates how the IaC MCP Server tools work together in a real workflow — from authoring and validation through deployment and runtime troubleshooting — all within a single AI assistant conversation.

In this post, you can move through a complete CloudFormation development cycle using your AI assistant. You generate a template for an Amazon Simple Storage Service (Amazon S3) bucket, an AWS Lambda function, an AWS Identity and Access Management (IAM) execution role, and an Amazon CloudWatch Logs log group. You then validate, deploy, diagnose a deployment failure, and redeploy, all in a single interface.

Solution overview

The walkthrough follows four steps that map to IaC MCP Server tools:

Author: Search CloudFormation documentation and generate a template
Validate: Check syntax with cfn-lint and compliance with cfn-guard
Deploy: Deploy the stack using a CloudFormation service role
Troubleshoot: Diagnose a deployment failure using CloudTrail correlation

Figure 1 shows the four-step workflow. Steps 1, 2, and 4 run inside the IaC MCP Server, while Step 3 uses the AWS CLI directly.

Figure 1. End-to-end CloudFormation workflow with the IaC MCP Server

In the prerequisites, you deploy a CloudFormation service role stack that deliberately omits the iam:PassRole permission. During the walkthrough, you use the AI assistant to generate and deploy an application stack. When CloudFormation tries to assign the Lambda execution role, the deployment fails with AccessDenied. The troubleshoot tool then correlates stack events with CloudTrail to pinpoint the root cause.

For an introduction to each IaC MCP Server tool, see Introducing the AWS Infrastructure as Code MCP Server.

Prerequisites

Before you start the walkthrough, set up your AWS account and AI assistant and deploy the service role stack that the walkthrough depends on.

To follow along, you need:

An AWS account with permissions to create IAM roles and CloudFormation stacks
Kiro or another MCP-compatible AI assistant with the IaC MCP Server configured
AWS Command Line Interface (AWS CLI) configured with valid credentials (see Configuring the AWS CLI if you haven’t set this up yet)

This walkthrough uses the us-east-1 Region. You can use a different Region, but make sure to use the same Region consistently across each step.

Clone the companion repository and deploy the service role stack:

git clone https://github.com/aws-samples/sample-accelerate-cloudformation-with-iac-mcp-server.git

cd sample-accelerate-cloudformation-with-iac-mcp-server

aws cloudformation deploy \
  --template-file iac-mcp-blog-role-stack.yaml \
  --stack-name iac-mcp-blog-role-stack \
  --capabilities CAPABILITY_NAMED_IAM

This role grants CloudFormation permission to create S3 buckets, Lambda functions, and CloudWatch Logs log groups, but deliberately omits iam:PassRole — you’ll diagnose this gap in Step 4.

You use the --capabilities CAPABILITY_NAMED_IAM flag to acknowledge that the stack creates IAM resources with custom names.

We provide this role template for demonstration purposes only and do not intend it for production use.

Note the role ARN from the stack outputs. You must use this ARN in Step 3:

aws cloudformation describe-stacks \
  --stack-name iac-mcp-blog-role-stack \
  --query "Stacks[0].Outputs[?OutputKey=='ServiceRoleArn'].OutputValue" \
  --output text

Walkthrough

The four steps that follow map to IaC MCP Server tools: authoring with documentation search, validating with cfn-lint and cfn-guard, deploying with a CloudFormation service role, and troubleshooting with CloudTrail correlation.

Step 1: Generate a CloudFormation template

Start by asking your AI assistant to search CloudFormation documentation and generate a template. The IaC MCP Server calls the search_cloudformation_documentation tool behind the scenes to retrieve up-to-date resource property references.

Prompt:

Create a CloudFormation template with an S3 bucket, a Lambda function (Python 3.13 runtime, inline hello-world code), an IAM execution role for the function, and a CloudWatch Logs log group. Include common security configurations. Save it as iac-mcp-blog-app-stack.yaml in the current directory.

The AI assistant calls the search_cloudformation_documentation tool to look up resource properties for AWS::S3::Bucket, AWS::Lambda::Function, AWS::IAM::Role, and AWS::Logs::LogGroup. You can see the tool invocations in Kiro’s chat interface. The search results include up-to-date property references and example configurations, which the AI assistant uses to generate a template.

The generated template should include resources similar to the following (your output may vary):

An S3 bucket with versioning, encryption, and public access block
A Lambda function with inline Python code
An IAM role with a least-privilege policy for CloudWatch Logs
A log group with a retention policy

The following snippet shows the key resources. Your AI assistant’s output may differ in naming or structure, but the core configuration should be similar:

Resources:
  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      VersioningConfiguration:
        Status: Enabled

  LambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
      Runtime: python3.13
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Code:
        ZipFile: |
          def handler(event, context):
              return {"statusCode": 200, "body": "Hello from Lambda!"}

Step 2: Validate the template

Before deploying, ask the AI assistant to validate the template. The IaC MCP Server provides two validation tools that wrap open source checkers: cfn-lint for syntax validation and cfn-guard for policy-as-code compliance checks.

Prompt:

Validate iac-mcp-blog-app-stack.yaml for syntax errors and compliance violations.

The AI assistant runs two checks:

Syntax validation (validate_cloudformation_template): Uses cfn-lint to catch structural errors, invalid property names, and schema violations.
Compliance check (check_cloudformation_template_compliance): Uses cfn-guard to evaluate the template against security rules such as S3 bucket encryption, public access block settings, and log group retention.

If either check reports issues, ask the AI assistant to fix them. Continue iterating until both checks pass.

Note that the compliance check might flag violations related to S3 object lock, access logging, replication, and inline IAM policies. For a production workload, you would address each of these issues. In this walkthrough, the AI assistant resolves them to demonstrate the iterative validate-and-fix workflow. Your results might vary depending on the template the AI assistant generated in Step 1.

After the AI assistant resolves the violations, the S3 bucket resource gains access logging and object lock properties. The following snippet shows the typical shape of these additions (see iac-mcp-blog-app-stack-fixed.yaml in the companion repository for the complete hardened template):

  S3Bucket:
    Type: AWS::S3::Bucket
    Properties:
      # ... existing properties ...
      LoggingConfiguration:
        DestinationBucketName: !Ref S3LoggingBucket
        LogFilePrefix: access-logs/
      ObjectLockEnabled: true
      ObjectLockConfiguration:
        ObjectLockEnabled: Enabled
        Rule:
          DefaultRetention:
            Mode: GOVERNANCE
            Days: 30

Your template now passes both cfn-lint and cfn-guard checks. These security improvements improve your template’s security posture but are unrelated to the deployment failure you’ll encounter next. The failure in Step 3 is caused by missing permission on the service role, not by anything in the template itself.

Step 3: Deploy the stack

With validation complete, deploy the template. This deployment will fail — not because of a template error, but because the CloudFormation service role deployed in the prerequisites is missing iam:PassRole. This is the scenario you’ll diagnose in Step 4.

Now deploy the validated template using the service role you created in the prerequisites:

Prompt:

Deploy iac-mcp-blog-app-stack.yaml as a stack named “iac-mcp-blog-app-stack” in us-east-1 using the service role ARN from iac-mcp-blog-role-stack.

The AI assistant runs the AWS CLI deployment command for you. If your AI assistant doesn’t support running shell commands directly, you can deploy manually with the AWS CLI:

Manual CLI deployment

ROLE_ARN=$(aws cloudformation describe-stacks \
  --stack-name iac-mcp-blog-role-stack \
  --query "Stacks[0].Outputs[?OutputKey=='ServiceRoleArn'].OutputValue" \
  --output text)

aws cloudformation deploy \
  --template-file iac-mcp-blog-app-stack.yaml \
  --stack-name iac-mcp-blog-app-stack \
  --role-arn $ROLE_ARN \
  --capabilities CAPABILITY_NAMED_IAM

The deployment fails. The stack event shows an AccessDenied error on the IAM role resource, but doesn’t identify which permission on the CloudFormation service role is missing or why. At this point, we move from static analysis to runtime troubleshooting.

Step 4: Troubleshoot the failure

Ask the AI assistant to diagnose the failure:

Note: CloudTrail events typically take 5–15 minutes to appear. Wait at least 5 minutes after the deployment failure before running the troubleshoot tool for the most complete analysis.

Prompt:

Troubleshoot the failed deployment of iac-mcp-blog-app-stack in us-east-1.

The AI assistant calls troubleshoot_cloudformation_deployment, which:

Retrieves the stack events and identifies the failed resources
Correlates the failure timestamps with CloudTrail API calls
Identifies AccessDenied errors and the missing permissions that caused them

The troubleshoot tool identifies that the CloudFormation service role is missing iam:PassRole — the permission required to assign the Lambda execution role to the function. If your template includes the cfn-guard hardening from Step 2 (access logging, object lock), the tool may also surface additional missing S3 permissions such as s3:PutBucketObjectLockConfiguration for the logging bucket.

Prompt:

Fix iac-mcp-blog-role-stack.yaml to add the missing permissions identified by the troubleshoot tool. Save it as iac-mcp-blog-role-stack-fixed.yaml.

The AI assistant adds the missing permissions to the service role template. Now ask the AI assistant to deploy the fix, delete the failed stack, and redeploy:

Prompt:

Deploy iac-mcp-blog-role-stack-fixed.yaml to update iac-mcp-blog-role-stack, then delete the failed iac-mcp-blog-app-stack and redeploy it with the same service role.

The AI assistant runs the necessary CLI commands: updating the role stack, deleting the failed application stack, and redeploying the application stack. The failed stack is in ROLLBACK_COMPLETE state, a terminal state that CloudFormation cannot update in place, so you must delete it before redeploying.

The stack deployment succeeded.

Cost considerations

For information about costs associated with the resources in this walkthrough, including S3 storage, Lambda invocations, CloudWatch Logs, and CloudFormation operations, see AWS Pricing. Confirm that your account usage falls within any applicable free tier limits. If you enabled S3 access logging or object lock through the validation-and-fix workflow in Step 2, the logging bucket stores a small amount of access log data that falls under S3 standard pricing. See AWS Pricing for current rates and confirm that your account is within the Free Tier limits before you deploy.

Cleaning up

To avoid ongoing charges, delete both stacks.

Option A: Clean up with your AI assistant

Ask your AI assistant to run the cleanup for you. The IaC MCP Server lets the AI assistant inspect stack outputs, empty buckets, and delete both stacks in the correct order:

Clean up the iac-mcp-blog-app-stack and iac-mcp-blog-role-stack stacks in us-east-1. Empty any S3 buckets they created (including access log buckets) before deleting the application stack, then delete the role stack.

Option B: Clean up manually

Delete the application stack first because it was deployed with the service role:

Warning: If your template included access logging, the logging bucket may contain objects. CloudFormation cannot delete a non-empty bucket. Empty it first:

aws s3 rm s3://<logging-bucket-name> --recursive

Then proceed with stack deletion.

aws cloudformation delete-stack --stack-name iac-mcp-blog-app-stack
aws cloudformation wait stack-delete-complete --stack-name iac-mcp-blog-app-stack

aws cloudformation delete-stack --stack-name iac-mcp-blog-role-stack
aws cloudformation wait stack-delete-complete --stack-name iac-mcp-blog-role-stack

If any S3 bucket was created with DeletionPolicy: Retain or still contains objects (for example, server access logs), CloudFormation leaves it in place. Empty and delete those buckets from the S3 console or with aws s3 rb s3://<bucket-name> --force.

Next steps

If you manage CloudFormation infrastructure and find yourself losing time to context-switching between docs, linters, consoles, and logs, here’s how to streamline your workflow starting today:

Set up the IaC MCP Server — Install and configure the IaC MCP Server with an MCP-compatible AI assistant such as Kiro to bring documentation search, validation, and troubleshooting into a single conversational interface.
Run the walkthrough end-to-end — Clone the companion repository and follow this post step by step to experience the full author-validate-deploy-troubleshoot loop in your own AWS account.
Integrate into your team’s workflow — Replace manual context-switching by embedding the IaC MCP Server’s tools into your day-to-day CloudFormation development process, reducing iteration time from hours to minutes.
Extend to AWS CDK — Apply the same conversational workflow to CDK-based infrastructure using the IaC MCP Server’s CDK capabilities described in the introductory blog post.
Contribute and share feedback — Report issues or suggest enhancements on the AWS MCP GitHub repository to help shape future capabilities.

Conclusion

In this walkthrough, you used the IaC MCP Server to move through a complete CloudFormation development cycle without leaving your AI assistant. The documentation search tool retrieved up-to-date resource property references that the AI assistant used to generate a template. The validation tools caught syntax errors and compliance gaps before deployment. When the deployment failed due to missing permissions on the service role (an issue that static analysis cannot detect), you used the troubleshoot tool to correlate stack events with CloudTrail and pinpoint the root cause in seconds.

By combining static validation with runtime diagnostics, you shorten your develop-validate-fix cycle for CloudFormation. Instead of switching between browser tabs, CLI sessions, and the CloudTrail console, you stay in one interface — turning a multi-step troubleshooting session that previously meant switching between consoles, CLI sessions, and CloudTrail into a few prompts in a single conversation.

To get started, explore the companion GitHub repository for the complete sample code. Learn more about the IaC MCP Server in the introductory blog post and the AWS CloudFormation documentation. To set up Kiro, visit kiro.dev.

About the authors

Shuto Yukawa is an Associate Delivery Consultant at AWS Professional Services. He helps customers modernize their applications and adopt cloud-native practices on AWS.

G SS Harsha Vardhan is an Associate Delivery Consultant at AWS Professional Services. He guides customers to migrate and transform their workloads to AWS, driving modernization across people, process, and technology.

Streamline Apache Kafka cluster operations and migrations with Agent Skills for Amazon MSK

2026-08-03 Huyam Hasan

Post Syndicated from Huyam Hasan original https://aws.amazon.com/blogs/big-data/streamline-apache-kafka-cluster-operations-and-migrations-with-agent-skills-for-amazon-msk/

Amazon Managed Streaming for Apache Kafka (Amazon MSK) manages core operational tasks for running Apache Kafka, including cluster provisioning, patching, high availability, and more. But operating Kafka clusters at scale still involves decisions that benefit from deep domain knowledge. For example, where do I start investigating application latency? How do I right-size a cluster to balance performance and cost? How do I analyze my applications, cluster configurations, and other requirements to support a smooth migration from self-managed Kafka to Amazon MSK?

With the new Agent Skills for Amazon MSK, you can access AI-assisted guidance for operations and migration planning directly in your development environment. Two complementary skills, managing-amazon-msk and migrate-to-msk, encode domain expertise based on AWS best practices, structured troubleshooting workflows, and programmatic sizing and compatibility analysis.

In this post, we walk through installing both skills and demonstrate their key capabilities. These include diagnosing a performance issue, sizing a cluster with cost breakdowns, and migration planning from self-managed Kafka to Amazon MSK including discovery, compatibility assessment, and target sizing.

How Agent Skills enhance documentation

Baseline large language models encode knowledge from their training data. That data can go stale as services evolve, and it often lacks the specific, contextual detail a task needs. As a result, a general-purpose assistant can produce answers that sound convincing but are factually wrong (hallucinations). For example, Amazon MSK Provisioned clusters come in two broker types, Standard and Express. Both broker types include their own considerations to achieve your performance, latency, availability, and durability requirements. Because training data mixes the two together, general-purpose assistants routinely conflate them and apply advice to the incorrect broker type.

These skills solve this problem by encoding the correct context for Amazon MSK broker operations, performance management, client configuration, and migrations, aligned with AWS best practices. This helps agents give more accurate, contextual guidance.

Overview of solution

The two Amazon MSK Agent Skills cover the full lifecycle of Amazon MSK cluster ownership:

Skill 1: managing-amazon-msk

Operations expertise for Amazon MSK Provisioned clusters with both Standard and Express broker types:

Workflow	What it does
Performance troubleshooting	Structured decision tree: CPU saturation, batch size analysis, Amazon Elastic Block Store (Amazon EBS) throughput entitlements (Standard), Express brokers entitlements
Consumer lag diagnosis	Determines if lag is broker-side, partition-level (hot keys), or client-side. Provides targeted fixes
Storage management	Amazon EBS expansion, auto scaling, retention planning, tiered storage (Standard only)
Cluster sizing and pricing	Programmatic right-sizing and cost estimate tool comparing all Standard and Express instance types with cost breakdowns
Monitoring and alarms	Set up actionable Amazon CloudWatch alarms with broker-type-aware thresholds that follow best practices for monitoring
Maintenance operations	Rolling restart impact analysis, patching and broker upgrades, version upgrade planning, and transient failure analysis (distinguishing expected maintenance disruptions from real issues).

Skill 2: migrate-to-msk

Migration planning from self-managed Apache Kafka to Amazon MSK in three phases:

Phase	What it does
Discovery	Inventories your source cluster from infrastructure as code (IaC) files, Kafka CLI output, or manual input. Produces a standardized `cluster-config.json`
Assessment	Five-pillar compatibility check (topology, version, configs, auth, quotas) plus target cluster sizing using the AWS-published Amazon MSK Sizing and Pricing workbook
Simulation (Optional)	Deploys temporary Amazon MSK cluster and Amazon EC2 load-generation fleet in your account to test performance under synthetic load before you migrate. Produces an Amazon CloudWatch dashboard with throughput, broker health, latency, and consumer lag metrics.

After assessment, the skill provides guidance on using Amazon MSK Replicator for the actual data migration to your new Amazon MSK cluster.

Prerequisites

To use the tool, you need:

An AI coding assistant that supports Agent Skills, such as Kiro IDE, Amazon Q Developer, or any tool that supports the Model Context Protocol (MCP).
AWS Command Line Interface (AWS CLI) version 2.35.0 or later.
Python 3.12+ and uv installed (Python package runner used by the migrate-to-msk skill).
Agent Toolkit for AWS and AWS MCP server installed.
- For installation instructions, refer to Setting up the AWS MCP Server in the Agent Toolkit for AWS User Guide.
An AWS Identity and Access Management (IAM) role configured with access scoped to each skill’s needs:
- For managing-amazon-msk:
  - Permissions to describe and manage Amazon MSK clusters, retrieve Amazon CloudWatch metrics for performance diagnostics, and create and delete CloudWatch alarms.
- For migrate-to-msk:
  - Optional read-only access (CloudWatch metrics, describe clusters) to gather runtime metrics from an existing AWS estate for a more accurate assessment.
  - The optional Simulation phase requires permissions to create AWS CloudFormation stacks.

Installing the AWS MCP server and skills

Both skills are available in the Agent Toolkit for AWS on the GitHub website.

After initial setup following the steps in the Agent Toolkit instructions, install the Amazon MSK skills with:

aws agent-toolkit add-skill --skill-name managing-amazon-msk
aws agent-toolkit add-skill --skill-name migrate-to-msk

For more information on managing skills, refer to Managing skills with the AWS CLI in the Agent Toolkit for AWS User Guide.

Verify MCP installation by checking the MCP server status in your IDE’s MCP panel.

Verify skill installation with:

aws agent-toolkit list-installed-skills

You should see both skills listed for your detected agents. To confirm they’re active, ask your AI assistant an Amazon MSK question, and it should load the skill to engage with broker-type-aware guidance.

Scenario 1: Diagnosing high latency

During your evaluation of Amazon MSK your team notices elevated produce latency. You ask the AI assistant for help,

“Our Amazon MSK Express broker cluster is experiencing high produce latency that we think is related to our client application. The producer code is in this working directory. Can you help diagnose?”

AI assistant recognizing the latency question and activating the managing-amazon-msk skill

The agent immediately identifies that this question would be well suited for the managing-amazon-msk-skill and activates it. In the same step, the agent opens your producer code to diagnose the real client configuration. The skill ships with reference guides, and the agent selects the two that matter for this specific problem. It then maps your application code directly onto the skill’s diagnostic workflow, landing on a diagnosis:

Skill mapping the producer code to its diagnostic workflow and reaching a latency diagnosis

The skill identifies three compounding anti-patterns in the configuration, specifically linger.ms=0, an undersized batch.size, and compression.type=none. It then explains why they negatively impact Kafka cluster performance: every tiny message becomes its own produce request, saturating broker request-handler threads. Based on these observations, the skill delivers a targeted solution:

Skill’s targeted fix for the linger.ms, batch.size, and compression.type client anti-patterns

The skill uses best practice client-configuration references to provide specific recommendations to improve your application. It then goes on to provide additional context, considerations, and the Amazon CloudWatch metrics to observe to verify that the configurations have improved your end-to-end performance.

Skill listing the Amazon CloudWatch metrics to watch after applying the configuration changes

You can try this yourself by bringing your own producer code and letting the skill diagnose it. If you give it access to the AWS CLI the agent can pull live Amazon CloudWatch metrics from your actual cluster. This lets it correlate broker-side signals with what it sees in your client configuration for a more complete diagnosis.

Scenario 2: Migrating to Amazon MSK Express brokers

The migrate-to-msk skill guides you through a structured migration from self-managed Apache Kafka to Amazon MSK in three phases: discovery, assessment, and optional simulation. When you prompt the skill, it launches the discovery phase.

Phase 1: Discovery — analyze your source cluster

In this scenario, you point the skill at your infrastructure as code (IaC) files describing a self-managed Kafka deployment:

“Here’s our Kafka infrastructure, can you help us plan a migration to Amazon MSK Express brokers?”

migrate-to-msk skill starting the discovery phase against the source Kafka infrastructure

The skill pulls static details: broker topology, versions, security configuration, and topic definitions directly from your IaC files.

Skill extracting broker topology, versions, security, and topics from the IaC files

For runtime values the skill can’t derive from IaC, such as actual peak throughput or consumer-group count, the skill identifies these as flagged gaps. For each gap, the skill provides the specific Kafka CLI commands you can run against your live cluster to capture those values.

Skill listing runtime-value gaps and the Kafka CLI commands to capture them

The skill supports discovery from multiple source types: Terraform, CDK, CloudFormation, Docker Compose, Kubernetes manifests, or manual input in conversation.

Phase 2: Assessment — validate compatibility and size the target

With discovery complete, the assessment phase runs two independent analyses against your current cluster infrastructure.

Compatibility assessment evaluates your source cluster across five pillars:

Pillar	What it checks
Topology	AZ count, broker count, KRaft or ZooKeeper
Kafka version	Source version against Amazon MSK supported set (3.6, 3.8, 3.9)
Configs	Broker and topic configs against Amazon MSK’s editable/enforced/range-restricted sets
Auth	Authentication mechanism compatibility
Quotas	Peak workload against Amazon MSK per-broker ceilings

Each pillar produces one of the following finding types:

Verdict	Meaning
`INFO`	Already aligns with Amazon MSK. No action needed.
`ADVISORY`	Amazon MSK handles this differently, but migration can proceed. Review so the behavior change is expected.
`ACTION_REQUIRED`	Amazon MSK will not accept this in its current form. Remediation recommended.

Target sizing uses your current cluster’s usage metrics to perform right-sizing for Amazon MSK, including instance type, broker count, and projected monthly cost for your workload. This gives you a shareable artifact to use for sizing against different inputs and assumptions.

Next, you ask the skill to run the assessment:

“Assess my cluster for Amazon MSK Express broker compatibility and size the target”:

Skill running the compatibility assessment and target sizing for Amazon MSK Express brokers

The skill runs both analyses against your cluster configuration. It outputs a compatibility report, sizing inputs, and sizing outputs, giving you a complete picture of what needs attention before migration and what your target cluster should look like.

Assessment output with the compatibility report, sizing inputs, and sizing outputs

Once you’ve validated compatibility and provisioned your Amazon MSK Express brokers, Amazon MSK Replicator handles the actual data migration. Amazon MSK Replicator is the native AWS solution for replicating data between Amazon MSK Provisioned clusters. For migrations, it supports replication of data from self-managed Apache Kafka clusters (including on-premises, self-hosted on AWS, or other cloud providers) to Amazon MSK Provisioned clusters.

Phase 3: Simulation (optional) — validate performance before cutover

With assessment complete, you can optionally ask the skill to guide you through setting up a live test environment:

“Can we run a simulation to see how Amazon MSK Express brokers handle our workload before we commit to migrating?”

Skill outlining the temporary Amazon MSK Express and Amazon EC2 simulation before deployment

The skill walks you through deploying temporary Amazon MSK Express brokers and EC2 client fleet in your own AWS account. These are sized from your Phase 2 workbook or numbers you provide, so that you can see real performance on your actual workload rather than relying on estimates. It confirms the target account and permission before deploying any billable resources.

Once the cluster is up, you choose a provided test (end-to-end latency or broker restart under load), and the skill runs it. It then surfaces metrics related to throughput, broker health, latency, and consumer lag on a CloudWatch dashboard. When you’re done, the skill helps you tear the stack down so you stop incurring cost.

Scenario 3: Sizing a cluster with cost breakdowns

You’re planning a new streaming workload and need to determine the right configuration:

“Size an Amazon MSK cluster for 200 MiB/s peak ingress, 600 MiB/s peak egress (3 consumer groups), 1,500 partition replicas, 168 hours retention. Compare Standard and Express.”

Sizing calculator evaluating the workload against Standard and Express instance types

The skill’s programmatic sizing calculator evaluates your workload against every available instance type simultaneously, sizing across four constraints: ingress capacity, egress capacity, partition limits, and storage volume. Each is rounded up to a multiple of 3 Availability Zones (AZs).

When you ask the skill to size a cluster, it uses its sizing script to identify and recommend the least expensive viable option per broker class, and to break down the cluster cost across various sizing dimensions.

Sizing output recommending the least expensive viable broker per class with a cost breakdown

The calculator accounts for factors that manual sizing often misses, such as replication overhead on EBS, network bandwidth, and cross-AZ data transfer costs. The skill flags exactly which constraint is the bottleneck for each instance type, so you understand why a particular broker count was chosen.

Sizing results flagging the bottleneck constraint that sets the broker count per instance type

Security considerations

Both skills recommend Transport Layer Security (TLS) encryption and IAM authentication. Discovery and assessment outputs contain broker addresses and configuration details. Treat them as sensitive and avoid sharing them in public channels without redaction. The migration artifacts do not store passwords or secrets.

Cleaning up

If you ran the optional Simulation phase with the migrate-to-msk skill, it deployed real resources in your AWS account, including an Amazon MSK Express cluster and an EC2 load-generation fleet, that continue to incur charges until you delete them. Ask the skill to tear down the simulation, or delete its CloudFormation stack yourself, to stop incurring cost. Only one simulation can exist per account at a time.

Migration artifacts (migrate-to-msk-skill-artifacts/) are local files that you can delete at your discretion.

Conclusion

Traditionally, Kafka administrators have relied on web-based UIs and dashboards for cluster health management and troubleshooting. With these skills, you can accelerate agent workflows that integrate directly into development environments and DevOps processes. Amazon MSK aims to expand this Agent Skills portfolio with additional tools and capabilities, so customers can build more sophisticated agentic DevOps workflows for their streaming infrastructure.

The Amazon MSK Agent Skills bring structured, broker-type-aware expertise to operating and migrating Amazon MSK clusters. Instead of searching through documentation to determine whether a metric applies to Standard or Express, or manually cross-referencing compatibility matrices for a migration, you get targeted guidance that routes to the correct path based on your cluster’s actual configuration.

Get started by installing both skills from the Agent Toolkit for AWS on the GitHub website into your development environment. Then try a prompt like:

“Size Amazon MSK Express brokers for 100 MiB/s ingress with 3 consumer groups and 72-hour retention”

“My Amazon MSK Express brokers have high produce latency. Help me diagnose”

The skills support you at any stage in the cluster lifecycle.

To learn more, visit the Amazon MSK documentation or open the Amazon MSK console. Have questions or feedback? Open an issue in the Agent Toolkit for AWS repository on the GitHub website.

About the authors

Upgrade Amazon Redshift DC2 clusters to the new Amazon Redshift RG

2026-08-03 Ricardo Serafim

Post Syndicated from Ricardo Serafim original https://aws.amazon.com/blogs/big-data/upgrade-amazon-redshift-dc2-clusters-to-the-new-amazon-redshift-rg/

When you upgrade your Amazon Redshift DC2 (Dense Compute) clusters to RG instances powered by AWS Graviton, you gain access to capabilities that were never available on DC2. These include managed storage, data sharing, zero-ETL integrations, streaming ingestion, and faster query compilation. You also gain availability zone (AZ) features such as cross-AZ cluster relocation for disaster recovery (DR) and concurrency scaling for writes. RG also adds a built-in data lake engine for querying Apache Iceberg and Parquet tables directly on your cluster nodes.

This post covers the new features you gain when upgrading from DC2 to RG, the node mapping guidance for sizing your new cluster, the upgrade methods available, and validation options including Amazon Redshift Test Drive.

Why upgrade from DC2 to RG instances

As data volumes grow, DC2 customers face a choice: add extra compute nodes only to get more storage, or offload data elsewhere. The local SSD capacity on each node is fixed, and there is no managed storage tier to absorb growth. Both RA3 and RG instances solve this with Amazon Redshift Managed Storage, which decouples storage from compute. You can scale data volume independently of node count, paying only for the storage you use with no fixed ceiling per node. This means you no longer need to over-provision compute to accommodate data growth.

RG is the recommended upgrade path over RA3. RG instances run on AWS Graviton processors, delivering higher throughput for data warehouse and data lake workloads at a lower price per vCPU compared to RA3. Because both RA3 and RG share the same managed storage architecture and feature set, RG provides more performance for less cost. For current pricing details, visit Amazon Redshift pricing.

Amazon Redshift RG instances run on AWS Graviton processors. These processors provide more compute cores and lower memory latency compared to the previous-generation hardware behind DC2. This can translate to faster query execution for data warehouse workloads, particularly for large scans where memory throughput is the bottleneck. Exact performance improvements depend on workload characteristics, cluster size, and query complexity. Use Redshift Test Drive to measure the difference for your specific workload.

Data lake access: New with RG

DC2 clusters can query data in Amazon Simple Storage Service (Amazon S3) through Amazon Redshift Spectrum. However, Spectrum adds a per-TB scanning cost on top of your cluster pricing, and does not support enhanced VPC routing on DC2 provisioned clusters (requiring additional configuration for secure S3 access).

RG addresses these constraints with an integrated data lake engine that processes queries directly on your cluster’s dedicated compute nodes:

	DC2 (Spectrum)	RG (Integrated Engine)
Data lake query cost	Extra $5/TB scanned on top of cluster cost	Included in node pricing, no extra charge
Apache Iceberg	Queries via Spectrum	Native queries on cluster compute, no Spectrum needed
Apache Iceberg Statistics	Manual collection	JIT-Analyze auto-collects statistics
VPC routing	Not compatible with enhanced VPC routing	No conflict, runs on the cluster itself

With RG, you can consolidate warehouse and data lake workloads on a single cluster with no extra per-query charges for data lake access.

Features available with RG

Upgrading from DC2 to RG gives you access to the full set of modern Amazon Redshift capabilities. Three of the most impactful for DC2 customers are data sharing, zero-ETL integrations, and managed storage. With data sharing, you can query live data from other Amazon Redshift clusters or accounts without copying or moving data, reducing storage duplication and keeping consumers always up to date. Zero-ETL integrations automatically replicate data from Amazon Aurora, Amazon Relational Database Service (Amazon RDS), and Amazon DynamoDB into Amazon Redshift without building or maintaining ETL pipelines. This reduces operational overhead and data freshness lag. Managed storage scales independently from compute, so you can grow your data without adding nodes and only pay for the storage you use.

Additional capabilities available with RG:

Streaming ingestion – ingest data from Amazon Kinesis Data Streams and Amazon Managed Streaming for Apache Kafka (Amazon MSK) in near real-time, so you can build dashboards and analyze the latest data without batch delays.
Concurrency scaling for writes – automatically add transient capacity during burst write workloads, so ingest operations don’t slow down your analytical queries.
Cross-AZ cluster relocation – relocate your cluster to another Availability Zone with no endpoint changes, supporting disaster recovery without the cost of a standby cluster.
Multi-AZ deployments – run your cluster across multiple Availability Zones as a single database delivering high availability (HA) and automatic failover without a passive standby.
Faster query compilation – queries compile faster on Graviton processors, reducing cold-start latency for new or modified queries.

RG instance details and node mapping

This table shows the available RG instance configurations:

RG Instance	vCPUs	Memory
rg.large	2	16 GiB
rg.xlarge	4	32 GiB
rg.4xlarge	16	128 GiB
rg.12xlarge	48	384 GiB

For current pricing, visit Amazon Redshift pricing for more information.

DC2 to RG node mapping guidance

Use this table to determine the recommended starting configuration when upgrading from DC2:

Current Node Type	Node Ratio	RG Node Type	Guidance
dc2.large (1–3 nodes)	1:1	rg.large	1 rg.large for every 1 dc2.large
dc2.large (4 nodes)	4:3	rg.large	3 rg.large for 4 dc2.large
dc2.large (5–15 nodes)	8:3	rg.xlarge	3 rg.xlarge for every 8 dc2.large
dc2.large (16–32 nodes)	10:1	rg.4xlarge	1 rg.4xlarge for every 10 dc2.large
dc2.8xlarge (2–15 nodes)	2:3	rg.4xlarge	3 rg.4xlarge for every 2 dc2.8xlarge
dc2.8xlarge (16–128 nodes)	2:1	rg.12xlarge	1 rg.12xlarge for every 2 dc2.8xlarge

Extra nodes might be needed depending on workload requirements. Add or remove nodes based on the compute requirements of your required query performance. Validate your specific configuration using Redshift Test Drive before migrating production workloads.

Prerequisites

Before starting the upgrade, confirm the following:

Snapshot availability — a recent snapshot of your DC2 cluster is required for all upgrade methods. If automated snapshots are disabled, create a manual snapshot before starting. Visit Amazon Redshift snapshots for more information.
Network configuration — verify that your virtual private cloud (VPC), subnet groups, and security groups are configured to support the new RG cluster. If you use enhanced VPC routing, confirm your S3 endpoint and route table configuration. Visit Enhanced VPC routing for more information.
Cluster version — your DC2 cluster must be running a supported Amazon Redshift version. Check the release notes for minimum version requirements.

Upgrade methods

Three methods are available for migrating from DC2 to RG instances. The right choice depends on your operational constraints: whether you need write access during migration, whether the target configuration supports elastic resize, and how much downtime your workload can tolerate.

Elastic resize (recommended)

Elastic resize is the fastest and most efficient path. Amazon Redshift creates a snapshot, provisions the RG cluster, and redirects the endpoint automatically. The cluster remains in read-only mode for a few minutes during the operation, and the endpoint doesn’t change, meaning no application-side updates are required. This is the recommended method when the target configuration is supported by elastic resize.

Classic resize

Use classic resize when the target configuration is not available through elastic resize, or when you need data slice rebalancing. Downtime is similar to elastic resize (a few minutes of read-only mode in Stage 1). In Stage 2, data redistributes to its original distribution patterns in the background without blocking queries. The advantage of classic resize is that it rebalances data slices evenly across nodes. This matters when you move to a different node type that might require a different number of slices. Stage 2 can take time on busy clusters, and the duration depends on data volume, cluster utilization, and target cluster size. Queries might run slower until redistribution completes.

Snapshot and restore with cluster identifier swap

This method uses snapshot and restore of the existing DC2 cluster to provision a new RG cluster with a different identifier. After validating the new cluster, you swap the cluster identifiers to redirect application traffic without changing the endpoint. This approach provides these benefits:

Test and validate the RG cluster while the DC2 cluster continues serving production traffic.
Roll back by reversing the identifier swap if issues arise.
No application-side endpoint changes required after the swap.

The trade-off is that data written to the source cluster after the snapshot requires manual synchronization before the cutover. If your migration plan includes a write-freeze window, you can take the final snapshot at the start of that window and avoid synchronization entirely.

This AWS Command Line Interface (AWS CLI) command illustrates restoring a DC2 snapshot to an RG cluster:

aws redshift restore-from-cluster-snapshot \
    --cluster-identifier my-cluster-rg \
    --snapshot-identifier my-dc2-snapshot \
    --node-type rg.4xlarge \
    --number-of-nodes 3 \
    --cluster-subnet-group-name my-subnet-group \
    --vpc-security-group-ids sg-abc123 \
    --cluster-parameter-group-name my-param-group \
    --port 5439 \
    --no-publicly-accessible \
    --enhanced-vpc-routing \
    --iam-roles 'arn:aws:iam::111122223333:role/RedshiftRole'

After restoring, validate your workload on the new cluster. When ready, swap the cluster identifiers:

aws redshift modify-cluster \
    --cluster-identifier my-cluster \
    --new-cluster-identifier my-cluster-dc2-old

aws redshift modify-cluster \
    --cluster-identifier my-cluster-rg \
    --new-cluster-identifier my-cluster

Validating your target configuration

Before migrating production clusters, validate that your target RG configuration meets performance requirements. There are several ways to approach this depending on your needs:

Run your existing QA process on a test cluster. Create an RG cluster from a snapshot, then execute the same test suites and validation scripts you would use for any code or infrastructure change. This approach helps confirm basic compatibility and catch regressions.

Use lower environments first. Migrate your development or staging clusters to RG before production. This gives your team hands-on experience with the new instance type and surfaces any configuration differences in a low-risk setting.

Replay production workloads with Redshift Test Drive. For production-level validation with real traffic patterns, Redshift Test Drive is an open source utility that automates workload replay across multiple target configurations. It extracts queries from your source cluster’s audit logs and replays them against the target, then provides a comparison UI for latency, errors, and deviation.

For a detailed walkthrough, read Find the best Amazon Redshift configuration for your workload using Redshift Test Drive.

Best practices

Before migrating, run Amazon Redshift Advisor on your current cluster to identify optimization opportunities such as unused tables, missing sort keys, or distribution style changes. Drop unnecessary tables to reduce data transfer time, and schedule the migration during off-peak hours for minimal business impact. Removing tables that are no longer used (for example, tables with suffixes like _bkp, _tmp, or _old) also speeds up classic resize. These unused tables would otherwise be rebalanced across nodes during Stage 2, adding time to a process that delivers no value for data no one queries.

During migration, communicate the cutover window to stakeholders. Because the DC2 cluster remains active until the identifier swap, coordinate a brief write-freeze period before the final snapshot to minimize data synchronization effort.

After migration, monitor the cluster for 48–72 hours to identify any performance deviations and adjust node count if needed. Update your runbooks and operational documentation with the new cluster details, node types, and any endpoint changes if you used the snapshot and restore method. Once the migration is considered successful you may delete the DC2 cluster.

Conclusion

Upgrading from Amazon Redshift DC2 to RG instances powered by AWS Graviton gives you a Graviton-based architecture with managed storage and improved query performance. It also gives you access to the full suite of Amazon Redshift features that were never available on DC2: data lake queries, data sharing, zero-ETL, faster query compilation, and cross-AZ relocation. The snapshot restore and cluster identifier swap method provides a safe migration path with built-in rollback. Use Redshift Test Drive to validate your target configuration with real workload data before committing.

To get started, review the RG instance availability and pricing, determine your target configuration using the node mapping guidance, and run Redshift Test Drive against your production workload.

About the authors

Lowering AWS KMS decrypt API costs in EMR Spark jobs

2026-07-30 Navaneedha Krishnan Jagathesan

Post Syndicated from Navaneedha Krishnan Jagathesan original https://aws.amazon.com/blogs/big-data/lowering-aws-kms-decrypt-api-costs-in-emr-spark-jobs/

Modern organizations processing vast amounts of data on Amazon EMR with Apache Spark face a growing cost challenge. As the number of encrypted Amazon Simple Storage Service (Amazon S3) objects grows, AWS Key Management Service (AWS KMS) decrypt API calls multiply rapidly, driving up operational costs. Consider a retail organization processing hundreds of terabytes of customer transaction data daily in S3 encrypted with AWS KMS. Each Spark task accessing an encrypted S3 object triggers an AWS KMS decrypt API call. At scale, these calls accumulate into significant and often unexpected cost increases. This is especially true for workloads that require key auditability and cannot switch to S3 Bucket Keys. S3 Bucket Keys reduce AWS KMS request costs by decreasing the number of calls from S3 to AWS KMS. However, S3 Bucket Keys limit per-object key auditability in AWS CloudTrail, which might not meet the compliance requirements of some organizations.

This post introduces practical techniques to reduce AWS KMS decrypt costs. You can reduce API call volume and lower costs without compromising encryption. It covers three techniques: optimizing file formats (including Apache Iceberg), aggregating data, and using AWS Glue Data Catalog partition indexes.

Optimization techniques

The following sections describe three techniques you can apply independently or together to reduce the number of AWS KMS decrypt API calls.

Use data aggregation

Data aggregation reduces redundant AWS KMS decrypt API calls by consolidating smaller files into larger blocks. When Spark reads many small files from S3, each file triggers its own decrypt call. By combining multiple small files into fewer, larger files, you can reduce the total number of AWS KMS API invocations. This technique is effective for read-heavy workloads that involve numerous small files stored on S3.

You can use AWS CloudTrail to monitor changes in API call frequency and validate the effectiveness of data aggregation in reducing costs.

Step 1: Benchmark baseline performance

Before applying optimizations, establish baseline metrics to quantify improvements.

from pyspark.sql import SparkSession
import time

spark = SparkSession.builder.appName("Baseline Job").getOrCreate()
start_time = time.time()
data = spark.read.format("csv").load("s3://amzn-s3-demo-bucket/data/")
end_time = time.time()
load_time = end_time - start_time
print(f"Load time: {load_time:.2f} seconds")
data_count = data.count()
print(f"AWS KMS calls triggered: {data_count} rows processed")

The following figure shows the number of AWS KMS Decrypt API calls captured in AWS CloudTrail. Use these baseline metrics to compare against optimized results in subsequent steps.

Amazon Athena console displaying CloudTrail log query results with a KMS Decrypt API call events triggered during the baseline Spark job reading unoptimized CSV files from S3

AWS CloudTrail log showing baseline AWS KMS Decrypt API call count

Step 2: Aggregate files using Spark

Consolidating smaller files into fewer, larger files stored in S3 minimizes redundant decrypt API calls.

consolidated_data = data.coalesce(10)
consolidated_data.write.mode("overwrite").parquet("s3://amzn-s3-demo-bucket/optimized-data/")

Step 3: Rerun the job with optimized files

Read the aggregated data created in Step 2 and compare the AWS KMS Decrypt API call count against the baseline metrics from Step 1.

optimized_data = spark.read.parquet("s3://amzn-s3-demo-bucket/optimized-data/")
optimized_data.count()

The following figure shows the AWS CloudTrail logs after reading the aggregated data.

Amazon Athena console displaying CloudTrail log query results with a reduced number of KMS Decrypt API call events after reading aggregated Parquet files

AWS CloudTrail logs in Amazon Athena showing AWS KMS Decrypt API call count after data aggregation

CloudTrail metrics comparison

Track the number of API calls and observe the direct impact of data aggregation on reducing AWS KMS decrypt API calls for the same amount of data.

The following figure compares the AWS KMS Decrypt API call count before and after data aggregation for the same dataset.

Comparison chart showing AWS KMS Decrypt API call count for the same dataset, with a significant reduction after consolidating small CSV files into fewer aggregated Parquet files

AWS KMS Decrypt API call comparison before and after data aggregation

Aggregating small files into fewer large files reduces decrypt calls and shortens load time.

Optimize file formats and compression

Selecting appropriate file formats and applying compression minimizes the amount of data read from S3 and the number of AWS KMS decrypt operations.

Columnar formats (Parquet/ORC)

Columnar file formats like Parquet and ORC let Spark read only the required columns for analysis, which improves performance for analytical queries. For example, you can convert raw CSV data to Parquet to benefit from better I/O efficiency and query optimization.

df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load("s3://emr-kms-demo/data/")

# Set compression for Parquet files
spark.conf.set("spark.sql.parquet.compression.codec", "snappy")

df.write.format("parquet").save("s3://amzn-s3-demo-bucket/parquet-data/")

Iceberg format

Apache Iceberg is a modern table format designed for large-scale analytic datasets. It supports schema evolution, snapshot isolation, and time travel, making it an excellent choice for data lakes on S3. When used with PySpark, Apache Iceberg simplifies data management by automatically optimizing file layouts, handling partitions, and integrating with Spark catalogs.

The following PySpark example uses Iceberg with Amazon EMR and S3:

pyspark \
  --packages org.apache.iceberg:iceberg-spark-runtime-3.4_2.12:1.4.2 \
  --conf spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions \
  --conf spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog \
  --conf spark.sql.catalog.spark_catalog.type=hadoop \
  --conf spark.sql.catalog.spark_catalog.warehouse=s3://amzn-s3-demo-bucket/iceberg-warehouse \
  --conf spark.sql.defaultCatalog=spark_catalog

df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load("s3://amzn-s3-demo-bucket/data/")

spark.conf.set("spark.sql.parquet.compression.codec", "snappy")

# Write data to Iceberg table
df.writeTo("iceberg_from_emr_data").using("iceberg").create()

# Read from Iceberg table
spark.read.table("iceberg_from_emr_data").show()

Compression

Using compression reduces data size and speeds up reads and writes between S3 and Spark. Note: ZSTD is the recommended and default compression codec for Iceberg, offering better compression ratios. For this demonstration, we use Snappy to illustrate the concept.

spark.conf.set("spark.sql.parquet.compression.codec", "snappy")

Compression not only minimizes I/O and network overhead but also accelerates job execution in distributed Spark environments.

CloudTrail comparison on API calls

The following figure shows the reduction in AWS KMS Decrypt API calls when using optimized file formats with compression.

Comparison showing AWS KMS Decrypt API call count for Parquet files with Snappy compression

AWS KMS Decrypt API calls on optimized file formats with compression

The following table illustrates the reduction in AWS KMS decrypt calls when moving from raw, uncompressed CSV data to optimized Parquet files with compression enabled.

Comparison showing AWS KMS Decrypt API call count for Uncompressed CSV vs Parquet files with Snappy compression

AWS KMS Decrypt API call comparison for CSV and compressed Parquet with Snappy

AWS Glue Data Catalog partition index

Partitioning data helps Spark jobs retrieve subsets of relevant data, reducing scan ranges, and decrypt operations. Using AWS Glue Data Catalog partition indexes reduces scanning overhead and the number of AWS KMS API calls.

Without a partition index, when Spark queries a partitioned table, AWS Glue Data Catalog returns all partitions by calling the GetPartitions API. Spark then reads every S3 object across all returned partitions. Because each S3 object is individually encrypted, Spark must call the AWS KMS Decrypt API once per object. More objects mean more decrypt calls and higher costs. With a partition index, AWS Glue performs server-side partition filtering, returning only matching partitions.

Step 1: Baseline query without partition index

Using an Amazon EMR Spark job:

spark.sql("SELECT * FROM default.`kms-demoevents` WHERE year='2000' AND month='04'").count()

Then check CloudTrail for the kms:Decrypt call count.

The following figure shows the AWS KMS Decrypt API call count when running the baseline query without a partition index. Spark scans all partitions, resulting in a higher number of decrypt calls.

Amazon Athena console displaying CloudTrail log query results showing the total number of AWS KMS Decrypt API calls triggered when querying without a partition index

AWS KMS Decrypt API call count without a partition index

Step 2: Add partition index and rerun the baseline query from Step 1

In the AWS Management Console or through the AWS Command Line Interface (AWS CLI), create partition indexes and rerun the same baseline query from Step 1. Create partition indexes on the year and month columns. Then check CloudTrail for the kms:Decrypt call count.

The following figure shows the AWS KMS Decrypt API call count after adding a partition index. With the partition index, AWS Glue filters partitions server-side, resulting in fewer S3 objects read and fewer decrypt calls.

Amazon Athena console displaying CloudTrail log query results showing a reduced number of AWS KMS Decrypt API calls after adding a partition index on year and month columns, compared to the baseline query without partition index

AWS KMS Decrypt API call count with a partition index

Conclusion

Optimizing EMR Spark jobs ensures cost-effective and efficient processing of encrypted data at scale. S3 Bucket Keys is the most effective way to reduce AWS KMS Decrypt API calls. The techniques covered in this post are additional optimizations that you can use together with S3 Bucket Keys for further cost reduction. You can also use them independently when S3 Bucket Keys cannot be used because of per-object auditability requirements in CloudTrail. Start implementing these strategies today to improve your Spark workload efficiency and achieve cost savings.

We welcome your feedback. If you have questions or suggestions about this post, leave a comment below.

About the author

Secure your npm and pip package updates in Amazon Linux

2026-07-29 Norbert Manthey

Post Syndicated from Norbert Manthey original https://aws.amazon.com/blogs/security/secure-your-npm-and-pip-package-updates-in-amazon-linux/

If you use and install packages from npm or PyPI, the first hours after a package is published are the riskiest because scanners can’t analyze packages before publication. Recent supply chain events affecting NodeJS and Python packages have been detected and removed within hours. However, while those packages were available to the general public, it’s possible that they were installed by users, creating the potential for a security incident. As you will see from the data that follows, if users had waited 1 day before accessing those packages, none of the recent supply chain security events would have had an impact.

In this post, I show you a one-line configuration that you can use to eliminate this exposure in your environment: a dependency cooldown for npm and pip. This change tells your package manager to skip versions published in the last 24 hours, giving the security community time to detect and remove unexpected packages before they reach your systems. These settings secure the default setup. There’s another use case of package updates: receiving security fixes to address security risks. This process involves updating packages to a more recent version. I also show you how to override the cooldown configuration so you can install the latest security patches while newly installed package updates are delayed. We recommend that you assess the severity of code defects and apply security fixes if there’s known risk. Handling security fixes based on their severity—and how to specify SLAs for these fixes based on severity—is beyond the scope of this blog post.

Background: Two risks pull in opposite directions

Software delivered by Amazon Linux packages go through review by Amazon package maintainers and pass guardrails before release. Open source software is developed and maintained with similar processes and guardrails. The npm and PyPI registries have open publishing access and don’t enforce reviews. Unexpected packages are potentially added to the registries because of risks like impersonation or stolen credentials. You’re caught between two risks: older software accumulates unpatched vulnerabilities, while new packages potentially contain unexpected vulnerabilities that haven’t been detected yet. The best approach is to stay current without adopting the newest releases immediately, while applying recommended security fixes. The following diagram illustrates the relation between the two types of risks in an abstract way, where the supply chain risk is highest immediately after a package is published, because unexpected updates can potentially bypass guardrails. After a package is published, auditing can review it and identify potential defects over time. If no security fixes are applied, the risk of all the code defects adds up.

Figure 1: Software risk over lifetime. Unpatched vulnerabilities risk increases over time. Very recent software also carries more supply chain risk.

The problem: The first day presents the highest risk

Supply chain events follow a consistent pattern. An unexpected author publishes an unexpected package or package version and waits for automated systems and users to pull it in. Security researchers and automated scanners typically detect and remove these packages within hours, but by then, systems have been exposed to the risk.

Datadog’s 2026 State of DevSecOps report found that 54% of JavaScript applications install at least one dependency within a day of its release. That’s the time window that presents the highest supply chain risk. Recent events show how fast detection happens:

Event	Exposure window
Nx s1ngularity (Aug 2025)	4–5 hours
axios (Mar 2026)	2–3 hours
Bitwarden CLI (Apr 2026)	93 minutes
TanStack (May 2026)	30 minutes
node-ipc (May 2026)	less than 24 hours

The solution: Skip packages published today

A dependency cooldown tells your package manager to skip recently published versions. If a version hasn’t existed on the registry for the configured timespan, for example, 1 day, it won’t be installed, giving the security community time to detect and remove unexpected versions.

A 1-day cooldown blocks each event listed in the preceding table. Notably, several of these events produced valid provenance attestations and passed build verification. These provenance checks alone didn’t stop them. A cooldown works independently of authorization mechanisms, because it blocks by age rather than by trust.

Both npm (v11.10.0+) and pip (v26.1+) support cooldowns . Amazon Linux 2023 ships these packages in NodeJS 24 and Python 3.14 since release 2023.11.20260608.

If you use lockfile-based installations through npm ci or pip install -r requirements.txt with pinned versions, you won’t pull latest package updates. The cooldown doesn’t apply to those installations. The cooldown only affects resolution of new or updated packages. See the Lockfile-based installs and the cooldown section for details.

Prerequisites

To implement the following solution, you first need to have the following prerequisites in place:

Node.js 24 with npm 11.10.0 or later (in nodejs24-24.14.1-1.amzn2023.0.1 or later).
Python 3.14 with pip 26.1 (in python3.14-pip-26.1.1-1.amzn2023.0.1 or later)
pip-audit (tool to scan python packages required for defect-based override scripts). Use python3.14 -m pip install pip-audit to install.

Future versions of Node.js and Python will bring new commands. The following tool commands work for Amazon Linux 2023 with Node.js 24 and Python 3.14. The provided commands target specific package versions. Adjust the commands if you use later releases.

To set up the npm cooldown

Create the global configuration directory, depending on your NodeJS version.
sudo mkdir -p /usr/lib/nodejs24/etc
Add the npm configuration file with the cooldown setting.
sudo npm-24 config set min-release-age 1 --location=global
Check that the cooldown is active by running the next command.
npm-24 config list

You will see before = "<timestamp from 24 hours ago>" in the output, confirming npm converted the 1-day cooldown into a date filter.
For more information, see the npm min-release-age documentation.

To set up the pip cooldown

Create the system-wide pip configuration file with the cooldown setting.
sudo python3.14 -m pip config set --global global.uploaded-prior-to P1D
Verify the configuration (for Python 3.14 and pip 26.1+).
python3.14 -m pip config list

You will see global.uploaded-prior-to='P1D’ in the output.

This configuration is safe to deploy immediately, because older pip versions (25.x) silently ignore the setting.

To install a package’s latest version without cooldown

What if you want to install the latest version of a package, for example to receive security fixes? The following sections describe how to override the flag using the tool command line. To identify which packages need urgent updates, run the appropriate audit command for your package manager.

npm auditor python3.14 -m pip_audit

For npm packages

Install the package with the cooldown override.
npm-24 install <package-name> --min-release-age=0

For pip packages

Install the package with the cooldown override.
python3.14 -m pip install <package-name> --uploaded-prior-to="P0D”

Update packages that need urgent updates

We recommend that you apply security fixes for packages that have known security risks. You don’t need to turn off the cooldown entirely to apply security fixes. Use the audit tools to identify packages with known issues, then override the cooldown for only these packages.

Prerequisites: Ensure you have Python 3 and pip-audit installed (python3.14 -m pip install pip-audit).

Important: These scripts demonstrate the concept. For production use, add error handling, logging, and testing. Review packages before updating them in automated pipelines.

For npm packages

The following script demonstrates the required steps to identify npm packages with a known security fix. The npm audit command prints these packages as JSON. Next, packages in this list are updated with an npm install command, where their cooldown is overridden so that the latest version is picked up.

npm audit --json | python3 -c "
import json, sys, subprocess
data = json.load(sys.stdin)
for pkg in data.get('vulnerabilities', {}):
subprocess.run(['npm-24', 'install', f'{pkg}@latest', '--min-release-age=0'])
"

For pip packages

The following script demonstrates the required steps to identify pip packages with a known security fix. The pip_audit command prints these packages as JSON. Next, all packages in this list are updated with an pip install command that overrides the cooldown so that the latest version can be picked up.

python3.14 -m pip_audit --format=json | python3 -c "
import json, sys, subprocess
data = json.load(sys.stdin)
for vuln in data:
    pkg = vuln['name']
    fix = vuln.get('fix_versions', ['latest'])[0]
    subprocess.run(['python3.14', '-m', 'pip', 'install', f'{pkg}=={fix}', '--uploaded-prior-to="P0D"'])
"

Lockfile-based installs and the cooldown

If you use npm ci or pip install -r requirements.txt with pinned versions, the cooldown doesn’t apply. These commands install what the lockfile specifies, regardless of package age. The cooldown only affects resolution of new or updated packages.

Industry adoption: Cooldowns are now used across PyPI and NodeJS

Major package managers and enterprises have started to adopt dependency cooldowns. As of May 2026, several popular package management tools now include cooldown features: pnpm (a fast Node.js package manager), Renovate (an automated dependency update tool), and StepSecurity (a supply chain security platform).

pnpm 11 ships with minimumReleaseAge enabled by default. It’s one of the first major package manager to make cooldowns opt-out rather than opt-in.
Renovate’s config best-practices preset has included a 3-day npm cooldown since 2025 and is widely adopted across enterprises.
StepSecurity Secure Registry uses a configurable cooldown period for enterprise customers. StepSecurity recommends a 10 day delay as default.

How AWS is helping protect the open source supply chain

AWS scans upstream package registries to catch unexpected packages before they reach customers.

Amazon Inspector, a security management service that continuously scans workloads for software vulnerabilities and network exposure, uses AI-assisted detection rules to scan upstream package registries. In 2025, Amazon Inspector researchers identified over 150,000 unexpected npm packages linked to a token farming campaign.
AWS Security published detailed response guidance for the Shai-Hulud worm and the axios event.
AWS contributes to the Open Source Security Foundation $12.5M grant for open source security, funding proactive scanning infrastructure that benefits the entire ecosystem.

Unexpected packages are typically caught within hours of publication. A 1-day cooldown ensures you don’t install them during that detection window.

Recommendations

To secure your Amazon Linux 2023 configuration:

Set a 1-day cooldown for npm and pip as shown in the preceding sections. External registries don’t have human review, so give the defenders time to catch problems.
Override when needed for urgent security patches using the per-command flags.
Run npm audit or pip_audit regularly to identify packages that need immediate attention.

Set up the cooldown with one line of configuration, and the protection is immediate.

Conclusion

By implementing the solutions presented in the post, you secure your npm and PyPI environment from most instances of unexpected code. The update delay of 1 day protects your environment, while still allowing to apply the latest security fixes. To learn about how to protect your environment further, see the following resources:

If you have feedback about this post, submit comments in the Comments section below.

Add security context to operational investigations with AWS DevOps Agent and Wiz

2026-07-29 Yuriy Prykhodko

Post Syndicated from Yuriy Prykhodko original https://aws.amazon.com/blogs/devops/add-security-context-to-operational-investigations-with-aws-devops-agent-and-wiz/

This post was co-authored by Ayelet Harcz (Product Manager), Hen Perez (CTO Architect), and Shani Gafni (Product Manager) at Wiz.

When an on-call engineer receives an alert at 2 AM, a CPU spike, a latency anomaly, or an unexpected API error, the first question is whether this is an operational issue or a security incident. A CPU spike could be a scaling problem or a cryptominer. A latency anomaly could be a bad deployment or data exfiltration. Without security context in the investigation loop, engineers lack the information to distinguish between the two, delaying resolution and increasing risk.

AWS DevOps Agent is a frontier agent that autonomously investigates incidents and identifies operational improvements across AWS, multicloud, and on-premises environments. It reduces mean time to resolution (MTTR) by performing the triage and investigation work that would otherwise take an on-call engineer hours of manual effort. With the Wiz integration, AWS DevOps Agent queries Wiz’s security graph during investigations through the Model Context Protocol (MCP), surfacing vulnerability data, security findings, and exposure analysis alongside operational telemetry so engineers can quickly determine whether an alert is a performance issue or a security incident.

In this post, we walk through how the integration works, demonstrate a real-world incident investigation where AWS DevOps Agent uses Wiz MCP to surface a critical vulnerability behind an API latency spike, and show how to configure the integration in your environment. If you already use Wiz to secure your AWS environment, this integration puts your existing security data to work during incident investigations.

AWS DevOps Agent

AWS DevOps Agent investigates incidents and identifies operational improvements as an experienced DevOps engineer would: by learning your resources and their relationships, working with your observability tools, runbooks, code repositories, and CI/CD pipelines, and correlating telemetry, code, and deployment data across all of them. For a deeper look at how it works, see How AWS DevOps Agent uses multi-agent reasoning to find root causes.

AWS DevOps Agent is extensible through MCP, which allows the agent to call external tools during its investigation without requiring custom development. This is the mechanism that makes the Wiz integration possible. When the agent identifies a resource under investigation, it queries Wiz MCP for security findings associated with that resource and incorporates the results into its analysis and recommendations.

Wiz MCP

Wiz is designed to secure cloud and AI applications through a unified, graph-powered platform. The Wiz Security Graph connects infrastructure, identities, data, AI components, and runtime activity into a single contextual view. This approach identifies toxic combinations across layers – where exposures, permissions, data access, AI vulnerabilities, and runtime behaviors intersect in ways attackers can realistically exploit.

The Wiz MCP Server acts as a standardized gateway that allows AWS DevOps Agent to query this security graph during investigations. Wiz knows whether your Amazon Elastic Compute Cloud (Amazon EC2) instance has an exploitable Common Vulnerabilities and Exposures (CVE), whether it is publicly exposed, and whether endpoint protection is in place. AWS DevOps Agent, looking at the same instance, knows that CPU spiked, and a deployment happened 20 minutes ago. Separately, each tool tells a partial story. Together, they give the engineer the complete picture needed to act.

Better together: how combined context changes triage

The value of this integration is easiest to understand through three scenarios. Each starts with the same operational signal: a CPU spike on an EC2 instance.

Figure 1 – AWS DevOps Agent sees operational telemetry, Wiz sees security posture. The combination changes the triage decision.

Figure 1 – AWS DevOps Agent sees operational telemetry, Wiz sees security posture. The combination changes the triage decision.

Scenario A: No security findings. A CPU spike fires on an instance. AWS DevOps Agent queries Wiz and confirms the instance is fully monitored, has no known vulnerabilities, and shows zero active threat detections. This is an operational issue. The engineer scales, investigates the deployment, tests, and moves on.

Scenario B: Security issue detected. The same CPU spike fires, the same Amazon CloudWatch alarm triggers, and the same engineer wakes up. But when AWS DevOps Agent queries Wiz, it finds a validated remote code execution vulnerability on that instance, confirmed exploitable, with the resource exposed to the internet. The operational symptoms are identical to Scenario A. The correct response is the opposite: isolate immediately, engage your security team, treat this as a potential compromise.

Scenario C: Wiz coverage gap. The resource isn’t in Wiz at all. AWS DevOps Agent includes this as a finding in the investigation report, noting that no security context was available for the resource. Your team can then address the coverage gap by onboarding the resource into Wiz.

Without the Wiz integration, all three scenarios look the same in your dashboard. With it, AWS DevOps Agent routes each to the correct response path before a human needs to context-switch between tools.

How the integration works: the MCP bridge

The integration uses MCP, the same protocol AWS DevOps Agent uses for many of its external tool connections. When the agent identifies affected resources during an investigation, it calls Wiz’s remote MCP server as part of its evidence collection – no separate step, no manual trigger. The security query happens alongside the operational investigation, not after it. During the MCP call, AWS DevOps Agent sends resource identifiers to Wiz’s MCP endpoint and receives security findings in response. No operational telemetry or broader investigation context is shared with Wiz.

Figure 2 – The investigation flow: operational alert triggers AWS DevOps Agent, which queries Wiz via MCP before reaching a triage decision.

During the MCP call, AWS DevOps Agent queries Wiz tools to build a complete risk picture of the affected resource, here are a few examples:

Wiz MCP Tool	What it tells the agent
list_cloud_resources	Whether Wiz monitors this resource at all (coverage check)
list_findings	All finding types in one call: vulnerabilities, misconfigurations, secrets, data, and host config
list_vulnerability_findings	Deep CVE detail – severity, fix version, and exploitability (CISA KEV / known exploit)
list_issues	Prioritized risk issues, including toxic combinations (internet-facing + no Endpoint Detection and Response (EDR) + exploitable CVE)
list_threats / list_malware_findings	Active threats and malware: cryptomining, data exfiltration, backdoors
list_detections	Recent threat detection signals and anomalous activity
get_green_agent_analysis	AI-generated remediation steps for the issues found

The agent runs these queries together through a single security-auditing skill that loads automatically when it connects to Wiz’s MCP server with the DevOps toolset, so the full security picture comes back in seconds. If the Wiz MCP server is unreachable, times out mid-query, or returns an authentication error, the agent continues its investigation with the operational data it has and flags the missing security context in the investigation findings (Scenario C). You can review exactly which MCP tools were called and what data was returned in the AWS DevOps Agent investigation log for full auditability.

Based on what comes back, the agent classifies the situation: no security findings (operational issue, proceed normally), compromised or at-risk (active threats, exploitable vulnerabilities, or toxic combinations – apply relevant security runbooks to isolate the resource or escalate to security, with Wiz Green Agent remediation steps attached), or unmonitored by Wiz (flag and close the coverage gap). The classification feeds directly into the investigation findings your team receives.

The following demonstration shows AWS DevOps Agent investigating a reported CPU spike. The agent queries Wiz MCP and identifies a critical, internet-exposed Remote Code Execution (RCE) under active exploitation – turning an ambiguous alert into a confirmed security incident.

Video 1 – AWS DevOps Agent investigates a CPU spike and uses Wiz MCP security context to identify a critical RCE exploited through a public endpoint

Getting started

Prerequisites

To use AWS DevOps Agent with Wiz MCP, you need:

An active AWS DevOps Agent configuration with at least one Agent Space
A Wiz tenant with a remote MCP server endpoint (Streamable HTTP transport)
Authentication credentials for the Wiz MCP server. AWS DevOps Agent supports multiple MCP auth methods; for Wiz, use a Wiz service account (Client ID and Secret) or OAuth. Choose the method that matches your Wiz MCP server configuration. For setup details, see Connect remote Wiz MCP server in the Wiz documentation (requires Wiz login)

Enabling the integration

Step 1: Register the Wiz MCP server at account level

Sign in to the AWS Management Console and navigate to the AWS DevOps Agent console.
Go to the Capability Providers page from the side navigation.
Find MCP Server in the Available providers section and choose Register.
Enter the Wiz MCP server details:
- Name: e.g., “Wiz Security”
- Endpoint URL: https://mcp.app.wiz.io/?toolset=devops
- Description: e.g., “Wiz security context for incident triage”
Choose Next.
Select the authentication method that matches your Wiz MCP server configuration.
Review your configuration and choose Submit. AWS DevOps Agent validates the connection to the Wiz MCP server. Upon successful validation, the server is registered at the account level.

Step 2: Allowlist Wiz tools in your Agent Space

In the AWS DevOps Agent console, select your Agent Space.
Go to the Capabilities tab.
In the MCP Servers section, choose Add.
Select the registered Wiz MCP server.
Select all the Wiz MCP tools.
Choose Add.

Step 3: Choose how the Wiz security audit runs

Pick one of three options:

Use the Wiz skill tool (recommended). With the Wiz MCP tools allowlisted, AWS DevOps Agent automatically runs the latest devops_resource_auditing_skill workflow from Wiz during investigations. You always get the most up-to-date version, maintained by Wiz.
Import the ready-made skill. Import the wiz-security-context skill from the AWS DevOps Agent skills repo directly into your Agent Space. It is a lightweight skill that calls the Wiz workflow for you, so you get a one-step setup that stays current with Wiz.
Create your own custom skill. Use AWS DevOps Agent’s Create skill with Chat to build a custom skill based on the devops_resource_auditing_skill workflow and tailor it to your environment. This lets you review and tailor the workflow to your environment.

For detailed MCP configuration guidance, refer to the AWS DevOps Agent documentation on connecting remote MCP servers.

The power of co-build: extending context through MCP

This integration started from a recurring customer question: how do I know if what I’m seeing is an operational problem or an active attack? We worked with Wiz to close this gap. AWS DevOps Agent provides operational investigation and reasoning; Wiz provides cloud security intelligence. MCP provided the integration path without either side needing to reimplement what the other already does well.

Because AWS DevOps Agent supports connecting remote MCP servers as a first-class extension mechanism, co-building new integrations with AWS Partners follows a repeatable pattern. Each integration adds a new dimension of context to the agent’s reasoning, and you benefit without writing custom code or middleware on your side. For example, connecting a change management MCP server would let the agent correlate deployment approvals with incident timing, adding change context alongside security context.

For you, this means the richer the toolset you run in your environment, the more context the agent brings to each investigation. Your existing investments get amplified rather than duplicated, and you benefit each time you connect a new partner MCP server to your Agent Space.

Conclusion

Operational incidents and security incidents often start with the same symptoms. The difference between the right response to each is context that lives in a different tool than the one that fired the alert. The AWS DevOps Agent and Wiz integration brings that context into the investigation loop automatically through MCP.

To get started, visit the AWS DevOps Agent console and follow the getting started guide. To learn more about Wiz’s MCP server, see Introducing the MCP Server for Wiz.

Wiz is an AWS Partner and AWS Marketplace Seller providing cloud security across the full development lifecycle. If you’re not already using Wiz, you can get started through the AWS Marketplace.

About the Authors

Automate Spark Scala migration to 4.x with AWS Spark Upgrade Agent

2026-07-28 Bezuayehu Wate

Post Syndicated from Bezuayehu Wate original https://aws.amazon.com/blogs/big-data/automate-spark-scala-migration-to-4-x-with-aws-spark-upgrade-agent/

If you’re a data worker responsible for managing Apache Spark 3.x workloads on Amazon EMR before, you’ve likely faced the challenge of migrating hundreds of jobs to Spark 4.0 without disrupting production pipelines. In this post, you will learn how to automate Spark 3.x to 4.0 migration using the AWS Spark Upgrade Agent covering API deprecations, behavioral changes, build configuration updates, and job validation. What once took months of manual effort can be completed in hours.

This is part 3 of a three-part series on how the AWS Spark Upgrade Agent can automate and simplify Spark upgrades.

Part 1 introduces the agent’s architecture and capabilities. Part 2 walks through a complete PySpark migration from Spark 3.5 to Spark 4.0 on Amazon EMR Serverless. This post walks through Scala migration from Spark 3.3 (Scala 2.12) to Spark 4.0 (Scala 2.13).

Apache Spark 4.0 on Amazon EMR 8.x delivers improvements like native merge_into() support, enhanced Adaptive Query Execution, improved Python UDF performance through Arrow-based serialization, and major Structured Streaming enhancements. For teams on Spark 2.4 or 3.x, the complexity lies in managing API deprecations, behavioral changes, build configuration updates, and re-validating hundreds of jobs while maintaining production pipelines.

Prerequisites

This post assumes you’ve completed the one-time AWS CloudFormation setup and proxy configuration detailed in the introduction post.

What is the Spark Upgrade Agent?

The AWS Spark Upgrade Agent is a fully managed remote server that automates Spark migration using a Model Context Protocol (MCP) interface for code analysis and transformation. For details, see the introduction post.

Architecture: How it works

The architecture follows a least-privilege security model:

Scoped IAM roles — AWS IAM roles are scoped to only MCP server calls, Amazon Simple Storage Service (Amazon S3) staging bucket access, and Amazon EMR job submission.
Local source code — Your source code stays local, with only minimal diagnostic information transmitted.
Encryption in transit — All data is encrypted in transit.
Audit trail — AWS CloudTrail records every tool invocation for full auditability.

Example use case: Enterprise-scale Spark migration

To illustrate the capabilities of the agent at scale, consider a real-world migration scenario from a large company. This company runs a data processing platform with thousands of Spark jobs across Scala, PySpark, and Spark SQL workloads, with a code base spanning Spark 3.3 and 3.5.

The data engineering team faces a migration across three workload types with different complexity profiles:

Spark SQL applications: The most portable, but still requiring validation of behavioral changes in the query optimizer and join strategies introduced in Spark 4.0.
PySpark workloads: Requiring updates to UDF serialization patterns, Arrow-based optimizations, and DataFrame API changes.
Scala applications: The most complex, involving build system updates (Maven and SBT), API deprecations, and recompilation against new Spark 4.0 JARs.

To tackle this, the team uses the Spark Upgrade Agent to migrate all three workload types to Spark 4.0 on Amazon EMR 8.x. For each workload, the agent is invoked directly from Kiro or VS Code with Cline, applying targeted transformations and immediately validating results against a live Amazon EMR 8.x Serverless application running Spark 4.0.

Migrations that would traditionally require months of manual engineering effort complete in a fraction of the time. The agent follows an iterative refinement loop: it performs local validation, then submits the job to a remote Amazon EMR cluster. This loop catches and resolves runtime failures automatically, reducing the need for manual debugging cycles. Build configuration files (pom.xml and build.sbt) are updated automatically by the agent, eliminating a common source of migration errors.

Solution walkthrough

The following sections walk through the complete migration workflow, from initial setup through advanced code transformations and validation.

1. Setup

This section lists what you need before starting. Some items, such as Amazon EMR Serverless applications, can be created during the walkthrough using the agent if they don’t already exist.

Must have before starting:

AWS Command Line Interface (AWS CLI) configuration: Your AWS CLI must be configured with a profile that has the necessary permissions. See Configuring the AWS CLI for details.
IAM role with Amazon EMR permissions: An AWS CloudFormation template is provided in the setup guide to provision the required IAM role. The role is scoped to the permissions needed for the upgrade process: calling the MCP server, reading and writing to the Amazon S3 staging bucket, and submitting Amazon EMR jobs.
Amazon S3 staging bucket for artifacts: Used to store code artifacts and Amazon EMR job outputs during the validation phase.
Integrated development environment (IDE) installation: Kiro or VS Code with MCP support (Cline extension). Either IDE can interact with the Spark Upgrade Agent through natural language prompts. Consult the setup guide for Kiro and Cline’s documentation to use the MCP server with Cline.
One-click MCP server installation: The dataprocessing-mcp server is installed and configured as described in the setup guide.
Amazon EMR Serverless applications: An Amazon EMR Serverless application is required for the validation workflow:
- Target application (Spark 4.0): An Amazon EMR Serverless application configured with release label emr-spark-8.0.0, used to validate migrated jobs against Spark 4.0 on Amazon EMR 8.0.

1.1 Infrastructure setup (AWS CloudFormation)

Two AWS CloudFormation stacks create the required resources: an AWS IAM role, an Amazon S3 staging bucket, an Amazon EMR Serverless application (Spark 4.0), and its execution role.

Stack 1: AWS IAM role and Amazon S3 staging bucket

The spark-upgrade-mcp-setup template creates the AWS IAM role and Amazon S3 staging bucket required by the upgrade agent. Choose the Launch Stack button for your Region. For additional Regions, see the full Region list.

Region	Launch
US East (N. Virginia)	Launch Stack
US East (Ohio)	Launch Stack
US West (Oregon)	Launch Stack
Europe (Ireland)	Launch Stack

After deployment, open the AWS CloudFormation Outputs tab, copy the ExportCommand value, and run it in your terminal. This sets SMUS_MCP_REGION, IAM_ROLE, and STAGING_BUCKET_PATH automatically.

The following figure shows the Outputs tab with the ExportCommand value.

AWS CloudFormation console Outputs tab with the ExportCommand value ready to copy

Outputs tab of the AWS CloudFormation stack showing the ExportCommand value

# Sets SMUS_MCP_REGION, IAM_ROLE, and STAGING_BUCKET_PATH
export SMUS_MCP_REGION=<YOUR-REGION> && export IAM_ROLE=arn:aws:iam::<YOUR-ACCOUNT-ID>:role/spark-upgrade-role-* && export STAGING_BUCKET_PATH=<amzn-s3-demo-bucket>

Then configure the AWS CLI profile:

aws configure set profile.spark-upgrade-profile.role_arn ${IAM_ROLE}
aws configure set profile.spark-upgrade-profile.source_profile default
aws configure set profile.spark-upgrade-profile.region ${SMUS_MCP_REGION}

Stack 2: Amazon EMR Serverless target application and execution role

The emr-serverless-target-setup template creates an Amazon EMR Serverless application configured with Spark 4.0 (release label emr-spark-8.0.0) and a shared execution role used for job submission during the validation phase. Deploy it as follows:

git clone https://github.com/aws-samples/sample-amazon-emr-spark4-examples
cd sample-amazon-emr-spark4-examples/scala3/demo_1_spark_change_focus

The Scala sample lives at sample-amazon-emr-spark4-examples/scala3/demo_1_spark_change_focus. The CloudFormation template lives at resources/cloudformation/.

Deploy the CloudFormation template to create the target Amazon EMR Serverless application and a shared execution role:

aws cloudformation deploy \
  --template-file resources/cloudformation/emr-serverless-target-setup.yaml \
  --stack-name spark-emr-serverless-upgrade \
  --region ${SMUS_MCP_REGION} \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
  StagingBucketName=${STAGING_BUCKET_PATH} \
  TargetReleaseLabel=emr-spark-8.0.0 \
  TargetApplicationName=spark-upgrade-target

This creates an Amazon EMR Serverless target application (Spark 4.0) for upgrade validation, with a shared execution role. The application auto-stops after 15 minutes of idle time, so there is no cost when not in use. To upgrade between different Spark versions, override the SourceReleaseLabel and TargetReleaseLabel parameters with the Amazon EMR release labels that you want.

After the stack completes, note the outputs:

aws cloudformation describe-stacks \
  --stack-name spark-emr-serverless-upgrade \
  --region ${SMUS_MCP_REGION} \
  --query "Stacks[0].Outputs" --output table

This gives you the TargetApplicationId and ExecutionRoleArn needed for the upgrade prompt. Make a note of them.

2. Upgrade

This section covers a complete end-to-end upgrade using a representative ecommerce pipeline, a Scala application that processes order events, applies transformations, and writes results using merge-style upsert patterns. The same workflow applies to Scala and Spark SQL workloads covered in subsequent sections.

Step 1: Clone the sample project

Start by cloning the sample project from the AWS samples repository:

git clone https://github.com/aws-samples/sample-amazon-emr-spark4-examples
cd sample-amazon-emr-spark4-examples/scala3/demo_1_spark_change_focus

The repository includes representative PySpark, Scala, and Spark SQL applications designed to demonstrate common Spark 3.x patterns and their Spark 4.0 equivalents.

Step 2: Open in your IDE and connect to the MCP server

Open the project in Kiro or VS Code with the Cline extension. Verify that the dataprocessing-mcp server is active and connected. You can see it listed as an available MCP server in your IDE’s MCP panel. If you haven’t completed the one-time setup, follow the setup guide before proceeding.

Step 3: Start the upgrade with a natural language prompt

Once connected, initiate the upgrade by entering the following request in the agent interface:

Use the dataprocessing-mcp server to upgrade my local project at <path-to-your-project>.
Upgrade my Spark application from Amazon EMR Serverless version 6.9.0 to Amazon EMR Serverless version 8.0.0.
Use Amazon EMR Serverless app-id <your-app-id> for validation.
Store artifacts at s3://amzn-s3-demo-bucket/spark4-upgrade/

The agent responds by invoking generate_spark_upgrade_plan, analyzing your project structure, identifying incompatible patterns, and presenting a prioritized upgrade plan before proceeding with any code changes.

After you confirm the plan, the agent proceeds autonomously through the remaining phases:

Build configuration update — update_build_configuration rewrites pom.xml, build.sbt, or requirements.txt to target Spark 4.0 dependencies.
Environment validation — Java and Python environments are checked and updated as needed.
Code transformation — fix_upgrade_failure applies targeted fixes for each identified incompatibility, iterating until the project compiles cleanly.
Remote validation — run_validation_job submits the upgraded application to your Amazon EMR Serverless target application and monitors execution through check_job_status.
Data quality check (optional) — get_data_quality_summary compares output between the Spark 3.5 baseline and the Spark 4.0 run, confirming correctness before sign-off.

With the sample Scala ecommerce pipeline cloned from the sample-amazon-emr-spark4-examples repository and your IDE connected to the MCP server, you submitted a natural language prompt. This triggered the agent to analyze the project structure and generate a prioritized five-step upgrade plan, all before making any code changes.

Now that the upgrade plan is in place, the following sections walk through the specific code transformations the agent applies for a Scala workload.

Scala workload migration

This section covers the complete migration of a representative Scala Spark application from Amazon EMR Serverless 6.9.0 (Spark 3.3, Scala 2.12) to Amazon EMR Serverless 8.0.0 (Spark 4.0, Scala 2.13), using the demo_1_spark_change_focus sample from the AWSSpark4AutoUpgradeDemo repository.

Sample project: Ecommerce product change focus pipeline

The sample application processes product catalog change events from Amazon S3, applies enrichment transformations, and writes aggregated results back to Amazon S3. It represents a common pattern in ecommerce data platforms: incremental processing of catalog updates with downstream aggregation. In this example, we use VS Code with Cline, but you can also use Kiro or any other MCP-enabled IDE.

Project structure:

demo_1_spark_change_focus/
├── build.sbt
├── project/
│   ├── build.properties
│   └── plugins.sbt
└── src/
    └── main/
        └── scala/
            └── job_script.scala

The following figure shows the project structure as it appears in the IDE, with the build configuration and Scala source files.

IDE file tree showing the project layout with build.sbt, the project folder, and the Scala source file

Project structure with build.sbt and Scala source files in the IDE

Step 1: Invoke the upgrade agent

Enter the following prompt:

Use the dataprocessing-mcp server to upgrade my local project at
/Users/<your-path>/AWSSpark4AutoUpgradeDemo/resources/scala/demo_1_spark_change_focus/
from Amazon EMR Serverless version 6.9.0 to Amazon EMR Serverless version 8.0.0.
Use Amazon EMR Serverless app-id <your-app-id> for validation.
Store artifacts at s3://amzn-s3-demo-bucket/spark4-upgrade/scala/

The agent reads your project files, identifies the current Spark and Scala versions, and generates a structured upgrade plan (the analysis ID is logged for traceability). The following figure shows the agent generating the upgrade plan.

Agent output listing the detected Spark 3.3 and Scala 2.12 versions and the generated upgrade plan

Agent generating the upgrade plan with Spark and Scala version analysis

When prompted, choose Yes, proceed with the upgrade as planned, then choose Approve on the subsequent permission dialog to allow the agent to update the build configuration.

Step 2: Build configuration update — build.sbt

The first action of the agent is to invoke update_build_configuration, rewriting build.sbt to target Spark 4.0 and Scala 2.13, as shown in the following figure.

Agent output showing the build.sbt dependency changes from Spark 3.3 to Spark 4.0.1

Agent output showing build.sbt updates from Spark 3.3 to Spark 4.0

The build configuration update is generated. The following are the recommended changes for build.sbt.

Changes summary:

Scala version: Updated from 2.12.15 to 2.13.16 (required for Spark 4.0).
Spark dependencies: Updated from 3.3 to 4.0.
- spark-core: 3.3 → 4.0.
- spark-sql: 3.3 → 4.0.
Assembly settings: Added configuration for creating uber JARs with proper merge strategies.
Dependency exclusions: Added rules to exclude provided dependencies (Spark, Scala, Hadoop) from assembly.

The following figure shows the updated build.sbt and plugins.sbt files after the configuration changes are saved.

The updated build.sbt and plugins.sbt files open in the editor after the configuration changes

Updated build.sbt and plugins.sbt files saved after configuration changes

Step 3: Java environment check

Amazon EMR Serverless 8.0.0 runs on Java 17. The agent invokes check_and_update_build_environment to verify your local Java Development Kit (JDK) and upgrade it from Java 11 to Java 17, as shown in the following figure.

Agent output verifying the local Java version and recommending an upgrade to JDK 17

Agent verifying Java environment and recommending JDK 17 for Amazon EMR 8.0

Step 4: Scala source code transformations

After updating the build configuration, the agent compiles the project and applies fix_upgrade_failure iteratively to resolve Scala 2.13 and Spark 4.0 breaking changes. Scala 2.13 removed several deprecated collection methods that were available in 2.12. The compilation failed with errors related to the Scala 2.13 syntax change. The .to[Set] syntax needs to be updated to .to(Set) for Scala 2.13. The agent used the fix_upgrade_failure tool to resolve the compilation errors. The following are the key transformations applied to job_script.scala.

The following figure shows the agent applying Scala 2.13 source code transformations to resolve the compilation errors.

Agent output showing the Scala 2.13 source code edits applied to job_script.scala

Agent applying Scala 2.13 source code transformations to resolve compilation errors

// Disable ANSI (American National Standards Institute) (SQL compliance mode)(ANSI) mode to handle overflow and malformed cast operations
spark.conf.set("spark.sql.ansi.enabled", "false")

df.createOrReplaceTempView("airports")
// With ANSI mode disabled, overflow values will be handled gracefully
var new_df = spark.sql("SELECT *, CAST(build_time AS SMALLINT) as numeric_build_time FROM airports")
new_df.show()
new_df.createOrReplaceTempView("airports")

// Migration change: The to[Collection] method was replaced by the to(Collection) method.
val airports_in_us: Set[String] = spark.sql("SELECT name FROM airports WHERE country='USA'").collect().map(_.getString(0)).to(Set)
println(airports_in_us)
val airports_in_us_java: java.util.Set[String] = airports_in_us.asJava

// With ANSI mode disabled, malformed CAST operations will return null instead of failing
new_df = spark.sql("SELECT *, CAST(code AS INT) as numeric_code FROM airports")
new_df.show()
new_df.write
  .mode("overwrite")
  .parquet(outputPath)

Before

val airports_in_us: Set[String] = spark.sql("SELECT name FROM airports WHERE country='USA'").collect().map(_.getString(0)).to[Set]

After

val airports_in_us: Set[String] = spark.sql("SELECT name FROM airports WHERE country='USA'").collect().map(_.getString(0)).to(Set)

Code change explanation:

Scala 2.13 changed the collection conversion API. The .to[Collection] syntax was replaced with .to(Collection) using parentheses instead of square brackets.
Updated collection conversion from .to[Set] to .to(Set) to comply with Scala 2.13+ syntax requirements.
Changed import scala.collection.JavaConverters._ to import scala.jdk.CollectionConverters._ and updated .to[Set] to .to(Set).
Renamed the object from Spark3_3_Job to Spark4_0_Job. Updated the Parquet config keys from spark.sql.legacy.parquet.int96RebaseModeInRead/Write to spark.sql.parquet.int96RebaseModeInRead/Write.
Added spark.conf.set("spark.sql.ansi.enabled", "false") to handle overflow and malformed cast operations gracefully.
The output path was updated to s3://xxxxxxxxx/output.

After the compilation succeeds, the agent builds the assembly JAR. Choose Save to create a report for the build result.

Step 5: Runtime validation

Provide the following information to run the validation job on Amazon EMR Serverless.

Amazon EMR Serverless application ID (target application running Spark 4.0 on Amazon EMR 8.0.0):

To create an Amazon EMR application, follow the Amazon EMR documentation, or provide a prompt for the agent to create one for you.
Format: 00xxxxxxxxxxxxxxxxxxxxxxxxxx.

Execution role Amazon Resource Name (ARN) (IAM role for the job):

Set up the execution role following the IAM role guide.
Format: arn:aws:iam::123456789012:role/YourRoleName.

Amazon S3 staging path (for uploading the JAR and storing results):

Format: s3://amzn-s3-demo-bucket/path/.

AWS profile (the AWS profile to use for CLI commands, found in your mcp_settings.json file):

Example: default, dev, pro.

After you submit this information, the agent uploads the JAR to Amazon S3 and submits the validation job with the following arguments:

{
  "analysis_id": "a8869720-e005-41b1-89f3-620e1c5663c0",
  "application_type": "EMR-Serverless",
  "compute_id": "xxxxxxxxxxxxxx",
  "compute_run_config": {
    "executionRoleArn": "arn:aws:iam::xxxxxxxx:role/data-processing-mcp-role",
    "jobDriver": {
      "sparkSubmit": {
        "entryPoint": "s3://xxxxx/xxxxxxx/xxxxxxxxx-xxxxxxx-xxxxxxxx/xxx-job-assembly-1.0.jar",
        "entryPointArguments": [],
        "sparkSubmitParameters": "--class Spark4_0_Job --conf spark.executor.cores=4 --conf spark.executor.memory=16g --conf spark.driver.cores=4 --conf spark.driver.memory=16g --conf spark.executor.instances=2"
      }
    },
    "configurationOverrides": {
      "monitoringConfiguration": {
        "cloudWatchLoggingConfiguration": {
          "enabled": true,
          "logGroupName": "/aws/emr-serverless"
        }
      }
    }
  },
  "enable_data_quality_check": false,
  "s3_staging_path": "s3://xxxxxxx/xxxxxxxxx/",
  "is_source_version_run": false,
  "spark_metadata_file_put_presigned_url": null,
  "static_artifacts_file_paths": null
}

The agent monitors the job status upon approval.

Agent output showing the Amazon EMR Serverless validation job completing with a success status

Amazon EMR Serverless job validation output showing successful completion

Once you receive a success response, the agent proceeds to generate the upgrade summary for your Spark project.

{"success":true,"message":"EMR SERVERLESS job completed successfully","compute_run_id":"xxxxxxxxxx","compute_id":"xxxxxxxxxx","status":"SUCCESS","application_type":"EMR-Serverless","next_action":"Proceed to next step in the plan"}

After the job succeeds, you can optionally cross-check the results by navigating to your Amazon S3 bucket to verify the output files were written successfully, or by reviewing the job run details in the Amazon EMR Serverless console to confirm execution status and logs. Choose Approve to allow the agent to save the summary to a local file and upload it to Amazon S3.

You can also request ANSI mode enablement through the agent interface, and it applies the necessary updates accordingly.

Relevant Spark changes:

Migration rule: Spark 4.0 enables ANSI mode by default. To handle type conversion errors gracefully while keeping ANSI mode enabled, use TRY_CAST instead of CAST.
Change description: Enabled ANSI mode and replaced CAST with TRY_CAST for operations that might fail, specifically timestamp-to-smallint overflow and string-to-int malformed value conversions.

Applied changes:

Code diff — src/main/scala/job_script.scala:
- Changed spark.conf.set("spark.sql.ansi.enabled", "false") to spark.conf.set("spark.sql.ansi.enabled", "true").
- Replaced CAST(build_time AS SMALLINT) with TRY_CAST(build_time AS SMALLINT).
- Replaced CAST(code AS INT) with TRY_CAST(code AS INT).

The agent compiles the change and follows the previous steps to run the job on Amazon EMR Serverless.

Result: SUCCESS

In this Scala workload migration section, the agent automatically upgraded the ecommerce pipeline from Spark 3.3 and Scala 2.12 on Amazon EMR 6.9.0 to Spark 4.0 and Scala 2.13 on Amazon EMR 8.0.0. It rewrote build.sbt and plugins.sbt, upgraded the JDK from 11 to 17, and applied Scala 2.13 syntax fixes (.to(Set), CollectionConverters), Parquet config key updates, and ANSI mode handling with TRY_CAST replacements. The upgraded JAR was compiled, submitted to Amazon EMR Serverless, and validated with a SUCCESS status, completing the full migration without manual code edits.

Clean up

To avoid ongoing charges, delete the resources created during this walkthrough. Start by emptying the Amazon S3 staging bucket, then delete both AWS CloudFormation stacks in reverse order:

Empty the Amazon S3 staging bucket.

aws s3 rm s3://${STAGING_BUCKET_PATH} --recursive

Delete the Amazon EMR Serverless application stack.

aws cloudformation delete-stack --stack-name spark-emr-serverless-upgrade

Delete the MCP setup stack (IAM role and Amazon S3 bucket).

aws cloudformation delete-stack --stack-name spark-upgrade-mcp-setup

Conclusion

The AWS Spark Upgrade Agent transforms what has traditionally been a months-long, error-prone migration process into an automated, IDE-driven workflow that completes in hours. By combining intelligent code analysis, targeted transformations, and an iterative local-to-remote validation loop, the agent handles the complexity of upgrading Scala workloads from Spark 3.x to Spark 4.0 on Amazon EMR 8.x. The demo_1_spark_change_focus walkthrough demonstrates the ability of the agent to automatically update build configurations, apply Scala 2.13 syntax changes, handle Spark 4.0 breaking changes like ANSI mode defaults, and validate results against live Amazon EMR clusters, all through natural language prompts in your IDE. For teams managing large-scale Spark estates, this approach eliminates manual debugging cycles, reduces migration risk, and unlocks the performance gains of Spark 4.0 without the traditional engineering overhead.

Next steps:

If you’re new to the Spark Upgrade Agent, start with the introduction post for a lighter-weight introduction before tackling Scala workloads.
For a complete PySpark implementation and demo, refer to Upgrade PySpark from Spark 3.5 to Spark 4.0 with AWS Spark Upgrade Agent.
When you are ready for production, review the security model in the Architecture section and the IAM role setup guide to confirm your least-privilege configuration before running against production workloads.

Useful resources:

Have questions or feedback? Share your migration experience in the AWS re:Post community or open an issue in the sample repository. We’d love to hear how the agent performs on your workloads.

About the authors

Getting started with AWS End User Messaging Notify

2026-07-28 Brett Ezell

Post Syndicated from Brett Ezell original https://aws.amazon.com/blogs/messaging-and-targeting/getting-started-with-aws-end-user-messaging-notify/

One-time passwords (OTPs) are the backbone of modern user verification, from account creation to wallet additions, to password resets. But for businesses operating globally, delivering OTP messages reliably across dozens of countries is complex. It has traditionally meant navigating a maze of country-specific registrations, originator provisioning, and carrier compliance requirements. AWS End User Messaging Notify changes that equation entirely by removing the complexity of global OTP delivery.

Before we dive into the technical implementation, let’s explore what Notify is, why it matters for businesses sending verification codes at scale, and how you can go from zero to sending OTPs in minutes, not weeks.

The problem with traditional OTP sending

Sending OTP messages through traditional SMS channels requires significant upfront investment. For each country you want to reach, you need to research which origination identity types are supported: short codes, long codes, toll-free numbers, sender IDs, or 10DLC. Each has different registration timelines, throughput limits, and costs. Some countries require pre-registered message templates. Others mandate sender ID registration or risk having your messages displayed as “LIKELY-SCAM” to recipients.

For a business targeting multiple countries, that complexity multiplies fast. Each market carries its own registration process, approval timeline, and compliance requirements. A US short code can take 8–12 weeks to provision. India requires Distributed Ledger Technology (DLT) registration with the Telecom Regulatory Authority of India (TRAI), including entity verification, template approval, and header registration. Even straightforward markets like the UK now require sender ID registration to prevent carrier filtering.

The result? Businesses spend weeks or months on registration paperwork before sending their first verification code. And that assumes every registration is approved on the first attempt.

What is Notify?

Notify is a fully managed OTP and verification code sending service within AWS End User Messaging. Instead of provisioning your own phone numbers, managing carrier registrations, and building routing logic, you create a single Notify configuration and AWS handles the rest: the origination identities, carrier compliance, message routing, and even fraud protection.

The core benefit behind Notify is simplicity. You provide your brand name, select your target countries, choose a pre-approved message template, and start sending. AWS automatically validates your account, assigns the appropriate origination identities for each destination country, configures SMS Protect for fraud prevention, and routes your messages through the optimal delivery path. What previously required weeks of registration work now takes minutes.

The business case for Notify

The value of Notify becomes clear when you look at the traditional OTP implementation journey. A business targeting the United States, United Kingdom, Japan, and India would face four completely different registration processes:

United States: Choose between toll-free (15 business days, best case), 10DLC (moderate timeline), or short code (8-12 weeks or more). Sender IDs are not supported.
United Kingdom: Sender ID registration is required. Unregistered sender IDs risk being displayed as “LIKELY-SCAM.” Short codes are available through an AWS support case.
Japan: Sender IDs are still supported without pre-registration, but number display isn’t guaranteed across all carriers (particularly KDDI). Dedicated short codes are available through an AWS support case.
India: Requires DLT registration including entity verification, content template approval, and header (sender ID) registration. Traditionally one of the longest lead-time markets to onboard.

With Notify, your OTP use case in all four of these countries can be operational in minutes. For markets that previously required weeks, you can send immediately while working through dedicated registrations in parallel.

To illustrate the business impact across industries:

Hospitality and entertainment: Theme parks and resorts with global audiences can verify guest accounts across 50+ countries without managing individual country registrations, so international visitors can add passes to digital wallets, create accounts, and complete elevated security actions.
Ecommerce: Global marketplaces can onboard customers from any supported country with verified OTPs, eliminating the registration bottleneck that delays international expansion.
Financial services: Banks and fintech companies can deploy multi-factor authentication globally, with built-in fraud protection that automatically filters artificially inflated traffic and SMS pumping.
Healthcare: Patient portals can verify identities across international locations, with pre-approved templates that help maintain compliance without custom message body management.

Where to start?

When evaluating Notify for your OTP program, we recommend starting with your highest-volume verification use case. OTP and code verification messages follow predictable patterns, making them ideal candidates for Notify pre-approved templates. More importantly, they provide the most immediate return on investment (ROI) by eliminating the registration overhead for your most critical customer touchpoint.

Consider a phased approach:

Phase 1 – Quick win with Notify: Set up Notify for your primary markets. Get OTP sending operational in minutes. Use this to validate your integration and monitor deliverability.
Phase 2 – Evaluate and scale: As volumes grow, evaluate whether specific high-volume countries benefit from dedicated origination identities for higher throughput or branding purposes. Notify supports a hybrid approach in which you can use your own originators alongside Notify managed identities.
Phase 3 – Complex markets: For countries with strict registration requirements, begin the dedicated registration process in parallel. After they’re approved, integrate those dedicated identities into your sending architecture while Notify continues to handle the rest.

Now that you have decided where to start, the next sections walk you through setting up your first Notify configuration, from prerequisites to sending your first OTP.

Prerequisites

Before you begin, make sure that you have the following prerequisites in place:

An active AWS account with billing configured.
Access to the AWS End User Messaging SMS console.
AWS Identity and Access Management (IAM) permissions for AWS End User Messaging SMS operations.
A brand display name for your Notify configuration.
A planned list of target countries for your OTP messages.

Note: The display name must contain only letters, numbers, spaces, hyphens, or underscores, and can be up to 15 characters.

Understanding Notify tiers

Notify offers two tiers, each designed for different stages of your OTP implementation:

Feature	Basic tier	Advanced tier
Transactions per second (TPS)	1	25
Daily message limit	200 messages/day	Unlimited
Country availability	30 pre-approved low-risk countries	Full country list
Short code access	No	Yes
Fraud protection (SMS Protect)	Mandatory – AWS managed	Mandatory – AWS managed
Compliance verification	Trust-based with audit	AWS verifies opt-in compliance

The Basic tier is designed for getting started, testing, and low-volume use cases. It provides immediate access with conservative limits, which is well suited to validating your integration before scaling to production.

The Advanced tier unlocks higher throughput, unlimited daily sending, and access to the full list of supported countries. To upgrade, you complete a streamlined verification process where AWS confirms your opt-in compliance. This is significantly faster than traditional carrier registration because the verification stays within the AWS boundary. No downstream carrier approval is required.

The Basic tier is live almost immediately. The Advanced tier upgrade requires a brand verification registration that demonstrates a compliant opt-in flow, and most requests are processed within 3-5 business days, which is still much faster than traditional carrier registrations.

Spend limits: Notify has a separate spend limit from standard SMS sending. AWS auto-approves more generous limits for Notify, but plan accordingly if you anticipate high volumes at launch.

Testing tip: While testing, keep in mind there is a per-recipient cap of 10 messages per day per Notify configuration (and 10 per day per account) for any single destination phone number. If repeated test sends to your own phone suddenly stop arriving, this limit, not a configuration error, is the likely cause.

A closer look at Advanced tier country coverage

The Advanced tier documentation states it supports all countries available on AWS End User Messaging SMS. While technically accurate, there is an important distinction not immediately obvious from the console: not all countries are fully managed with Notify.

At the time of writing, 68 of the 247 countries available on the Advanced tier (28%) require you to provide your own origination identity. For those destinations, you still need to go through number provisioning and carrier registration. AWS does not automatically assign managed identities for these countries.

To identify which countries require customer-owned identities, use the AWS End User Messaging SMS v2 API:

aws pinpoint-sms-voice-v2 list-notify-countries --tier ADVANCED

For table view, use:

aws pinpoint-sms-voice-v2 list-notify-countries \
  --tier ADVANCED \
  --output table \
  --query 'sort_by(NotifyCountries, &CountryName)[].{
    "Country Name": CountryName,
    "ISO Code": IsoCountryCode,
    "Supported Channels": join(`, `, SupportedChannels),
    "Supported Use Cases": join(`, `, SupportedUseCases),
    "Supported Tiers": join(`, `, SupportedTiers),
    "Customer Owned Identity Required": CustomerOwnedIdentityRequired
  }'

The output is a formatted table as seen here:

Terminal table of Notify countries showing ISO code, channels, use cases, tiers, and the Customer Owned Identity Required column

Look for the CustomerOwnedIdentityRequired field in the response.

Coverage type	Count	Percentage
Fully managed by AWS	179	72%
Customer-owned identity required	68	28%
Total Advanced tier countries	247	100%

Note: These figures reflect coverage at the time of publication. As AWS continues expanding Notify fully managed coverage, these numbers may change – always use the API for the most current breakdown.

Generating the countries list as a CSV

Because Notify’s country coverage evolves over time as AWS expands fully managed support, we recommend always generating the latest list with the AWS CLI, in a spreadsheet-friendly format, whenever you need it.

aws pinpoint-sms-voice-v2 list-notify-countries \
--tier ADVANCED \
--output json \
--query 'sort_by(NotifyCountries, &CountryName)[].{CountryName: CountryName, IsoCountryCode: IsoCountryCode, SupportedChannels: join(`, `, SupportedChannels), SupportedUseCases: join(`, `, SupportedUseCases), SupportedTiers: join(`, `, SupportedTiers), CustomerOwnedIdentityRequired: CustomerOwnedIdentityRequired}' \
| jq -r '["Country Name","ISO Code","Supported Channels","Supported Use Cases","Supported Tiers","Customer Owned Identity Required"], (.[] | [.CountryName, .IsoCountryCode, .SupportedChannels, .SupportedUseCases, .SupportedTiers, (.CustomerOwnedIdentityRequired | tostring)]) | @csv' > notify-countries.csv

What this means for your planning

For 179 countries, coverage is turnkey. Create your configuration and start sending immediately.
For 68 countries, you need a hybrid setup. Provision origination identities and associate them through a phone pool. Notify still handles routing, templates, and fraud protection.
This reinforces the phased approach: start with the 179 fully managed countries, then register for the remaining 68 in parallel.
Eight countries support SMS only, with no voice: Austria, China, France, Gabon, Germany, Italy, Pakistan, and Slovenia.

Pro tip: Before committing to your target country list, run the ListNotifyCountries API call and look for CustomerOwnedIdentityRequired: true to identify which markets need additional lead time.

How Notify works

Notify simplifies the OTP sending workflow into three steps:

Create a Notify configuration – Provide your brand display name and select your use case (currently code verification). Optionally configure target countries, preferred templates, and channel settings.
Enable countries and channels – Select which countries you want to send to and enable SMS, voice, or both. AWS automatically configures the appropriate origination identities and fraud protection for each country.
Send messages – Use the SendNotifyTextMessage API to deliver OTPs. Pass your Notify configuration ID, the destination phone number, and your template variables (such as the OTP code). AWS handles identity selection, template resolution, and message delivery.

Registration steps

To create your first Notify configuration, open the AWS End User Messaging SMS console, choose Notify configurations, and choose Create configuration.

Enter your brand display name, which cannot be changed after creation.
Select the Code verification use case. Optionally, under Advanced, set your target countries, channels, language code, a default template, and a phone pool (for a hybrid setup).
Choose Create configuration.

For full step-by-step details, see the Getting started with Notify tutorial.

Notify configuration creation form in the AWS End User Messaging SMS console with display name and use case fields

Your configuration activates within moments.

Sending your first OTP

After your configuration is active, you can send a test message directly from the console:

Navigate to your Notify configuration and choose the Test tab.
Enter a destination phone number.
Select a message template and language.
(Optional) Configure the OTP code value and expiration.
Choose Send test message.

The recipient receives a message similar to the following:

“[YourBrandDisplayName], your one-time password verification code is [PASSWORD]. Please do not share this message. Sent by Notify.”

Note: This post focuses on delivering the OTP message itself. Generating a secure OTP code and verifying the code the customer submits back are outside the scope of this post. For guidance on that side of the architecture, including code generation, storage, and verification, see Build a Secure One-Time Password Architecture with AWS.

For programmatic sending, use the AWS End User Messaging SendNotifyTextMessage API (this example uses the AWS End User Messaging SMS phone number simulator):

In your terminal (or AWS CloudShell), create a new Python file:

cat > send_notify_message.py << 'EOF'
import boto3

client = boto3.client('pinpoint-sms-voice-v2')

response = client.send_notify_text_message(
    NotifyConfigurationId='notify-config-1234567890',
    DestinationPhoneNumber='+15555550123',
    TemplateId='your-template-id',
    TemplateVariables={
        'otp': '123456'
    },
    MessageFeedbackEnabled=True
)

print(f"Message ID: {response['MessageId']}")
EOF

Before running, replace the placeholder values with your own:

notify-config-1234567890 – Your Notify configuration ID.
+15555550123 – The destination phone number in E.164 format.
your-template-id – The template ID you created earlier.
123456 – Your OTP or variable value.

Run the script:

python3 send_notify_message.py

You receive output similar to the following:

Message ID: msg-1234567890abcdef0

Note: Notify uses the same pinpoint-sms-voice-v2 API namespace as standard AWS End User Messaging SMS. If you have an existing integration, the migration path is straightforward. The primary difference is the API endpoint and parameters.

Notify and standard AWS End User Messaging SMS: Better together

Notify works alongside your existing AWS End User Messaging SMS infrastructure. This hybrid approach combines the strengths of both:

Notify for rapid global coverage: Use Notify for countries where you don’t have dedicated origination identities. Eliminate registration delays and start sending immediately.
Dedicated identities for high-volume markets: For countries where you need maximum throughput, custom branding, or two-way messaging, continue using your own short codes, toll-free numbers, or registered sender IDs.
Pool-based prioritization: Associate your dedicated numbers in a phone pool with your Notify configuration. Notify prioritizes your numbers first and falls back to AWS managed identities only when needed.

This approach works well for businesses expanding internationally. You can launch OTP verification in new markets on day one with Notify, then transition to dedicated identities as volume and requirements justify the investment.

How Notify compares to standard AWS End User Messaging SMS

The following table compares standard SMS sending with Notify across the capabilities that matter most when planning your OTP program.

Feature	Standard SMS	Notify
Number provisioning	Customer managed	AWS managed
Carrier registration	Customer managed	AWS managed
Time to first message	Days to weeks	Minutes
Message templates	Customer created	Pre-approved by AWS
Fraud protection	Optional (SMS Protect)	Mandatory (SMS Protect)
Country rules	Customer configured	AWS managed with customer controls
Throughput (Basic)	Varies by originator	1 TPS, 200 msgs/day
Throughput (Advanced)	Varies by originator	25 TPS, unlimited daily
Hybrid with own numbers	N/A	Supported via phone pools
API namespace	`pinpoint-sms-voice-v2`	`pinpoint-sms-voice-v2`

Protecting your OTP traffic

Every Notify configuration includes mandatory SMS Protect integration. This means:

Country rules: Control which countries can receive messages. We recommend disabling all countries by default and enabling only your target markets. You can adjust these settings at any time through the console, API, or CLI.
Fraud filtering: AWS automatically filters artificially inflated traffic and SMS pumping. On the Basic tier, filter settings are AWS managed. On the Advanced tier, you retain the same protections with additional controls.
Spend controls: Notify includes a dedicated spend limit separate from your standard SMS spend limit. Monitor your usage through Amazon CloudWatch metrics and set up billing alerts to track spending.

You can view and manage your country rules directly within the Notify configuration. To allow or block specific countries:

Navigate to your Notify configuration.
Choose the Countries tab.
Toggle countries between Allow and Block status.

Note: SMS Protect is part of all AWS End User Messaging, not exclusive to Notify. If you decide to transition from Notify to dedicated origination identities for specific countries, your Protect configurations carry over without additional configuration.

Important considerations

Display name is permanent: Your display name appears in every message and cannot be changed after creation. If you need to experiment, create a test configuration first, then create a production configuration with your finalized brand name.

Templates are pre-approved: You cannot create custom OTP message bodies with Notify. This is by design. Pre-approved templates help maintain carrier compliance and reduce the risk of message filtering. If you need custom message content, use standard SMS sending with your own origination identities.

Architecture compatibility: Notify uses the same V2 API (pinpoint-sms-voice-v2) as standard AWS End User Messaging SMS. If you have an existing architecture using Amazon API Gateway, AWS Lambda, and Amazon Simple Queue Service (Amazon SQS) for message routing, integrating Notify requires only parameter changes, not an architectural redesign.

Message feedback: Notify supports the Message Feedback API for tracking OTP conversion rates. Pass the MessageFeedbackEnabled parameter when sending to track whether recipients successfully verify their codes. This provides first-party data for measuring authentication success rates across countries and carriers.

Managing costs and usage

Notify pricing includes a per-message service fee on top of standard SMS transport rates. The service fee covers origination identity management, fraud protection, and routing optimization.

Monitor your Notify message volume through Amazon CloudWatch metrics and the analytics dashboard within your Notify configuration. Set up billing alerts to track spending against your budget. For the latest rates, see AWS End User Messaging pricing.

Conclusion

In this post, we showed you how AWS End User Messaging Notify eliminates the registration complexity of global OTP sending. You get a fully managed verification code service with built-in fraud protection, pre-approved templates, and coverage across 200+ countries, operational in minutes instead of weeks.

Evaluate your current OTP sending workflow and identify the countries where registration overhead is delaying your go-to-market. Consider starting with Notify for those markets to establish immediate coverage, then layer in dedicated origination identities for high-volume countries as your program scales.

Get started today

Ready to implement Notify? Here are your next steps:

If you’re ready to start: Open the AWS End User Messaging SMS console and create your first Notify configuration. You can be sending test OTPs within 5 minutes.
If you want to learn more: Review the AWS End User Messaging Notify documentation and the Getting started with Notify tutorial.
If you’re evaluating for a global rollout: Check the Notify supported countries list against your target markets and review the Notify tiers and quotas to plan your throughput requirements.
If you need to build a complete OTP architecture: Read our related post, Build a Secure One-Time Password Architecture with AWS, for a comprehensive guide to OTP generation, storage, delivery, and verification.

Resources

About the authors

Build a contract compliance search system with Amazon OpenSearch

2026-07-28 Durga Prasad

Post Syndicated from Durga Prasad original https://aws.amazon.com/blogs/big-data/build-a-contract-compliance-search-system-with-amazon-opensearch/

For legal and compliance teams, auditing a repository of thousands of contracts for a single regulatory obligation shouldn’t take weeks. But with keyword search, it often does. A search for “inadvertent access notification” returns exact matches while missing functionally equivalent clauses such as “security incident disclosure” or “unauthorized access reporting.” This creates two problems:

Discovery gap: Critical risk exposure goes undetected because keyword search cannot match semantically equivalent terms across different contracts.

Review latency: After finding relevant contracts, legal counsel must manually scan lengthy documents to locate the specific clauses that matter. This process can stretch from minutes to hours per document.

Amazon OpenSearch Service is a fully managed search and analytics service that configures, manages, and scales OpenSearch clusters in the AWS Cloud. It supports use cases from log analytics and application monitoring to full-text search and real-time security analytics. It also supports AI-powered semantic search.

Amazon OpenSearch Service addresses both problems through two capabilities:

Semantic search retrieves contracts based on meaning rather than exact keyword matches, closing the discovery gap.
Semantic highlighting pinpoints the exact clauses within retrieved contracts that answer the query, reducing review time from hours of manual scanning to seconds of targeted reading.

In this post, you build a contract compliance search system that combines semantic search with semantic highlighting in Amazon OpenSearch Service. You deploy the solution using two AWS CloudFormation stacks, test it with synthetic contract documents, and see how a single query surfaces both the right contracts and the right clauses within them.

Solution overview

The solution uses a two-stage retrieval and extraction pipeline. First, semantic search identifies relevant contracts across the repository. Then, semantic highlighting marks the specific clauses within those contracts that match the query intent.

The following diagram illustrates the solution architecture:

Solution architecture showing contracts flowing from Amazon S3 through OpenSearch Ingestion and Amazon Bedrock embeddings to semantic search and Amazon SageMaker AI highlighting

Upload contracts to Amazon Simple Storage Service (Amazon S3) – Contract documents (JSON format) are uploaded to an Amazon S3 bucket, which serves as the centralized document repository.
Amazon OpenSearch Ingestion (OSI) reads from S3 – A serverless OSI pipeline detects new documents in the S3 bucket and reads them for processing.
OpenSearch ingest pipeline generates embeddings through Amazon Bedrock – As documents arrive, the ingest pipeline’s text_embedding processor invokes Amazon Titan Text Embeddings V2 through an ML Commons Bedrock connector. This converts contract text into 1024-dimension vector representations, stored in a k-NN index that uses the faiss engine.
User submits a search query – A user queries the system with natural language (for example, “data protection regulations”) through a test AWS Lambda function that forwards the request to OpenSearch using the neural query type.
OpenSearch generates the query embedding – OpenSearch converts the user’s natural language query into a vector embedding using the same machine learning (ML) Commons Amazon Bedrock connector and Amazon Titan V2 model.
Amazon OpenSearch Service performs semantic search – OpenSearch uses k-NN vector similarity to retrieve contracts that are semantically relevant to the query, even when exact terminology differs.
Amazon SageMaker AI performs semantic highlighting – The opensearch-semantic-highlighter-v1 model, hosted on an Amazon SageMaker AI GPU endpoint, scores sentence relevance using cross-encoder inference and wraps the matching clauses in  tags for targeted reading.

How semantic search and semantic highlighting work together

The system processes queries in two steps:

Step 1 – Semantic search (document discovery): You query the contract corpus using natural language. The system retrieves contracts with semantically similar concepts, even when exact terminology differs. For example, searching for “force majeure” returns contracts discussing “natural disasters” or “unforeseeable circumstances” because the system understands these concepts are related.

Step 2 – Semantic highlighting (clause identification): After relevant contracts are retrieved, semantic highlighting automatically marks the clauses that semantically match your search intent. Instead of scanning pages of legal text, you immediately see the specific paragraphs that answer your question.

The difference between standard keyword highlighting and semantic highlighting is significant:

Keyword highlighting wraps individual matching words: termination and rights.
Semantic highlighting wraps entire relevant clauses: Upon termination, the consultant must return all confidential information and proprietary materials within 15 business days..

This reduces false positives, cuts review time, and provides explainability for why each document was retrieved.

Semantic highlighting model deployment

Before the system can highlight clauses based on meaning, the opensearch-semantic-highlighter-v1 model must be deployed to an Amazon SageMaker AI GPU endpoint and registered with the OpenSearch ML Commons plugin through a remote connector.

Stack 2 of the CloudFormation deployment automates this process. It performs the following steps:

Downloads the model artifact from an AWS-managed source and deploys it to an Amazon SageMaker AI endpoint (ml.g5.xlarge).
Creates a remote ML Commons connector in OpenSearch that points to the SageMaker endpoint.
Registers the model with the QUESTION_ANSWERING function so that OpenSearch can use the model’s cross-encoder capabilities to score sentence relevance at query time.

The equivalent manual registration call (handled automatically by the stack) is:

POST /_plugins/_ml/models/_register?deploy=true
{
  "name": "amazon/sentence-highlighting/opensearch-semantic-highlighter-v1",
  "version": "1.0.0",
  "model_format": "TORCH_SCRIPT",
  "function_name": "QUESTION_ANSWERING"
}

You don’t need to run this manually. The deployment script and CloudFormation stack handle model registration end-to-end. The resulting model ID is automatically passed to the query Lambda function for use in semantic highlighting requests.

Index configuration

The index uses a k-NN vector field with 1024 dimensions (matching the Amazon Titan V2 output) and the faiss engine with HNSW method. The mapping includes both a knn_vector field for semantic retrieval and a standard text field for keyword matching and highlighting. When you search for “liability limits,” OpenSearch first retrieves documents through vector similarity, then uses the Amazon SageMaker AI model to identify and wrap the specific relevant sentences in  tags.

PUT /legal-contracts-index
{
  "settings": { "index.knn": true },
  "mappings": {
    "properties": {
      "clause_text": { "type": "text" },
      "clause_embedding": {
        "type": "knn_vector",
        "dimension": 1024,
        "method": {
          "name": "hnsw",
          "engine": "faiss",
          "space_type": "l2"
        }
      }
    }
  }
}

Implementation steps

This section walks you through deploying the solution using two AWS CloudFormation stacks and two shell scripts. You first set up the core infrastructure (OpenSearch, ingestion pipeline, and ML Commons Bedrock connector), then deploy the semantic highlighting model on Amazon SageMaker AI.

Prerequisites

To deploy this solution, you need:

An active AWS account with permissions to create Amazon S3 buckets, AWS Lambda functions, Amazon SageMaker AI endpoints, Amazon Bedrock model access, Amazon OpenSearch Ingestion pipelines, Amazon OpenSearch Service domains, and AWS Identity and Access Management (IAM) roles (including iam:PassRole and sts:AssumeRole). For the exact least-privilege policy, see iam-deployer-policy.json in the repository. Both CloudFormation stacks require the CAPABILITY_NAMED_IAM acknowledgement.
Amazon Bedrock model access enabled for Amazon Titan Text Embeddings V2 (amazon.titan-embed-text-v2:0).
Familiarity with AWS CloudFormation.
Estimated deployment time: approximately 35 minutes.
Estimated cost: approximately USD $ 2.00–3.00 for a quick demo. Delete the stacks promptly after testing.
This post uses US East (N. Virginia) as the deployment AWS Region. Verify service availability in your preferred Region before deploying.

Deploy the solution

The solution deploys using two AWS CloudFormation stacks and two shell scripts. The demo includes synthetic contract documents covering common contract types including software licenses, data processing agreements, managed services, and software as a service (SaaS) subscriptions.

Clone the repository and run the deployment script:

git clone https://github.com/aws-samples/sample-contract-compliance-search-amazon-opensearch.git
cd sample-contract-compliance-search-amazon-opensearch
./deploy.sh

The deployment script creates the following resources across two stacks:

Stack 1:

An Amazon OpenSearch Service domain with fine-grained access control.
An Amazon OpenSearch Ingestion (OSI) pipeline that reads contracts from S3 and sends them to OpenSearch for indexing.
An ML Commons Bedrock connector and ingest pipeline that automatically generates 1024-dimension vector embeddings through Amazon Titan Text Embeddings V2 during document indexing.
A test Lambda function for querying the OpenSearch index using keyword, neural, or hybrid search with semantic highlighting support.
An S3 bucket for storing contract documents.
IAM roles for Lambda functions, the OSI pipeline, and OpenSearch access.

Stack 2:

An Amazon SageMaker AI endpoint hosting the semantic highlighting model.
A Lambda function that creates an ML Commons remote connector in OpenSearch and registers the highlighting model.

After both stacks deploy, the script automatically configures OpenSearch (role mappings, Amazon Bedrock connector, embedding model, k-NN index), ingests the sample contract data, and registers the semantic highlighting model.

The total deployment takes approximately 35 minutes to complete.

(Optional) Automated deployment with Claude Code CLI

If you have Claude Code CLI installed, you can deploy the solution using an AI-assisted workflow that creates a least-privilege IAM role scoped to this demo before deploying:

git clone https://github.com/aws-samples/sample-contract-compliance-search-amazon-opensearch.git
cd sample-contract-compliance-search-amazon-opensearch
./scripts/create-deployer-role.sh
export OS_DEMO_DEPLOYER_ROLE=arn:aws:iam::<ACCOUNT_ID>:role/os-demo-deployer-role
export AWS_DEFAULT_REGION=us-east-1
claude "Deploy the OpenSearch semantic search demo following README.md"

Claude Code reads the repository instructions, assumes the deployer role, deploys both CloudFormation stacks in order, runs the setup scripts, and verifies the deployment end-to-end. The deployer role restricts actions to resources prefixed with os-demo-*, following the principle of least privilege.

Test the solution

After the deployment succeeds, follow these steps to test the solution.

On the Lambda console, choose Functions in the navigation pane.
Choose the function that has os-demo-query in its name.
On the Test tab, in the Event JSON paste this keyword search query {"query": "data protection regulations?", "type": "keyword", "k": 3}
Choose Test to run the Lambda function.

The following screenshot shows the Lambda function test configuration on the AWS Management Console with the keyword search query.

Lambda console Test tab with the keyword search query entered in the Event JSON field

The function processes the query in two ways depending on the search type:

For keyword search (enter: keyword): The function sends a standard match query to OpenSearch, which returns documents containing the exact query terms. The highlight fragments wrap individual matching words like termination and rights.

For neural search (enter: neural): The function sends a hybrid query to OpenSearch combining k-NN (semantic similarity) with keyword matching. OpenSearch automatically generates the query embedding through the ML Commons Amazon Bedrock connector using the same Amazon Titan V2 model. This returns semantically related documents even if they don’t contain the exact query terms. The SageMaker endpoint powers the semantic highlighting, identifying the most relevant clauses within each retrieved document. It wraps entire passages like Upon termination, the consultant must return all confidential information and proprietary materials within 15 business days..

Download the highlight viewer HTML file and open it in the browser. This file helps you view the highlighted text.
Copy the entire execution output of the Lambda execution, paste it into the placeholder in the HTML file, and then choose Load Results.
The following screenshot shows that only the matching keywords are highlighted.

Highlight viewer showing only individual keywords highlighted in the keyword search results

Next, paste the neural search query as input to the Lambda function to see how semantic highlighting works: {"query": "data protection regulations", "type": "neural", "k": 1}
Choose Test to run, and then paste the entire output into the HTML viewer.

The viewer now displays entire sentences highlighted instead of individual keywords.

Highlight viewer showing entire relevant clauses highlighted in the neural search results

Optimizing for scale: batch semantic highlighting

In a standard search, a query might return dozens of relevant contracts. Using the default single inference mode, OpenSearch makes a separate ML call for every document in the result set. For a compliance officer reviewing 50 contracts, this sequential processing introduces noticeable latency.

OpenSearch 3.3 introduced batch inference mode to address this. Batch inference collects the matching documents and processes them in a single ML inference call. In the contract compliance use case, this shifts the performance characteristic from multiple sequential roundtrips to a single parallel execution on the Amazon SageMaker AI GPU.

To enable batch inference, first configure the cluster setting:

PUT _cluster/settings
{
  "persistent": {
    "search.pipeline.enabled_system_generated_factories": ["semantic-highlighter"]
  }
}

Then add batch_inference: true to your highlight options. The following query searches for data privacy clauses across the contracts and highlights the top 10 results using a single batch call:

POST /legal-contracts-index/_search
{
  "query": {
    "neural": {
      "clause_embedding": {
        "query_text": "standard for inadvertent access notification",
        "model_id": "<TEXT_EMBEDDING_MODEL_ID>",
        "k": 10
      }
    }
  },
  "highlight": {
    "fields": {
      "clause_text": { "type": "semantic" }
    },
    "options": {
      "model_id": "<REMOTE_HIGHLIGHTER_MODEL_ID>",
      "batch_inference": true,
      "max_inference_batch_size": 50
    }
  }
}

Best practices

Follow these recommendations to optimize performance, security, and cost-efficiency when deploying the contract compliance search system in production.

Experiment with overlapping chunk sizes (for example, 500 characters with a 10 percent overlap) in your OSI pipeline to verify that context is preserved for long indemnification or liability clauses.
Verify that your Amazon S3 buckets and OpenSearch domains are encrypted using AWS Key Management Service (AWS KMS). For production workloads containing sensitive data, make sure that all traffic stays within your virtual private cloud (VPC) through interface endpoints.

This demo uses simplified configurations for learning purposes. For production deployments, implement VPC isolation, AWS KMS encryption with customer-managed keys, and multi-AZ OpenSearch clusters.

Clean up resources

To avoid ongoing charges, delete the AWS CloudFormation stacks and associated resources:

On the AWS CloudFormation console, choose Stacks in the navigation pane.
Select the os-demo-highlighting stack (Stack 2) and choose Delete. Wait for deletion to complete.
Select the os-demo-search stack (Stack 1) and choose Delete. Stack deletion takes approximately 10–15 minutes to complete.

The stack deletion will automatically remove:

OpenSearch domain.
SageMaker model and endpoint.
Lambda functions.
IAM roles and policies.

After both stacks are deleted, manually delete the S3 bucket (opensearch-cfn-semantic-highlighting-us-east-1-<ACCOUNT_ID>) created for model artifacts. This bucket is provisioned at deploy time and is not managed by CloudFormation. Replace <ACCOUNT_ID> with your AWS account ID in the bucket name.

Conclusion

In this post, you built a contract compliance search system that combines semantic search with semantic highlighting in Amazon OpenSearch Service. The system helps close the discovery gap by retrieving contracts based on meaning rather than exact keywords, and it reduces review latency by highlighting the specific clauses that answer your query.

While we focused on legal agreements, the architecture described here is a blueprint for domains requiring high-stakes document discovery, including:

Regulatory filings: Identifying specific compliance mandates in financial reports.
Technical documentation: Pinpointing troubleshooting steps across massive product manuals.
Research and academia: Isolating specific methodologies within thousands of scientific papers.
Internal knowledge bases: Empowering employees to find exact policy language instantly.

To get started, deploy the solution from the sample repository on GitHub and try semantic search in the Amazon OpenSearch Service console. For more information about semantic search, see Semantic search in the Amazon OpenSearch Service Developer Guide.

About the authors

Accelerating AWS Network Firewall troubleshooting with AWS DevOps Agent

2026-07-24 Salman Ahmed

Post Syndicated from Salman Ahmed original https://aws.amazon.com/blogs/security/accelerating-aws-network-firewall-troubleshooting-with-aws-devops-agent/

When an administrator introduces a rule change in AWS Network Firewall and network connectivity is disrupted, pinpointing the cause requires inspecting multiple points in the traffic path. The firewall gives you stateless and stateful rule engines, domain rules, and routing to the firewall endpoint inside your Amazon Virtual Private Cloud (Amazon VPC). A network drop looks the same from the workload no matter where it started. Isolating the cause means correlating the alert and flow logs with the firewall configuration, route tables, and recent API calls in AWS CloudTrail that might have changed them. That manual correlation is exactly where AWS DevOps Agent helps, accelerating root cause analysis so you can restore connectivity in minutes instead of hours.

AWS DevOps Agent does that correlation for you. As your always-available operations teammate, it resolves and proactively prevents operational issues across AWS, multicloud, and on-premises environments. When an Amazon CloudWatch alarm triggers, it reaches the agent through a webhook. The agent then reads the firewall configuration and logs through AWS APIs, ties the drop to recent API activity, and returns a root cause with a mitigation plan you review before you apply it.

This post connects CloudWatch monitoring to DevOps Agent. It walks through three Network Firewall failures from end to end. The first is a domain deny list blocking a legitimate endpoint. The second is a stateless rule priority misconfiguration. The third is an asymmetric cross Availability Zone (AZ) routing drop. Each maps to a different layer, so each leads down a different investigation path. An AWS Cloud Development Kit (AWS CDK) app deploys the whole environment in your own account so you can reproduce each failure and follow along.

The sample workload

As part of this blog post, we provide a CDK stack that deploys both the AWS DevOps Agent Space and a sample workload used to walk through three separate troubleshooting scenarios. A single t3.micro instance in a protected subnet checks its connectivity to a test endpoint on a continuous loop and publishes results to CloudWatch. Traffic takes the internet egress path through Network Firewall, the NAT gateway, and the internet gateway, so the firewall can intercept or drop it. After completing the walkthrough, you can apply the same troubleshooting techniques with DevOps Agent against your own Network Firewall deployments.

The test endpoint runs in a separate VPC deployed by the same CDK app. It serves HTTPS on port 443 and TCP on port 9142, giving each scenario a different protocol layer to exercise: Scenario 1 targets a TLS connection on 443 (matched by Server Name Indication), Scenario 2 targets a TCP connection on 9142, and Scenario 3 exercises the whole egress path.

A live status page shows one card per scenario plus the network topology. The whole stack deploys from a single CDK app across two Availability Zones, each with a firewall endpoint and NAT gateway, which is what makes Scenario 3 possible.

As shown in the following figure, the egress data path runs from the workload through Network Firewall and the NAT and internet gateways to the test endpoint. The alarm pipeline runs from CloudWatch through Amazon Simple Notification Service (Amazon SNS) and the webhook AWS Lambda function to DevOps Agent.

Figure 1: The sample workload

To use this with your own workload, you need a CloudWatch alarm that detects the connectivity problem and the webhook pipeline (SNS topic and Lambda function) that delivers it to DevOps Agent. The agent reads your firewall configuration, logs, and CloudTrail through AWS APIs, so no additional instrumentation is needed on the firewall side.

Prerequisites

To follow along with this post, you need:

An AWS account with permissions to deploy Amazon VPC, Network Firewall, Amazon Elastic Compute Cloud (Amazon EC2), CloudWatch, Amazon SNS, Lambda, Amazon CloudFront, and AWS Identity and Access Management (IAM) resources
Access to AWS DevOps Agent, with permission to configure its webhook
Node.js 18 or later and npm installed
AWS Command Line Interface (AWS CLI) 2.x configured with credentials
AWS CDK 2.x is required. You can use it through the project’s npx dependency, or install it globally:
- npm install -g aws-cdk

Deploy the sample workload

Clone the project and deploy it into us-east-1 with one command (set awsRegion to use another AWS Region).

git clone https://github.com/aws-samples/sample-accelerating-aws-network-firewall-troubleshooting-with-aws-devops-agent.git
cd sample-accelerating-aws-network-firewall-troubleshooting-with-aws-devops-agent
bash scripts/deploy.sh

The script checks prerequisites, installs dependencies, compiles and tests, and bootstraps the CDK if needed. It then deploys all the stacks from a clean baseline and prints the outputs, including the status-page URL and sign-in details.

Open the status-page link (an https://<random-id>.cloudfront.net address).
Sign in using the username and password provided from the CDK output and confirm all three cards show the green Healthy status.
Keep the page open while you run the scenarios.

Connect AWS DevOps Agent

To connect AWS DevOps Agent to the alarm pipeline

In the AWS DevOps Agent console, open the nf-devops-agent-space Agent Space created by the CDK deployment.
Configure the DevOps Agent webhook and download the CSV file with the webhook URL and signing secret.
On the status page, choose Configure webhook, paste the URL and signing secret, and save. The page writes them to the nf-devops-agent-webhook-credentials AWS Secrets Manager secret, so there is no AWS CLI or console step. Until you set it, the bridge Lambda function sees a placeholder and skips delivery.

Verify the path before you run a scenario. In the Lambda console, open nf-devops-agent-webhook and use the Test tab with this event.

{
  "Records": [
    {
      "Sns": {
        "Message": "{\"AlarmName\":\"TEST-webhook-verification\",\"AlarmDescription\":\"[TEST] Webhook integration test - not a real alarm.\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"[TEST] Manual webhook connectivity test. Safe to ignore.\",\"Region\":\"us-east-1\"}"
      }
    }
  ]
}

A 200 response confirms the path, and a test investigation appears in the DevOps Agent Operator Web App view.

How the alarm pipeline works

Every scenario reaches DevOps Agent the same way. A CloudWatch alarm moves to ALARM and notifies the SNS topic. Amazon SNS invokes a Lambda function. The function reads the webhook URL and signing secret from Secrets Manager, signs an alarm payload, and POSTs it to the DevOps Agent webhook (as shown in Figure 1). Amazon SNS also provides delivery retries, fan-out to other subscribers, and cross-account publishing.

Prebuilt Network Firewall metric (Scenario 1) – Alarm-1 watches the DroppedPackets metric, summed across the stateful streams, and triggers when drops rise above a baseline threshold. This requires no workload or custom metric and works on an already-deployed firewall. However, it only tells you that the firewall is dropping packets, not which rule is responsible.
Application health metric (Scenarios 2 and 3) – Alarm-2 and Alarm-3 watch a custom metric from a connectivity check. Use this for an alarm tied to user-facing impact or to tell one traffic path from another, which requires running a component that emits the metric.

Alarm	Source	Triggers when
Alarm-1	`Native AWS/NetworkFirewall DroppedPackets`	The firewall’s dropped-packet count rises above the baseline
Alarm-2	Custom application health metric	The port 9142 (TCP) connectivity check to the test endpoint is being dropped
Alarm-3	Custom application health metric	The cross Availability Zone connectivity check is being dropped

Run the scenarios

Work through each of the scenarios one at a time, following the same cycle. Interrupt network connectivity, watch the alarm trigger, let DevOps Agent investigate, apply the recommended fix, and confirm recovery before moving on.

The status-page cards follow the live CloudWatch alarm state. A card shows a green dot and the word Healthy when its alarm is clear, and a red dot and the word DROPPED when its alarm triggers. In the DROPPED state the card also adds a Condition: line describing what’s being dropped, which isn’t shown when the card is healthy. Network Firewall applies changes to new flows, so a change shows within a minute or two. Recovery comes from the mitigation DevOps Agent recommends, which you review and apply.

Scenario 1. Domain deny list blocking a legitimate endpoint

At baseline, the rg-domain Suricata domain rule group denies only an unused placeholder, so the test endpoint stays reachable. The rule group inspects the TLS Server Name Indication (SNI) on each outbound connection and drops any that matches a denied domain. The exact rule syntax and console steps follow.

To add the domain deny rule

Go to the Amazon VPC console.
In the navigation pane, under Network Firewall, choose Network Firewall rule groups.
Choose the rg-domain rule group to open its details page.
In the Rules section, choose Edit.
The rules box already contains two baseline placeholder rules (they match blocked.placeholder.invalid, so nothing real is denied). Leave those in place. Find the <app-endpoint-dns> value for Scenario 1 in the deployment script output (a Nework Load Balancer (NLB) DNS name such as NfTest-AppNl-a1b2C3dEf4G5-1234abcd5678efgh.elb.us-east-1.amazonaws.com). On a new line below the existing rules, add a drop rule that matches that DNS name on the TLS SNI, then choose Save.
drop tls $HOME_NET any -> $EXTERNAL_NET any (ssl_state:client_hello; tls.sni; content:"<app-endpoint-dns>"; startswith; nocase; endswith; msg:"S1 domain denylist"; flow:to_server, established; sid:2000002; rev:1;)
After saving, the rules box holds all three lines. The two placeholders remain, plus the new drop rule for the endpoint DNS name (note the distinct sid 2000002).

Figure 2: Scenario 1 – Firewall rule change blocking the connection

What happens. The workload’s HTTPS check to the test endpoint times out, the “AWS/NetworkFirewall DroppedPackets metric climbs above baseline, and Alarm-1 moves to ALARM. The Scenario 1 card reads DROPPED (with the condition Firewall dropping the monitored domain on its allow/deny rules), while the Scenario 2 and Scenario 3 cards stay Healthy (Figure 3). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, because the packets are now dropped at the firewall. To demonstrate the resulting failure, the HTTPS · SNI line from the internet gateway to the test endpoint is shown in red, which the legend defines as dropped (root cause).

Figure 3: Scenario 1 active – Traffic blocked at the firewall

Let DevOps Agent investigate. The agent runs several lines of investigation in parallel and correlates them:

Reads the DroppedPackets metric and correlates the spike with a simultaneous drop in passed packets, confirming the firewall is actively blocking traffic.
Reads the ALERT log and finds the workload’s TLS connections to the test endpoint blocked by the S1 domain denylist rule.
Compares the current state against a baseline window, where the same endpoint was reachable with no alerts, which shows the block is new.
Searches CloudTrail and surfaces the UpdateRuleGroup call that added the deny rule, identifying the user, role, and timestamp approximately one minute before the drops began.
Reports the root cause as that manual rule-group change. Recommends removing the deny entry or adding an allow exception and enabling FirewallPolicyChangeProtection to prevent unauthorized changes.
Presents this as a plan you review and apply, not an automatic change.

In the DevOps Agent Operator Web App view, the agent first restates the Alarm-1 trigger and confirms the firewall is dropping packets above the threshold (Figure 4).

Figure 4: Scenario 1 – The symptom

Next, the agent identifies the root cause: a manual update to the rg-domain rule group that added a domain deny rule (SID 2000002) shortly before the alarm fired, blocking TLS connections to the ELB endpoint (Figure 5).

Figure 5: Scenario 1 – The root cause

Finally, the agent presents a mitigation plan, recommending you remove the problematic deny rule (SID 2000002) to restore connectivity (Figure 6).

Figure 6: Scenario 1 – The mitigation plan

Note: In a real-world environment, this type of rule typically exists for a reason. Before removing it, verify whether it was intentional but scoped too broadly. If so, refine the rule to block only unauthorized endpoints rather than removing it entirely.

Confirm recovery. Apply the change the agent recommends. After the deny entry is gone, DroppedPackets falls back to baseline, Alarm-1 clears, and the card returns to green. Move on to Scenario 2.

Scenario 2. Stateless rule priority misconfiguration

At baseline, the rg-stateless-priority stateless rule group keeps the allow rule at priority 100 and the drop rule at 200 for the test class, TCP destination port 9142. The workload opens a TCP connection to the test endpoint on this port. Lower priority numbers evaluate first, so the allow rule wins. This scenario uses port 9142 instead of 443 to demonstrate a stateless rule, which matches on the packet’s 5-tuple (protocol, ports, addresses) rather than application content.

Introduce the change. Invert the two rule priorities so the drop rule evaluates before the allow rule. This is the kind of change a rushed rule edit can introduce.

To invert the stateless rule priorities

Go to the Amazon VPC console.
In the navigation pane, under Network Firewall, choose Network Firewall rule groups.
Choose the rg-stateless-priority rule group to open its details page.
In the Rules section, choose Edit.
Raise the (Action: Pass) rule’s priority number so it sits after the (Action: Drop) rule, then choose Save. For example, change the (Action: Pass) rule from 100 to 300 (any number higher than the drop rule’s 200 works). You only need to move one rule, and using 300 avoids a clash with the drop rule that already sits at 200. Network Firewall evaluates the lowest priority number first, so the (Action: Drop) rule at 200 now wins for this traffic class, ahead of the (Action: Pass) rule at 300.

Figure 7: Scenario 2 – Rule priority change blocking the traffic class

What happens. The drop rule now wins, the TCP connection to the test endpoint on port 9142 times out, the StatelessRuleFailures metric climbs above baseline, and Alarm-2 moves to ALARM. The Scenario 2 card reads DROPPED (with the condition Stateless rules dropping the monitored traffic class), while the Scenario 1 and Scenario 3 cards stay Healthy (Figure 8). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, because the packets are now dropped at the firewall. To demonstrate the resulting failure, the TLS :9142 line from the internet gateway to the test endpoint is shown in red, which the legend defines as dropped (root cause).

Figure 8: Scenario 2 active

Let DevOps Agent investigate. A stateless drop happens before traffic reaches the stateful inspection engine, so it produces no ALERT log entries. The agent turns to configuration and flow logs instead:

Reads the stateless rule group state and finds the drop rule at the lower priority number, ahead of the pass rule, so the drop evaluates first.
Reads the flow logs and sees passed packets drop to zero within a minute of the change.
Searches CloudTrail and surfaces the UpdateRuleGroup call that inverted the priorities, identifying the user, role, and timestamp about a minute before the alarm.
Reports the root cause as that priority inversion. Recommends removing the redundant drop rule and managing the rule group through infrastructure-as-code (IaC) to prevent manual misconfigurations.
Presents this as a plan you review and apply, not an automatic change.

In the DevOps Agent Operator Web App view, the agent first restates the Alarm-2 trigger and confirms that a workload connectivity health check is failing because the firewall’s stateless rules are dropping egress (Figure 9).

Figure 9: Scenario 2 – The symptom

Next, the agent identifies the root cause, using the rule-group state and CloudTrail to pinpoint the conflicting DROP/PASS rules, where the new DROP rule’s lower priority number makes it match first (Figure 10).

Figure 10: Scenario 2 – The root cause

Finally, the agent presents a mitigation plan, recommending you remove the conflicting DROP rule at priority 200 to restore traffic flow (Figure 11).

Figure 11: Scenario 2 – The mitigation plan

Confirm recovery. Apply the change the agent recommends. After the allow rule is ahead of the drop rule again, Alarm-2 clears and the card returns to green. Move on to Scenario 3.

Scenario 3. Asymmetric cross Availability Zone routing drop

At baseline, the protected subnet in each Availability Zone routes its egress through the firewall endpoint in that same Availability Zone , and the matching return route uses that same endpoint. One endpoint sees both directions of the flow, so the stateful engine completes the handshake. The workload runs in the protected subnet in us-east-1a (CIDR 10.0.4.0/24), so at baseline its egress and its return both use the us-east-1a firewall endpoint.

Introduce the change. Make the flow asymmetric by sending egress out one Availability Zone endpoint while the return comes back through the other. This takes two route edits, and both are required. With only the first edit the flow can still complete, so the alarm will not trigger until both are saved. It makes no firewall-policy change, mirroring a real multi-Availability-Zone routing mistake.

To create asymmetric cross Availability Zone routing

Go to the Amazon VPC console and choose Route tables in the navigation pane.
Flip the egress. Select the NfNetworkStack/SampleVpc/protectedSubnet1 route table (the us-east-1a protected subnet, where the workload runs). On the Routes tab, choose Edit routes. Its 0.0.0.0/0 route currently targets the us-east-1a firewall endpoint. For the target, choose Gateway Load Balancer Endpoint and select the us-east-1b firewall endpoint, then choose Save changes.
Move the return. Select the NfNetworkStack/SampleVpc/publicSubnet2 route table (the us-east-1b public subnet, where egress now exits). Choose Edit routes, then Add route. For the destination enter the workload CIDR 10.0.4.0/24. For the target, choose Gateway Load Balancer Endpoint and select the us-east-1a firewall endpoint. Choose Save changes.

After both edits, a flow’s egress leaves through the us-east-1b endpoint while its return is directed to the us-east-1a endpoint. Neither endpoint sees the whole flow.

Figure 12: Scenario 3 routing change breaking the flow’s symmetry

What happens. A new connection leaves through one endpoint. Its return arrives at the other endpoint, which never saw the connection open, so the handshake fails. Unlike Scenarios 1 and 2, this affects the whole subnet, so all egress stops and Alarm-2 and Alarm-3 both move to ALARM. The AWS/NetworkFirewall DroppedPackets alarm (Alarm-1) stays quiet because no endpoint is making a drop decision. The flow is lost to asymmetric routing rather than counted as a firewall drop. This is why monitoring application connectivity matters. A routing fault is invisible to the firewall’s own drop counter. On the status page, the Scenario 2 card reads DROPPED (with the condition “Stateless rules dropping the monitored traffic class”) and the Scenario 3 card reads DROPPED (with the condition Return traffic dropped by asymmetric cross-Availability-Zone routing), while the Scenario 1 card stays Healthy (Figure 13). On the topology, the alarm pipeline from CloudWatch through Amazon SNS and Lambda to DevOps Agent and the workload-to-firewall inspect lines both turn amber, which the legend defines as collateral / alarm active, while the egress path from the firewall through the NAT gateway and the TLS :9142 and HTTPS · routing lines to the test endpoint turn red, which the legend defines as dropped (root cause).

Figure 13: Scenario 3 – The status page during a path-wide outage

Let DevOps Agent investigate. Both Alarm-2 and Alarm-3 fire in the same datapoint. DevOps Agent recognizes them as linked and merges them into a single investigation:

Reads the flow logs and sees bidirectional TLS connections stop abruptly, with only one-way traffic remaining and no flows reaching the established state.
Reads the firewall metrics and sees received and passed packets shift from one Availability Zone to the other at the moment of the change.
Calls DescribeRouteTables and finds the egress route pointing at one Availability Zone firewall endpoint while the return route points at the other.
Searches CloudTrail and surfaces the ReplaceRoute and CreateRoute calls by the same user, about a minute before both alarms fired.
Reports the root cause as that asymmetric routing change. Recommends restoring symmetric same-Availability-Zone routing so egress and return traverse the same endpoint.
Presents this as a plan you review and apply, not an automatic change.

A mitigation plan is a recommendation you review, not an automatic change, and the right fix depends on the intended design. Restoring symmetric routing can mean sending the workload subnet’s egress back through its own-Availability-Zone firewall endpoint (this sample’s architecture) or, in a design that doesn’t inspect this path, back through a NAT gateway. The agent infers a plausible target from what it can observe, so review the specific route it proposes against your intended topology before you apply it. (Connecting your pipeline or infrastructure-as-code, covered in the next section, lets the agent recommend the target that matches your design.)

In the DevOps Agent Operator Web App view, the agent restates the Alarm-3 (AsymmetricFlowFailures) trigger and confirms the workload’s egress to a monitored endpoint is being blocked by the Network Firewall (Figure 14).

Figure 14: Scenario 3 – The symptom

Next, the agent identifies the root cause: manual route table changes that created cross-AZ asymmetric routing through the network firewall, breaking its symmetric routing requirement (Figure 15)

Figure 15: Scenario 3 – The root cause

Finally, the agent presents a mitigation plan, recommending you restore symmetric routing by pointing protectedSubnet1‘s default route back to the same Availability Zone firewall endpoint, so one endpoint sees both directions of the flow again (Figure 16).

Figure 16: Scenario 3 – The mitigation plan

Confirm recovery. Apply the change the agent recommends, after checking the route target matches your intended design. After the workload subnet’s egress and return use the same Availability Zone firewall endpoint again, the control probe recovers, the alarms clear, and every card returns to green.

Further considerations

In production a single change can trigger several alarms at the same time, as Scenario 3 shows. DevOps Agent links related investigations and works them as one, so you review a single root cause. You can validate the linked findings or unlink an alarm to investigate it independently. If you would rather collapse alarms before they reach the agent, you can add correlation logic in the bridge Lambda function, buffering and grouping by firewall. You can also add email, Amazon Simple Queue Service (Amazon SQS), or HTTP subscribers to the SNS topic, or add the webhook Lambda function to a topic you already run. DevOps Agent produces a mitigation plan but does not change your environment on its own.

You can also give the agent more to work with. DevOps Agent connects to source repositories and CI/CD pipelines, integrating with GitHub (including GitHub Enterprise Server and GitLab Self-Managed through a private connection). It can associate AWS resources with deployments of AWS CloudFormation, AWS CDK, Amazon Elastic Container Registry (Amazon ECR) images, and Terraform. With deployed configuration and recent deployment events in view, the agent correlates the disruption against the change that introduced it and recommends a fix matching your intended design. For this sample, that means recommending the workload subnet’s own Availability Zone firewall endpoint rather than a generic symmetric path.

DevOps Agent also supports proactive incident prevention. It analyzes patterns across past investigations and delivers recommendations to prevent similar issues from recurring, including governance recommendations that strengthen deployment processes and pipeline controls. For Network Firewall rule changes, this means the agent can recommend guardrails for your CI/CD pipeline based on the classes of misconfigurations it has already resolved. You can access these recommendations through the Improvements page in the DevOps Agent Operator Web App.

Clean up

Clean up the environment with one command.

bash scripts/destroy.sh

It reverts any active scenario, runs cdk destroy for all stacks, and sweeps for stragglers by the Project = nf-devops-agent tag. The main cost drivers are the two Network Firewall endpoints, the NAT gateways (one in the main VPC for each Availability Zone, one in the test-endpoint VPC), and the test endpoint’s load balancers. Each of these bills at an hourly rate for as long as it’s provisioned, whether or not traffic is flowing, so a stack left running continues to accrue charges around the clock even while idle. Running the scenarios and tearing the stack down the same day limits the cost to a few active hours rather than days of idle hourly charges.

Conclusion

In this post, we showed you how AWS DevOps Agent accelerates troubleshooting for three common network firewall connectivity issues. The first was a domain deny list. The second was a stateless priority inversion. The third was an asymmetric cross-AZ routing drop. For each one, DevOps Agent investigated the drop and returned a root cause with a mitigation plan you approve before applying. The first scenario triggered on a prebuilt Network Firewall metric, and the other two on application health metrics. That shows both ways to alarm on a firewall problem through one pipeline.

The pattern isn’t specific to Network Firewall. The same flow fits any service that emits CloudWatch metrics and logs, such as AWS WAF, security groups, and network ACLs. Clone the sample repository to explore the solution, then apply what you learn to your own firewall, application, and alarms. For more details, see the AWS Network Firewall Developer Guide and the AWS Network Firewall pricing page. Start with the Getting Started with AWS DevOps Agent guide to connect your first webhook.

Automate creating AWS Glue Data Catalog views with AWS SDK for data mesh use case

2026-07-24 Aarthi Srinivasan

Post Syndicated from Aarthi Srinivasan original https://aws.amazon.com/blogs/big-data/automate-creating-aws-glue-data-catalog-views-with-aws-sdk-for-data-mesh-use-case/

AWS Glue Data Catalog view is a multi-dialect view that supports querying from multiple SQL query engines, such as Amazon Athena, Amazon Redshift Spectrum, Apache Spark in Amazon EMR and AWS Glue. You can create a Data Catalog view in one account, using an AWS Identity and Access Management (IAM) definer role in the same or different account and use AWS Lake Formation to share the view across multiple accounts. The definer role has the required full SELECT on the base tables to create the view and share it with other users for querying. The Data Catalog assumes the definer role and manages access of the base tables when the view is queried, thus allowing to share a subset of data without sharing the underlying base tables.

AWS Glue now adds AWS SDK support for creating and updating the ATHENA dialect of Glue views. With this addition, you can now create ATHENA and SPARK dialects of Glue views simultaneously, using a cross account IAM definer role. This feature enhances the automation to create and update Glue views, like that of Data Catalog tables. In our earlier blog Create AWS Glue Data Catalog views using cross-account definer roles, we had introduced IAM definer roles in a cross-account use case to create Data Catalog views with SPARK dialects using the APIs – CreateTable() and UpdateTable() – while creating and adding ATHENA dialects using Athena query editor. As a continuation to it, this post shows you how to use the Catalog objects API CreateTable() to programmatically create ATHENA and SPARK dialects using cross-account IAM definer roles, and how to add the ATHENA dialect programmatically for the views that were created earlier with only SPARK dialect.

Cross account definer roles enable enterprise data mesh architectures where multiple accounts are interconnected in a central governance and multiple producers and consumers. The central governance account hosts the database, tables and permissions, while the producer accounts maintain CI/CD pipelines to create and manage those data assets. Having the definer role in producer accounts allows those CI/CD pipelines to be fully managed by IAM roles in the individual accounts.

Key points on creating multi-dialect views using cross-account definer roles

ATHENA dialects are validated and asynchronously created. Hence, a cross-account Glue connection is required for validation for every producer account-central governance account pair. This is a one-time setup.
SPARK dialects are not validated. Hence SPARK dialect’s create syntax requires SubObjects list of the base tables and StorageDescriptor fields for the columns of the view.
Though queries on cross account views can be run using database resource link names, the view definition SQL query for creating the view requires the original database and base table names from the central governance account.
If a view has SPARK and ATHENA dialects available, we recommend updating both the dialects of the view simultaneously using update_table() API/SDK, for any changes in the SQL definition of the view or the base table. This will keep both the dialects queryable.
Creating and updating both SPARK and ATHENA dialects using cross account definer role is supported using AWS CloudFormation.
The Data Catalog view that can be created using cross account IAM definer roles are available in SPARK and ATHENA dialects and currently not supported for Redshift Spectrum dialect.

Prerequisites

We use the same setup used in Create AWS Glue Data Catalog views using cross-account definer roles for the sample database, tables, definer role, resource link, IAM and Lake Formation permissions on those resources and principals between the two AWS accounts. Summarizing the requirements as below.

The setup includes a central governance account with Data Catalog database bankdata_icebergdb and two tables transaction_table1 and transaction_table2, a producer account with a Data-Analyst role used as view definer role.
Lake Formation permissions on the central account’s database and tables are granted to the producer account Data-Analyst role as per the earlier blog. The definer role in producer account should have database DESCRIBE and CREATE_TABLE permissions, table SELECT and DESCRIBE permission on all columns and rows of the base tables. The IAM permissions required on the definer role are detailed in Prerequisites for creating views. Similarly, follow the earlier blog to create resource link for the shared database and grant Lake Formation permissions on the resource link to the Data-Analyst
An Athena data source named centraladmin in the producer account, pointing to the Data Catalog of the central governance account.

Creating ATHENA and SPARK dialects at the same time

Creating both ATHENA and SPARK dialects of a Glue catalog view simultaneously is now supported by the AWS SDK. In the producer account, create a new Glue connection, required for the Athena dialect validation. This is a prerequisite for creating the ATHENA dialect of the Glue catalog view using cross account definer role. Then we create a Glue view with both dialects.

Sign in to the producer account as the Lake Formation admin role, or any role with permission to create AWS Glue connections.

Using an AWS Command Line Interface (AWS CLI) environment, such as AWS CloudShell, create an AWS Glue connection as follows.

aws glue create-connection --cli-input-json file://athena-validation-connection.json

The content of athena-validation-connection.json is as follows.

{
    "CatalogId": "<producer-account-id>",
    "ConnectionInput": {
        "Name": "glue-view-validation-connection",
        "Description": "Glue view Athena cross-account validation connection",
        "ConnectionType": "VIEW_VALIDATION_ATHENA",
        "ConnectionProperties": {
            "WORKGROUP_NAME": "primary",
            "DATA_SOURCE": "centraladmin"
        }
    }
}

Note: If you are using Athena for the first time in your account or using Primary workgroup, setup the query results location bucket using Specify a query result location.

Sign out as the Lake Formation admin and sign back in to the producer account as the definer IAM role, Data-Analyst.

Create an AWS Glue view using the create-table CLI command and JSON file, or using the AWS SDK for Python (Boto3) script.

aws glue create-table --cli-input-json file://create_multipledialects.json

The content of create_multipledialects.json is as follows.

 {
   "DatabaseName": "rl_bank_iceberg",
   "TableInput": {
     "Name": "view_2dialects_2basetables_fromcli",
     "StorageDescriptor": {
       "Columns": [
         {
           "Name": "transaction_id",
           "Type": "string"
         },
         {
           "Name": "transaction_type",
           "Type": "string"
         },
         {
           "Name": "transaction_amount",
           "Type": "double"
         },
         {
           "Name": "transaction_location",
           "Type": "string"
         },
         {
           "Name": "transaction_date",
           "Type": "date"
         }
     },
     "ViewDefinition": {
       "SubObjects": [
         "arn:aws:glue:us-west-2:<central-account-id>:table/bankdata_icebergdb/transaction_table1",
         "arn:aws:glue:us-west-2:<central-account-id>:table/bankdata_icebergdb/transaction_table2"
        ],
       "IsProtected": true,
       "Representations": [
         {
           "Dialect": "SPARK",
           "DialectVersion": "1.0",
           "ViewOriginalText": "SELECT a.transaction_id, a.transaction_type, a.transaction_amount, b.transaction_location, b.transaction_date FROM bankdata_icebergdb.transaction_table1 a RIGHT JOIN bankdata_icebergdb.transaction_table2 b ON a.transaction_id = b.transaction_id",
           "ViewExpandedText": "SELECT a.transaction_id, a.transaction_type, a.transaction_amount, b.transaction_location, b.transaction_date FROM bankdata_icebergdb.transaction_table1 a RIGHT JOIN bankdata_icebergdb.transaction_table2 b ON a.transaction_id = b.transaction_id"
         },
         {
            "Dialect": "ATHENA",
            "DialectVersion": "3",
            "ViewOriginalText": "SELECT a.transaction_id, a.transaction_type, a.transaction_amount, b.transaction_location, b.transaction_date FROM bankdata_icebergdb.transaction_table1 a RIGHT JOIN bankdata_icebergdb.transaction_table2 b ON a.transaction_id = b.transaction_id",
            "ValidationConnection": "glue-view-validation-connection"
         }
       ]
     }
   }
}

Notes about fields in the above CLI input JSON (applies to all SDK):

The definer is by default the API caller, but a Definer field can be set to explicitly specify a different IAM role.
In the ViewDefinition, database qualifiers are required for SPARK dialect. That is, the SQL definition provided for ViewOriginalText and ViewExpandedText should be in <source_database_name>.<source_table_name> format.

After the view is created, you can inspect the details on the Lake Formation console. The SQL definitions show both ATHENA and SPARK as shown in the following screenshot.

Lake Formation console showing the SQL definitions tab for the new Data Catalog view, with both ATHENA and SPARK dialects listed

If your view creation fails for any of the dialects, you can use the AWS Glue get-table CLI command with --include-status-details to see what the error is and rectify it.

aws glue get-table --database-name <rl_database_name> --name <view_name> --include-status-details

Glue PySpark script

The PySpark script for creating a view with ATHENA and SPARK dialects are provided below. Download and edit the Pyspark script with your bucket name, producer and central account ids, region and relevant Glue resource names: bdb_5773_createview_bothdialects.py

Provide the following settings to run the script in your Glue Studio. For details on running a Spark job in Glue, refer Working with Spark jobs in AWS Glue.

Choose Data-Analyst as the job execution IAM role.
Choose Glue 5.1 for Glue version.
For the Requested number of workers, provide >=4. This is an FGAC Spark driver requirement, which is needed for Glue catalog views. Below screenshot shows these settings.
Add the following 2 properties as additional job parameters. A screenshot is shown for reference.
```
--datalake-formats = iceberg
--enable-lakeformation-fine-grained-access=true
```
Save and run the Glue job. Check the stdout logs to review the query on the newly created view.

A sample update_table script is also provided below, to illustrate changing the view definition with additional columns. Note the REPLACE keyword:

bdb_5773_updateview_bothdialects.py

Adding ATHENA dialect using SDK to an existing AWS Glue view

You can update an existing AWS Glue view that was created with the SPARK dialect and add the ATHENA dialect using the SDK. The following example uses the update-table CLI command.

aws glue update-table --cli-input-json file://add-athena-dialect.json

The content of add-athena-dialect.json is as follows.

{
    "DatabaseName": "rl_bank_iceberg",
    "ViewUpdateAction": "ADD",
    "TableInput": {
        "Name": "view_sparkfirst_athenanext",
        "ViewDefinition": {
            "Representations": [
                {
                    "Dialect": "ATHENA",
                    "DialectVersion": "3",
                    "ViewOriginalText": "SELECT a.transaction_id, a.transaction_type, a.transaction_amount, b.transaction_location, b.transaction_date FROM bankdata_icebergdb.transaction_table1 a RIGHT JOIN bankdata_icebergdb.transaction_table2 b ON a.transaction_id = b.transaction_id",
                    "ValidationConnection": "glue-view-validation-connection"
                }
            ]
        }
    }
}

Verify the added dialect on the view by reviewing the SQL definitions of the view in Lake Formation console or using GetTable(). If you want to edit the SQL definition or change the base tables of an existing view that has both SPARK and ATHENA dialects, you can do so using the update_table API (using SDK or CLI), with "ViewUpdateAction": “REPLACE” and provide both the dialect definition under ViewDefinition.

You can run queries on the view from the producer account as Data-Analyst. The view can be shared using Lake Formation Tags or named method, just like sharing tables, to additional consumer accounts from the central governance account. The consumer accounts will create a resource link and query the views.

Cleanup

To avoid incurring ongoing costs, clean up the resources you used for this post:

Revoke the Lake Formation permissions granted to the Data-Analyst role and the producer account from the central governance account.
Drop the Data Catalog tables, views, and the database.
Delete the Athena query results from your Amazon Simple Storage Service (Amazon S3) bucket.
Delete the Data-Analyst role from IAM.
Delete the AWS Glue connection and the Athena data source.
Delete the AWS Glue job, if you tried the Python script as an AWS Glue job.

Conclusion

In this post, I demonstrated how to use cross-account IAM definer roles with AWS Glue Data Catalog views, how to create and update ATHENA and SPARK dialects using the Data Catalog CreateTable() and UpdateTable() APIs. The multi-dialect Data Catalog views allow sharing a subset of data from different tables using Lake Formation permissions, including LF-Tags based access control. The cross-account definer roles support multi-account data mesh architectures so that the producer IAM roles can run the CI/CD pipelines in its account. We encourage you to try the feature and share your feedback in the comments.

Acknowledgements: I would like to thank all the team members who worked to add AWS SDK support for creating ATHENA and SPARK dialects together for AWS Glue views – Daniil Arushanov, Wyatt Hawes, Yuxi Wu, Santhosh Padmanabhan and Karthik Devaraj.

About the author

Efficient log management with Amazon OpenSearch Service data streams

2026-07-23 Praveen Krishnamoorthy Ravikumar

Post Syndicated from Praveen Krishnamoorthy Ravikumar original https://aws.amazon.com/blogs/big-data/efficient-log-management-with-amazon-opensearch-service-data-streams/

Time series data workloads in Amazon OpenSearch Service can present unique challenges for organizations, especially when dealing with continuously growing datasets. Many customers struggle with heavily loaded single indices that lead to high query latency, degraded performance, and unnecessary costs. In this post, we show you how to implement data streams with Index State Management (ISM) in Amazon OpenSearch Service. This approach automatically manages your time series data lifecycle and optimizes both performance and costs. Data streams distribute incoming data across multiple backing indices, helping to reduce single-index bottlenecks, while ISM policies automate rollover, retention, and storage tiering to help manage costs.

The challenge

While Amazon OpenSearch Service has long provided tools like Index State Management (ISM) for time series data management, many organizations still struggle with implementing optimal patterns for their continuously growing datasets. Common challenges include:

Performance degradation from index growth: As single indices grow unbounded, query latency increases, you might find shard sizes more difficult to manage, and you might experience strain on your cluster resources.
Manual index management overhead: Without automation, you must invest significant operational effort to manage index lifecycles, rollover, and retention.
Complex setup: Coordinating index templates, aliases, and ISM policies manually can be error-prone.
Inefficient resource utilization: All data residing in hot storage regardless of access patterns, leading to unnecessarily high costs.

Solution overview

As illustrated in Figure 1, our solution uses Amazon OpenSearch Service data streams combined with Index State Management to automatically distribute data across multiple indices and manage the data lifecycle. A data stream is an abstraction layer that simplifies time series data ingestion. It provides a single, consistent endpoint for writes while automatically managing multiple backing indices behind the scenes. Instead of writing directly to individual indices, applications write to the data stream, which routes data to the appropriate backing index.

Here’s how it works:

Data streams provide a single write index when data is first ingested, which can help streamline time series data ingestion.
When the backing index ages or grows to meet your defined criteria, ISM automatically performs the rollover operation.
You can use ISM policies to automatically transition your aged data to different storage tiers based on rules you define and configure.
You can automate the entire process through rules you define in the index template and ISM policies.

Architecture diagram showing time series data flowing into an Amazon OpenSearch Service data stream, which routes writes to multiple backing indices while ISM transitions aged indices from hot to UltraWarm storage

Figure 1 — Time series data workflow using Amazon OpenSearch Service data streams

Data ingests into an index according to your index template configuration. Over time, a data stream creates new indices automatically. Amazon OpenSearch Service manages the lifecycle, transitioning data from hot to warm storage according to the ISM policy configuration you define.

When to use this solution

This approach is ideal when:

You’re ingesting time series data such as logs, metrics, traces, or Internet of Things (IoT) events.
Your data is append-only and immutable.
You’re ingesting millions of documents daily with long-term retention requirements.
You want to reduce storage costs through storage tiering (UltraWarm, Cold storage), as outlined in the blog post Decrease your storage costs with Amazon OpenSearch Service index rollups.

Implementation steps

Prerequisites

Before you begin, make sure that you have the following:

OpenSearch Service domain with UltraWarm nodes (Sample CLI command to create a domain with UltraWarm nodes enabled).

The following steps walk through implementing a time series data solution, using a web server logs database as an example. You can run these steps using any of the following:

For this post, we use the Dev Tools Console in OpenSearch Dashboards. To access it:

Log in to OpenSearch Dashboards.
Navigate to Dev Tools (usually found on the menu under Management).
Use the interactive console to run the commands.

Section 1: Create data stream

Create an ISM policy

First, create an ISM policy that defines the rules for index rollover and storage tier transitions. The following policy defines two states (hot and warm) and sets rules for when indices transition between them. The policy triggers a rollover when the document count reaches 1,000 and moves indices to warm storage after 2 minutes.

Note: The rollover and transitions configurations are only for demo purposes.

PUT _plugins/_ism/policies/ds-ism-policy
{
"policy": {
"description": "rollover policy when index is large",
"default_state": "hot",
"ism_template": [
{
"index_patterns": ["webserver-logs-data-stream*"],
"priority": 300
}
],
"states": [
{
"name": "hot",
"actions": [
{
"rollover": {
"min_doc_count": 1000
}
}
],
"transitions": [
{
"state_name": "warm",
"conditions": {
"min_index_age": "2m"
}
}
]
},
{
"name": "warm",
"actions": [
{
"retry": {
"count": 3,
"backoff": "exponential",
"delay": "1m"
},
"warm_migration": {}
}
]
}
]
}
}

Create an index template for the data stream

An index template matches indices by a regex pattern and applies predefined settings and schema at index creation. The following template maps the required timestamp field for the data stream and associates matching indices with the ISM policy we created.

PUT _index_template/webserver-logs-data-stream-template
{
"index_patterns": ["webserver-logs-data-stream*"],
"data_stream": {
"timestamp_field": {
"name": "timestamp"
}
},
"template": {
"settings": {
"plugins.index_state_management.policy_id": "ds-ism-policy"
},
"mappings": {
"properties": {
"timestamp": {
"type": "date"
}
}
}
}
}

Create data stream

In this step, you create the data stream that handles the time series data. The data stream provides a single, unified write target for ingesting data while managing multiple backing indices behind the scenes.

PUT _data_stream/webserver-logs-data-stream

Validate ISM policy mapping

Verify that the ISM policy is correctly associated with the data stream that was created in the earlier step. This validation step confirms that automatic lifecycle management works as expected.

GET _plugins/_ism/explain/webserver-logs-data-stream

Output

Confirm that policy_id matches the ISM policy name you created earlier and that enabled is set to true.

OpenSearch explain output showing the ISM policy_id mapped to the data stream with enabled set to true

Section 2: Ingesting data to data stream

In real-world scenarios, log data is typically collected and streamed directly to Amazon OpenSearch Service data streams. However, to demonstrate rollover and migration scenarios in this post, we take a different approach. We first load sample log data into a standard OpenSearch Service index, then reindex and migrate that data to a data stream.

To get started, run the commands in the Dev Tools console to create an index and populate it with sample log data.

Reindex existing data

This step shows how to migrate existing data in a traditional index to the new data stream. The reindex operation includes a script that confirms each document has a valid timestamp field (timestamp). Documents without a timestamp field are skipped by the reindex operation to maintain data integrity.

POST _reindex
{
"source": {
"index": "webserver-logs"
},
"dest": {
"index": "webserver-logs-data-stream",
"op_type": "create"
},
"script": {
"source": """
try {
// Validate timestamp field exists and has a value
if (ctx._source.timestamp == null || ctx._source.timestamp.empty) {
ctx.op = 'noop';
}
} catch (Exception e) {
// Skip this document on any error
ctx.op = 'noop';
}
"""
},
"conflicts": "proceed"
}

Monitor index rollover

As shown in Figure 2, after reindexing, monitor the creation of backing indices. The ISM policy evaluates indices every 5 minutes by default. Rollover occurs based on the defined conditions (1,000 documents or 2 minutes of age). Verify this using the following command.

GET _cat/indices/.ds-*?v&h=index,status,health,pri,rep,docs.count,store.size,creation.date&s=index

The output should look like the following:

_cat/indices output listing the .ds backing indices with their status, health, and document counts

You can also validate this from OpenSearch Dashboards by navigating to Index Management, Data streams, webserver-logs-data-stream.

OpenSearch Dashboards Index Management page showing the webserver-logs-data-stream and its backing indices

*Figure 2 — Combined view of the _cat/indices CLI output and the OpenSearch Dashboards data stream details, showing a successful index rollover across multiple backing indices*

Validate warm transition

Verify that indices are correctly transitioning from hot to warm storage based on the ISM policy conditions. You can monitor this through OpenSearch Dashboards or API queries in the Dev Tools console.

GET _plugins/_ism/explain/webserver-logs-data-stream

OpenSearch explain output showing backing indices transitioning from the hot state to the warm state

Verify ingested data

Run a search against the data stream to confirm your documents were successfully indexed:

GET webserver-logs-data-stream/_search
{
"size": 1
}

Output

You should see your ingested documents returned in the hits.hits array, with the timestamp field and other fields you defined in the index template. A non-zero hits.total.value confirms data is flowing correctly through the data stream.

Search results showing an ingested document with the @timestamp field and a non-zero hits.total.value

Clean up

If needed, these commands remove the data stream and its template.

DELETE _data_stream/webserver-logs-data-stream
DELETE _index_template/ webserver-logs-data-stream-template

Delete the sample data.

Conclusion

OpenSearch data streams with ISM offer capabilities for managing time series data at scale. Organizations that implement this approach can see improved query performance through distributed load and smaller, time-based backing indices that support efficient time-range queries. Automated index management reduces operational overhead. Storage tiering automatically moves aged data to UltraWarm storage, which significantly lowers costs without sacrificing access to historical data. Combined with better scalability for growing datasets, this solution simplifies index management while delivering improved performance and a more cost-effective, maintainable infrastructure.

About the authors

Do more with AWS WAF labels using dynamic label interpolation

2026-07-21 Eitav Arditti

Post Syndicated from Eitav Arditti original https://aws.amazon.com/blogs/security/do-more-with-aws-waf-labels-using-dynamic-label-interpolation/

AWS WAF classifies web traffic by attaching metadata to each request it evaluates. Managed rule groups such as AWS WAF Bot Control and AWS WAF Fraud Control account takeover prevention (ATP) attach labels that describe what they found. A label can record that a request came from a known bot category or that it matched a credential-stuffing pattern. You can forward that metadata to your origin as request headers, which gives your backend visibility into the decisions AWS WAF made at the edge. You can also use labels to build tiered policies: a low-confidence bot signal might trigger a CAPTCHA challenge, whereas a high-confidence signal blocks the request outright.

With the AWS WAF AI Activity Dashboard, launched February 24, 2026, Bot Control now identifies more than 650 bots and agents, including search engine crawlers, data collectors, AI assistants, and large language model (LLM) training crawlers, which is ever increasing over time. In an earlier post, we showed how to group Bot Control labels into confidence levels and use them to drive adaptive user experiences in your application. That approach works well when you can list the labels you care about. After the catalog grows past what you can reasonably enumerate, writing a rule for each label becomes a maintenance burden and consumes rule capacity you’d rather spend elsewhere.

With dynamic label interpolation, you can reference labels by namespace instead of by individual name, so a single rule resolves to whichever labels matched during evaluation with no requirement to enumerate each one. You write a ${namespace:} clause in a header value or custom response body, and AWS WAF substitutes the matched values at evaluation time. The feature also gives you synthetic labels you can embed directly in responses, including the client IP address, request JA3 and JA4 fingerprints, and WAF request ID. The rest of this post explains how interpolation resolves labels by referencing four scenarios: forwarding classification data to your application, building custom block and challenge pages, redirecting traffic to a verification step, and segmenting Amazon CloudFront caches by bot category.

Interpolation syntax and behavior

Dynamic label interpolation uses a ${namespace:} syntax that resolves label values at evaluation time. You can use it in three places:

Where	What it does	Syntax
Custom request headers	Inserts resolved label values into headers that AWS WAF forwards to your origin. For example, set X-Bot-Category to so your application receives the matched bot category directly.	in the header value field
Custom response bodies	Embeds label values and synthetic labels (such as client IP or request ID) in block pages, challenge pages, and other custom responses.	in the response body Content field
Custom response headers	Insert label values into response headers (for example, Location for redirects).	in the response header Value field

In each case, AWS WAF reads the labels attached to the request and substitutes the resolved values into the string you provide.

The interpolation syntax

Include a ${namespace:} clause anywhere you would normally put a header value or custom response body. The trailing colon is what signals interpolation, telling AWS WAF to resolve every label in that namespace rather than match a single named label. AWS WAF evaluates each clause against the labels on the request and follows three rules:

Single match – The clause resolves to the label’s terminal value. If the request carries awswaf:managed:aws:bot-control:bot:category:scraping, then ${awswaf:managed:aws:bot-control:bot:category:} resolves to
scraping.
Multiple matches – AWS WAF strips the namespace prefix and returns the values as a comma-separated list, such as scraping,advertising.
No match – The clause resolves to an empty string.

This is backward compatible. AWS WAF only interpolates a value when it contains a ${...} clause, so anything else passes through unchanged. There are no new API fields to set because the syntax is written directly into your existing string values. AWS WAF label namespaces are already colon-delimited (for example, awswaf:managed:aws:bot-control:bot:category:), meaning the required trailing colon won’t collide with header values that don’t follow that pattern.

Synthetic labels

Not every value you might want comes from a rule match. Synthetic labels are derived from the request itself, such as the client’s IP address, the AWS WAF request ID, or the TLS fingerprint, and you interpolate them with the same syntax.

Synthetic label	Description
${awswaf:request_id:}	The unique AWS WAF request identifier
${awswaf:ip:}	The client IP address
${awswaf:ja3:}	The JA3 TLS fingerprint
${awswaf:ja4:}	The JA4 TLS fingerprint

Because synthetic labels work everywhere ${namespace:} interpolation does, you can mix them with namespace-based labels in a single value and pass both to your origin in whatever format suits your application.

The following examples use Bot Control labels, but interpolation isn’t limited to them. It works with most namespaces including labels from other AWS Managed Rules, such as account takeover prevention, account creation fraud prevention, and the IP reputation and anonymous IP lists, as well as labels from AWS Marketplace managed rule groups. This works with labels you custom define based on your own requirements in your own rules.

The same applies to custom labels you define in your own rules. Consider a configuration that classifies requests into tiers based on an API key header, where one rule applies the label and a second interpolates the namespace to forward the result. The first rule matches requests whose x-api-key header begins with pk_enterprise_ and applies the label app:tier:enterprise.

{
  "name": "classify-tier",
  "priority": 100,
  "statement": {
    "byte_match_statement": {
      "search_string": "pk_enterprise_",
      "field_to_match": {
        "single_header": {
          "name": "x-api-key"
        }
      },
      "positional_constraint": "STARTS_WITH",
      "text_transformations": [
        {
          "priority": 0,
          "type": "NONE"
        }
      ]
    }
  },
  "rule_labels": [
    {
      "name": "app:tier:enterprise"
    }
  ],
  "action": {
    "count": {}
  }
}

The second rule matches labels in the app:tier namespace and forwards the resolved value, enterprise, in the x-customer-tier header.

{
  "name": "forward-tier",
  "priority": 200,
  "statement": {
    "label_match_statement": {
      "scope": "NAMESPACE",
      "key": "app:tier:"
    }
  },
  "action": {
    "count": {
      "custom_request_handling": {
        "insert_headers": [
          {
            "name": "x-customer-tier",
            "value": "${awswaf:<ACCOUNT_ID>:webacl:<WEBACL_NAME>:app:tier:}"
          }
        ]
      }
    }
  }
}

In rule_labels, you use the short label name, app:tier:enterprise, and AWS WAF prefixes it with the web ACL context to produce the fully qualified label awswaf:ACCOUNT_ID:webacl:WEBACL_NAME:app:tier:enterprise. A label match statement accepts the short namespace (app:tier:) however an interpolation reference must use the fully qualified the account and web access control list (ACL) context. The payoff is that you can add app:tier:standard, app:tier:trial, or other tiers later, and the forwarding rule picks them up with no changes.

Interpolation also reaches namespaces that the static model never could. Values like the browser fingerprint (awswaf:managed:token:fingerprint) and the unique browser ID (awswaf:managed:token:id) change from request to request, so you can’t write a rule for each one. With interpolation you forward them as ${awswaf:managed:token:fingerprint:} and ${awswaf:managed:token:id:}, which means you can perform in real time device-level tracking, session correlation, and fraud detection that depend on these token-derived signals.

Application signaling

An application signaling pattern uses the labels and forwards them to the origin as customer request headers. After the headers arrive, your application can see how AWS WAF classified the request and decide what to do with that verdict.

Enumerating each label individually doesn’t scale. The common protection level of Bot Control alone tracks more than 650 self-identifying bots and agents, from crawlers to AI data collectors to monitoring services, and targeted protection adds behavioral and machine learning (ML) detection for bots that don’t announce themselves. Mapping only the known bot:category namespace to headers would take hundreds of rules, each one identical except for a hardcoded value. If you followed steps in the blog post How to use AWS WAF Bot Control for Targeted Bots signals and mitigate evasive bots with adaptive user experience, you’ve already mapped labels to confidence levels this way.

The following example forwards the advertising bot category as a header, one of the hundreds you would write to cover the namespace.

{
  "name": "add-header-for-bot-category-advertising",
  "statement": {
    "label_match_statement": {
      "scope": "LABEL",
      "key": "awswaf:managed:aws:bot-control:bot:category:advertising"
    }
  },
  "rule_action": {
    "count": {
      "custom_request_handling": {
        "insert_headers": [
          {
            "name": "bot-category",
            "value": "advertising"
          }
        ]
      }
    }
  }
}

Interpolation collapses that into a single rule. The scope changes from LABEL to NAMESPACE, and the value uses a ${...} clause instead of a hardcoded string. When a request matches, each header resolves to whatever the managed rule group actually applied, whether that is advertising, scraping, or a category that doesn’t exist yet.

{
  "name": "forward-waf-signals",
  "statement": {
    "label_match_statement": {
      "scope": "NAMESPACE",
      "key": "awswaf:managed:aws:bot-control:bot:category:"
    }
  },
  "rule_action": {
    "count": {
      "custom_request_handling": {
        "insert_headers": [
          {
            "name": "x-waf-bot-category",
            "value": "${awswaf:managed:aws:bot-control:bot:category:}"
          },
          {
            "name": "x-waf-bot-name",
            "value": "${awswaf:managed:aws:bot-control:bot:name:}"
          },
          {
            "name": "x-waf-bot-signals",
            "value": "${awswaf:managed:aws:bot-control:signal:}"
          },
          {
            "name": "x-waf-fingerprint",
            "value": "${awswaf:managed:token:fingerprint:}"
          },
          {
            "name": "x-waf-token-id",
            "value": "${awswaf:managed:token:id:}"
          },
          {
            "name": "x-waf-client-ip",
            "value": "${awswaf:ip:}"
          }
        ]
      }
    }
  }
}

This rule matches on the bot:category namespace, then forwards several related namespaces alongside it as separate headers. A more detailed analysis of The x-waf-bot-signals header shows multi-value resolution: the signal: namespace can hold several labels at one time, such as non_browser_user_agent and automated_browser, and they resolve to a comma-separated list. The x-waf-fingerprint and x-waf-token-id headers carry token-derived values unique to each device, which your origin can use for session correlation and fraud detection. And x-waf-client-ip uses a synthetic label to pass the client IP as AWS WAF sees it.

Using these headers, your application can make decisions that AWS WAF can’t make on its own. A signed-in customer flagged with a bot signal might get a simplified page or a different backend, whereas an anonymous session carrying the same signal is blocked outright. A request with several bot signals during a flash sale might be pushed down a queue rather than rejected. A load balancer or API gateway can read the headers and route to different origin pools, sending search_engine traffic, for instance, to a rendering service tuned for crawlers.

These headers are also available to Amazon CloudFront Functions so you can configure custom logic before the request ever reaches your origin.

AWS WAF supplies the signal, and your application supplies the judgment with AWS planning to keep extending this pattern with more detection signals at the edge and more ways to act on them in your application.

Custom block and challenge pages with debug information

False positives are an unavoidable cost of bot mitigation, and the harder problem is usually diagnosing them after they have occurred. Synthetic labels assist with this by embedding the client IP and the AWS WAF request ID in a custom response body, and you give blocked or challenged users a concrete reference to quote when they report a problem. The same approach works for a block page, a CAPTCHA challenge, or a silent challenge because each one supports interpolation in its response body.

{
  "CustomResponseBodies": {
    "BlockPage": {
      "Content": "Your request was blocked.\n\nIP: ${awswaf:ip:}\nRequestID: ${awswaf:request_id:}\n\nIfyou believe this is an error, contact support with the Request ID above.",
      "ContentType": "TEXT_PLAIN"
    }
  }
}

This helps your support workflow because a user who reports they’re blocked can give you the request ID from the page. You search the AWS WAF logs for that ID, look at the rules and labels that matched, and decide whether it was a false positive. There’s no requirement to go back to the user and ask them to reproduce the issue or guess when it happened. For applications where a wrongful block is costly, that shortcut between the user’s screen and your logs is worth building in.

Verification redirects with embedded context

Sometimes the right response isn’t a block but a detour sending suspicious traffic to a verification page before letting it continue. You can build this with AWS WAF by interpolating the client IP and request ID into the redirect target, which is shown in the following example.

{
  "Action": {
    "Block": {
      "CustomResponse": {
        "ResponseCode": 302,
        "ResponseHeaders": [
          {
            "Name": "Location",
            "Value": "/verify?ip=${awswaf:ip:}&rid=${awswaf:request_id:}"
          }
        ]
      }
    }
  }
}

The Location header resolves to an example such as /verify?ip=203.0.113.42&rid=a1b2c3d4-.... The verification endpoint can use the IP for a geo or rate-limit check and the request ID to align the visit with your AWS WAF logs, then send the user on when they pass. Because the redirect is constructed in AWS WAF, you get this behavior without touching the origin application.

CloudFront cache segmentation with AWS WAF labels

When AWS WAF is used in front of Amazon CloudFront, a header that a rule inserts is available to CloudFront when it computes the cache key, which means you can configure and segment your cache by classification. You can interpolate the bot category into a custom header to instruct CloudFront to include that header in the cache key and keep a separate cached response per category. The x-waf-bot-category header from the example forwarding rule above performs this action.

To put this into context, a search_engine request gets a pre-rendered, edge-cached version of the page built for crawling, and if there is a request with no bot label, this request gets the full dynamic page. A scraping request gets a minimal response, also from cache. Crawlers receive indexable content, scrapers stop consuming origin capacity, and human visitors notice no difference. After the first request in each category, all subsequent requests are served from the edge.

You can run the same approach at the origin instead for finer control over freshness. Configure your application to read the classification header and set Cache-Control accordingly and use no-store for unlabeled human traffic to provide fresh content, and longer TTLs for bot-targeted responses so they stay at the edge and off your origin. Which layer you choose depends on how much of this logic you want in CloudFront compared to your own code.

Conclusion

Dynamic label interpolation doesn’t change how labels work, it changes how much rule configuration you need to act on them. A namespace that used to take one rule per value now takes one rule total, and it keeps working as the Bot Control catalog grows past its current 650-plus entries. Along the way, you pick up request-specific block pages, redirects that carry their own context, and cache segmentation keyed on classification. None of these capabilities is dramatic on its own, but when you put them together, you can pair edge classification with judgment in your application.

The feature fits AWS WAF the same way you already use it, with no breaking changes, making adoption a matter of editing rule configurations rather than rebuilding anything. AWS will improve these features in the future by adding detection signals and interpolation capabilities. If you build something with this or would like to see a use case covered in a future post, let us know. You can contribute examples to the AWS Samples repository, start a discussion on AWS re:Post, or leave a comment.

To get started:

Using the URL of this post, you can enter the following examples as prompts in your coding assistant to use this new feature in your preferred environment.

“Using the patterns in the blog post, review my current AWS WAF configuration and identify which static label-to-header mappings can be replaced with dynamic interpolation rules.”
“Create a minimal WAF WebACL (CDK or AWS CloudFormation) with one rule that forwards Bot Control labels to the origin as request headers using `${namespace:}` syntax.”
“Using the AWS Sample referenced in this post, add a new rule that demonstrates dynamic label interpolation with a different managed rule group such as account takeover prevention.”
“My `${namespace:}` interpolation resolves to an empty string. Walk me through the debugging steps: verify the label namespace, check rule priority ordering, and confirm the fully qualified namespace for custom labels.”
“Design a CloudFront cache segmentation strategy using WAF dynamic label interpolation. Include the WAF rule and the origin-side Cache-Control header approach.”

If you have feedback about this post, submit comments in the Comments section below.

Introducing the Amazon GuardDuty investigation agent: on-demand AI-powered threat assessment

2026-07-21 Allan Holmes

Post Syndicated from Allan Holmes original https://aws.amazon.com/blogs/security/introducing-the-amazon-guardduty-investigation-agent-on-demand-ai-powered-threat-assessment/

The new Amazon GuardDuty investigation agent (now in public preview) investigates security findings across your Amazon Web Services (AWS) environment, reducing investigation time from hours to minutes.

GuardDuty is our managed threat detection service that continuously monitors your AWS accounts and workloads for suspicious, potentially malicious activity, and unauthorized behavior, delivering detailed security findings for visibility and remediation.

Whether you’re investigating a single suspicious finding or assessing security posture across your entire organization, the investigation agent provides structured assessments providing risk levels, confidence scores, and actionable recommendations.

Security teams can spend hours investigating security findings and correlating data across multiple tools. The GuardDuty investigation agent automates this correlation, providing actionable intelligence, built directly into GuardDuty and accessible on demand through the AWS Management Console, AWS Command Line Interface (AWS CLI), AWS APIs, or AWS SDKs.

This post shows you how to:

Enable the investigation agent in your GuardDuty console.
Create your first investigation through the console or AWS CLI.
Use the investigation agent with the AWS MCP server for AI-assisted security operations

Key features of the GuardDuty investigation agent

The GuardDuty investigation agent provides APIs using the same patterns you already know from GuardDuty. Each completed investigation returns a risk level, confidence assessment, MITRE ATT&CK® technique mapping, resource mapping, and prioritized recommendations.

You can scope investigations from the console for a specific finding, an account, or all accounts across your organization. Alternatively, the AWS CLI and API accept a free-form trigger prompt of up to 2,048 characters, so you can describe what to investigate in natural language and guide the analysis of the agent by specifying areas of concern, suspected root causes, or priorities for the investigation.

The investigation agent APIs are also available through the official AWS MCP server, part of the Agent Toolkit for AWS, enabling integration into your existing security toolchains and AI-powered workflows. You don’t need to manage or interact with the agent directly. Call API endpoints, and the agent investigates findings, correlates evidence, and delivers an assessment without the overhead of managing complex configurations.

How the investigation agent analyzes findings

When you create an investigation, the agent uses cross-Region inference to process your findings based on scope and produces a structured output.

Cross-Region inference – GuardDuty investigation uses the Cross-Region Inference Service (CRIS), which selects the optimal AWS Region within your geography to process the investigation assessment. Your data remains stored only in the Region where the investigation request originates. However, investigation data and summary results might be processed outside that Region. Data is transmitted encrypted across the secure network provided by Amazon.

For more information about which inference Regions your request might be routed to see the Cross-Region inference routing table located in the investigation section of the Amazon GuardDuty User Guide.

Investigation output – Each completed investigation produces the following insights: Risk level (Info, Low, Medium, High, or Critical), Confidence (Unknown, Low, Medium, or High), Summary (description of findings and key observations), Investigation Details (additional context), and Recommended Actions (detailed actions including AWS CLI commands).

Account scoping – Account specification is required only when investigating a specific member account. For broader scopes such as your entire organization, no account ID is needed. The agent will only investigate findings within accounts you’re authorized to access per the authorization model that follows.

Prerequisites

Before you get started, make sure you have the following prerequisites in place:

Amazon GuardDuty enabled in your account
AWS account in a supported Region (see Availability section)

Required IAM permissions

You will need three new permissions: guardduty:CreateInvestigation to start new investigations, guardduty:GetInvestigation to retrieve results, and guardduty:ListInvestigations to view investigations for a given detector.

Example IAM policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "guardduty:CreateInvestigation",
        "guardduty:GetInvestigation",
        "guardduty:ListInvestigations"
      ],
      "Resource": "*"
    }
  ]
}

Authorization model

Administrator accounts can create investigations, retrieve results, and view investigation lists for themselves and their member accounts. Member accounts can only retrieve results and view investigation lists for their own account. Member accounts can’t create investigations and can’t access investigations belonging to other accounts or the administrator account. Account specification is required only when investigating a specific member account. For your own account or accounts across your organization, no account ID is needed.

To enable and create your first investigation

Before you begin, verify you have the required IAM permissions as described in the prerequisites .

Open the AWS Management Console in the desired supported Region and navigate to Amazon GuardDuty.
In the navigation pane, choose Investigations.

Figure 1: GuardDuty investigation dashboard

If investigations aren’t enabled choose Go to Settings and then enable investigations by choosing Enable.

Figure 2: GuardDuty investigations enablement screen

After investigations are enabled, navigate back to the investigations page.
In the navigation pane, choose Initiate Investigation.

Figure 3: GuardDuty initiate investigation

Select a scope for your investigation:
- Enter a GuardDuty Finding ID: Use when you want to investigate a specific GuardDuty finding in depth
- Enter an AWS Account ID: Use when you want to assess the overall security posture of a specific AWS account
- All accounts: Use for organization-wide security assessment or when investigating potential lateral movement
- Choose Initiate investigation.

Figure 4: GuardDuty investigation setup

Wait for the investigation to complete (typically 2–5 minutes for account level and 10–12 minutes for specific finding investigations during preview). The status updates automatically.
When the investigation completes, select the investigation title to view the full assessment.

Figure 5: GuardDuty investigation completed menu

The investigation assessment contains detailed information about the investigation including general information, a summary of the investigation, mapping, assessment of the threat, and recommended actions.

The General Information section displays the investigation ID, status, triggered-by account, and creation timestamp.

Figure 6: General information section of the assessment

The summary section provides a narrative of key observations and findings.

Figure 7: Summary section of the assessment

The mapping section shows attack techniques and affected AWS resources.

Figure 8: MITRE ATT&CK mapping section of the assessment

The Threat Assessment section displays the risk level, confidence score, and detailed threat analysis.

Figure 9: Threat assessment section

The Recommended Actions section lists prioritized remediation steps.

Figure 10: Recommended actions section of the assessment

Investigations can also be conducted with the AWS CLI or SDK using the following API endpoints:

CreateInvestigation – Initiates a GuardDuty investigation that automatically analyzes security findings, correlates related activity, performs account-level analysis, and produces a structured investigation summary with recommended next steps.
GetInvestigation – Retrieve the status and results of a specific investigation, including the assessment from the agent, correlated evidence, and recommended actions when completed.
ListInvestigations – View investigations across your environment with filtering and pagination.

To run investigations using the AWS CLI

Investigations are asynchronous because the agent queries multiple data sources, correlates findings across services, and performs AI-based analysis. After creating an investigation, you’ll need to check its status periodically until it completes.

Step 1: Find your detector ID

Each GuardDuty deployment has a unique detector ID per-account and per-Region that identifies your specific GuardDuty configuration. You will need this for all AWS CLI operations, especially if you have GuardDuty enabled in multiple Regions. You can find your detector ID in the GuardDuty console under Settings, or by running the following command and specifying the Region. For example, if the GuardDuty detector of interest were in the us-east-1 (N. Virginia) Region

aws guardduty list-detectors –-region=us-east-1

Expected response:

{
  "DetectorIds": [
    "12abc34d567e8fa901bc2d34eexample"
  ]
}

Note: the DetectorIDvalue from the response, you will use it in all subsequent commands.

Or if working only in the same Region, the session can be set as an environment variable to avoid repetition, for example on Linux:

export AWS_DEFAULT_REGION=us-east-1

See the AWS CLI documentation for guidance on configuring this for additional operating systems.

Step 2: Create an investigation

The following is an example of code to investigate a specific finding:

aws guardduty create-investigation us-east-1 \
--detector-id 12abc34d567e8fa901bc2d34eexample \
--trigger-prompt "Investigate this finding ID 1ab2c3d4e5f6a7b8c9d0e1f2a3b4c5d6"

The --trigger-prompt parameter is useful when you have context that isn’t captured in GuardDuty metadata or consumable through the API.

Expected response:

{
  "InvestigationId":"a1b2c3d4-5678-90ab-cdef-ef1234567890"
}

To investigate findings across an entire AWS account, use the following example:

aws guardduty create-investigation –-region=us-east-1 \
--detector-id 12abc34d567e8fa901bc2d34eexample \
--trigger-prompt “Investigate findings in Account 123456789012”

To investigate findings across an entire organization:

aws guardduty create-investigation –-region=us-east-1 \
--detector-id 12abc34d567e8fa901bc2d34eexample \
--trigger-prompt “Investigate findings across my AWS Organization”

Step 3: Check investigation status

Check the status of the investigation shown here using the AWS CLI query command to filter and list only the Status section of the output for simplicity:

aws guardduty get-investigation –-region=us-east-1 \
--detector-id 12abc34d567e8fa901bc2d34eexample \
--investigation-id a1b2c3d4-5678-90ab-cdef-ef1234567890 --query 'Investigation.Status'

Repeat this command until the Status field shows COMPLETED.

Example completed response output:

{
  "Investigation": {
    "InvestigationId": "a1b2c3d4-5678-90ab-cdef-ef1234567890",
    "Status": "COMPLETED",
    "TriggerPrompt": "Investigate finding 1ab2c3d4e5f6a7b8c9d0e1f2a3b4c5d6 in account 123456789012",
    "TriggeredBy": "123456789012",
    "RiskLevel": "Critical",
    "Risk": "Active multi-stage runtime compromise on EKS worker node with root-privileged reverse shell, Docker socket access, malicious file execution, and 500 multi-tactic runtime signals — behavioral evidence is consistent with a genuine intrusion.",
    "Confidence": "High",
    "Summary": "{\"keyObservations\":{\"title\":\"...\",\"narrative\":\"...\",\"observations\":[...]},\"countermeasures\":[...],\"threatAssessment\":{...}}",
    "Cloud": {
      "Provider": "AWS",
      "Region": "us-east-1",
      "Account": "123456789012"
    },
    "Metadata": {
      "Product": {
        "Name": "AmazonGuardDuty AI Analyst",
        "Feature": "Investigation"
      },
      "Version": "1.0.0"
    },
    "StartTime": 1705319400.0,
    "EndTime": 1705319700.0
  }
}

Status values – RUNNING, COMPLETED, FAILED
Timing – Investigation times can very. Checking status every 30 seconds should be sufficient to yield results.
If status shows FAILED – Review the error message in the response and verify your permissions match the authorization model requirements.

To list all investigations for a given detector run the following, the max-results command is optional but useful to filter the number of returned results.

aws guardduty list-investigations –-region=us-east-1 \
--detector-id 12abc34d567e8fa901bc2d34eexample \
--max-results=10

Beyond running investigations manually, the API-first design addresses a common customer pattern: sending GuardDuty findings to third-party tools. You can now add automated investigation to those existing pipelines, so your team receives enriched, prioritized intelligence rather than raw alerts.

Consider a customer that routes GuardDuty findings through Amazon EventBridge to their Security Information and Event Management (SIEM) platform, where analysts manually investigate each alert. With the investigation agent, an AWS Lambda function can be placed into the pipeline that calls CreateInvestigation with the finding ID, waits for completion, and forwards the enriched results (risk level, confidence score, MITRE ATT&CK mapping, and recommended actions) to their SIEM alongside the original finding. Critical findings route directly to the customer incident response queue for further analysis or automation. Low-risk findings with high confidence get auto-closed or batched for weekly review. The analyst’s time shifts from repetitive log correlation to validating assessments and acting on confirmed threats.

This pattern works with SIEMs, ticketing systems, or automation platforms that can be customized to use the API or EventBridge messaging. The investigation agent fits into the pipeline as a processing step, not a destination.

The agent is fine-tuned on investigating GuardDuty findings. It’s distinct from other AWS frontier agents such as the AWS Security Agent and AWS DevOps Agent. The scope of the investigation agent is focused to deliver specialized analysis of GuardDuty findings.

Integration with the AWS MCP server

The Model Context Protocol (MCP) is an open standard that allows AI assistants to securely connect to external data sources and tools. Because the AWS MCP server implements this standard for AWS services, you can use it to add GuardDuty investigations into AI-powered workflows using tools like Kiro, Anthropic’s Claude, or other MCP-compatible clients.

To configure the AWS MCP server

Configure your MCP client to connect to the AWS MCP server.
Use natural language to invoke investigations (for example,“Investigate the recent Unauthorized Access finding for account 123456789012″).
Review the investigation results returned through your MCP client. These results can vary depending on the model or agent being used, configuration, and the non-deterministic nature of AI.

Integrate the results into your existing agent automation or take manual action based on the findings.

Additional usage examples

“Investigate the latest high-severity finding in my production account”
“Create an investigation for finding ID abc123 in account 987654321098 and summarize what happened”
“List investigations from the last 24 hours and flag those that need human review”

How the investigation agent relates to AWS Security Incident Response

At re:Invent 2024, AWS launched AWS Security Incident Response (AWS SIR), a managed service that you can use to quickly prepare for, respond to, and recover from security incidents. AWS SIR and the GuardDuty investigation agent address different stages of your security workflow. The GuardDuty investigation agent provides an on-demand assessment capability. When your team needs deeper context on a specific finding, an account security posture, or the overall security posture of your organization. You create an investigation and receive a structured assessment with risk levels, confidence scores, MITRE ATT&CK® technique mappings, and actionable recommendations. Security analysts can use this to quickly understand the scope and severity of what GuardDuty has detected.

When you create an AWS-supported case through AWS SIR, a SIR investigation agent activates, working in parallel with AWS Security Incident Response engineers to gather evidence and deliver an investigation summary within minutes. AWS SIR is purpose-built for active security events where you need both AI-powered automation and human expertise to coordinate containment and recovery.

Security teams can use these capabilities to assess and prioritize findings on demand using the GuardDuty investigation agent, escalate confirmed issues to stakeholders with supporting evidence, and create or update an AWS-supported case to accelerate involvement from the AWS SIR team when additional support is needed.

Availability and pricing

Public preview of the GuardDuty investigation agent is available in 10 AWS Regions including US East (N. Virginia), US East (Ohio), US West (Oregon), Canada (Central), Europe (Frankfurt), Europe (Ireland), Europe (London), Europe (Paris), Europe (Stockholm), and Asia Pacific (Tokyo).

During public preview, the investigation agent is available at no charge. Usage is limited to 10 investigations per account per day, with a cumulative limit of 100 investigations per account during the preview period. Failed investigations do not count toward these quotas.

Start investigating findings today

The Amazon GuardDuty investigation agent reduces investigation time from hours to minutes, letting your security team focus on confirmed security events rather than manual correlation.

Get started by:

Enabling the investigation agent in your GuardDuty console
Creating your first investigation using a recent GuardDuty finding
Reviewing the structured assessment, including risk level and recommended next steps

For organizations using the AWS MCP server, you can also invoke investigations through natural language in your AI assistant of choice.

Learn more

Amazon GuardDuty Investigation Agent documentation – Detailed configuration and usage guidance
IAM permissions reference for GuardDuty – Required permissions for investigation operations
Amazon EventBridge User Guide – Learn how to route investigation events to downstream systems
MITRE ATT&CK®Framework – Getting started resources
AWS Security Incident Response – AWS Security Blog article
AWS Activation Days – Information about upcoming complimentary events that provide you prescriptive guidance to achieve outcomes when using AWS

If you have feedback about this post, submit comments in the Comments section below.

Govern Amazon Redshift Data Warehouses Data Across Accounts using Amazon SageMaker Unified Studio

2026-07-20 Bandana Das

Post Syndicated from Bandana Das original https://aws.amazon.com/blogs/big-data/govern-amazon-redshift-data-across-accounts-with-sagemaker-unified-studio/

Managing data governance across multiple Amazon Redshift clusters in different AWS accounts presents significant challenges. Organizations operating multiple Amazon Redshift clusters across AWS accounts often rely on manual processes for secure data sharing, which increases operational overhead and governance requirements. In this post, we show you how to use Amazon SageMaker Unified Studio to implement cross-account data sharing in Amazon Redshift using data mesh principles. We demonstrate how to build a scalable data mesh architecture that supports secure, auditable data sharing across AWS accounts while reducing operational burden.

Amazon SageMaker Unified Studio as the backbone of our data mesh

Amazon SageMaker Unified Studio is a data and AI development service which brings together functionality and tools from existing AWS Analytics and AI and machine learning (ML) services, including Amazon EMR, AWS Glue, Amazon Athena, Amazon Redshift, Amazon Bedrock and Amazon SageMaker AI. With the service, organizations can catalog, discover, share, and govern data stored across Amazon Web Services (AWS) without relying on manual coordination between AWS accounts.

A data mesh is an architectural approach that treats data as a product, with decentralized ownership by data producers while maintaining centralized governance. This architecture separates source systems, data producers (data publishers), data consumers (data subscribers), and central governance. The solution we present is tailored for cross-AWS account usage, creating a foundation for data governance so you can share data across Amazon Redshift clusters in different AWS accounts.

Our proposed solution addresses the following common challenges that organizations face when sharing data across AWS accounts:

Manual, ad-hoc data sharing processes are replaced with automated, event-driven data publishing to the SageMaker Unified Studio catalog.
Inconsistent governance across different use cases is resolved through a consistent governance framework with proper access controls.
High load on producer Amazon Redshift clusters is reduced through decoupled publishing that lowers the operational burden on data producers.
Complex credential management is simplified using AWS Secrets Manager and AWS KMS encryption.
Lack of auditable data publishing is addressed with full traceability of access and permissions supported by the SageMaker Unified Studio service.

With this approach, you can help reduce the time and effort required for cross-account data sharing while maintaining security and governance standards.

Architectural overview

The architecture spans three AWS accounts, each with a distinct role in the data mesh:

Central Data Governance Account (Account A) hosts the Amazon SageMaker Unified Studio domain, which serves as the unified catalog and governance layer for data discovery, access control, and subscription management across accounts.

Data Producer (Account B) hosts the source of data and processing workflows. Raw data lands in an Amazon Simple Storage Service (Amazon S3) source bucket and is processed through AWS Glue extract, transform, and load (ETL) jobs or Amazon Redshift auto copy into the Amazon Redshift source database. Amazon Redshift credentials are securely stored in AWS Secrets Manager.

Data Consumer (Account C) hosts the target Amazon Redshift database and analytics workflows. After access is granted, consumers can query shared data and connect downstream visualization tools.

While this diagram shows a single producer and consumer for simplicity, in a real-world deployment there might be hundreds of producer and consumer accounts connecting through the central governance layer. Amazon SageMaker Unified Studio scales to support this by providing a single place for managing data products regardless of the number of participating accounts.

The data sharing workflow is driven by Amazon SageMaker Unified Studio. The data owner publishes data to the catalog, where it becomes discoverable by consumers across accounts. Consumers browse the catalog, subscribe to data products, and the data owner approves the request. After approval, Amazon SageMaker Unified Studio handles the cross-account sharing, granting the consumer access without requiring direct connectivity between producer and consumer Amazon Redshift clusters.

Publishing Amazon Redshift data assets to the data mesh

In a data mesh architecture, data producers need to make their data products discoverable and accessible across the organization. Amazon SageMaker Unified Studio provides a centralized catalog where data assets can be published for consumer subscription.

In practice, this means registering your data sources with the catalog so they can be discovered, governed, and subscribed to by consuming teams. This section walks through the steps required to register Amazon Redshift data sources with SageMaker Unified Studio.

Before you can publish data assets from your producer account, you need to complete several configuration steps across your Amazon Redshift cluster, AWS Secrets Manager, and Amazon SageMaker Unified Studio.

Prerequisites

Install the AWS Command Line Interface (AWS CLI) (v2.15+ recommended).
Obtain temporary credentials with permissions to administer each account (producer, consumer, and domain account)
IAM permissions required: redshift:* on the relevant clusters, secretsmanager:CreateSecret / PutResourcePolicy / TagResource, kms:CreateKey / PutKeyPolicy / TagResource, datazone:* for subscription-target creation, and iam:PassRole for the Amazon Redshift cluster role.
Amazon Redshift clusters must use RA3 node types (ra3.xlplus, ra3.4xlarge, or ra3.16xlarge). Data sharing is not supported on other node types.
Amazon SageMaker Unified Studio domain must already be created in Account A with the Tooling and LakeHouseCatalog blueprints available.
All resources must be in an AWS Region where Amazon SageMaker Unified Studio is available.

Step 1: Account association and blueprint enablement

To implement the data mesh architecture described in the previous section, you need to set up the following accounts and enable the required blueprints. This ensures that the central governance layer can discover and manage data assets across your producer and consumer accounts.

This post uses three separate AWS accounts to illustrate the cross-account data sharing pattern. However, Amazon SageMaker Unified Studio also supports publishing and subscribing to data within a single account or across any number of accounts depending on your organizational setup. Additionally, this walkthrough uses a provisioned Amazon Redshift cluster, but Amazon SageMaker Unified Studio also supports Amazon Redshift Serverless for both publishing and subscribing to data assets.

Amazon SageMaker Unified Studio Domain account (Account A) – This is the central data governance account, which will have the Amazon SageMaker Unified Studio Domain and Project.
Data producer account (Account B) – This account acts as the data domain producer. It has been added as an associated account to Account A.
Data consumer account (Account C) – This account acts as the data consumer. It has been added as an associated account to Account A.
- Log in to Data producer account (Account B) and Data consumer account (Account C).Open the Amazon SageMaker AI console, choose Associated domains, choose your domain, and enable the following blueprints: Tooling and LakeHouseCatalog
Create a project profile for each associated account (Account B and Account C). Make sure you can log in to the SageMaker Unified Studio domain portal and create a corresponding project with relevant project profile within Amazon SageMaker Unified Studio Domain. You must be added as a member of the corresponding project

Step 2: Configure your Amazon Redshift cluster and credentials

In the producer account (Account B), the data to be shared resides in an Amazon Redshift cluster.
Verify that your Amazon Redshift cluster uses node types from the RA3 family.
Add the following tags to your Amazon Redshift cluster.

Amazon Redshift console showing tags added to the cluster

Create a superuser in Amazon Redshift for Amazon SageMaker Unified Studio. For the Amazon Redshift cluster, the database user you provide in AWS Secrets Manager must have superuser permissions. With superuser permission, your Amazon Redshift cluster can publish data and subscribe from the data mesh created with Amazon SageMaker Unified Studio, and it manages the subscriptions (access) on your behalf. For reference, see the note section in this QuickStart guide with sample Amazon Redshift data.

Tag key-value pairs configured on the Amazon Redshift cluster

Store the user’s credentials in Secrets Manager. Select the credential type, enter the credential values, and choose the AWS Key Management Service (AWS KMS) key with which to encrypt the secret

QuickStart guide note about providing superuser credentials for Amazon SageMaker Unified Studio

Add the tags to the AWS Secrets Manager secret to allow Amazon SageMaker Unified Studio to find this secret and limit the access to a particular Amazon SageMaker Unified Studio domain. The Amazon Redshift cluster Amazon Resource Name (ARN) must be added as a tag. For reference, see the QuickStart guide with sample Amazon Redshift data.

Tags on the AWS Secrets Manager secret including the Amazon Redshift cluster ARN

Add the following resource policy to the secret. See the Amazon SageMaker Unified Studio Provisioning guide for the full reference.

Resource policy added to the AWS Secrets Manager secret for Amazon SageMaker Unified Studio access

If your secret is encrypted with a customer managed AWS KMS key, append the key policy with the following statement and add a tag to the key: AmazonDataZoneEnvironment = All. You can skip this step if you’re using an AWS managed KMS key.

{
    "Sid": "AllowSMUSRolesSecretsAccess",
    "Effect": "Allow",
    "Principal": {
        "AWS": "*"
    },
    "Action": [
        "kms:Decrypt",
        "kms:DescribeKey",
        "kms:GenerateDataKey"
    ],
    "Resource": "*",
    "Condition": {
        "StringEquals": {
            "kms:ViaService": "secretsmanager.<<AWS_Region>>.amazonaws.com"
        },
        "StringLike": {
            "aws:PrincipalArn": [
                "arn:aws:iam::<<Data_Producer_Acct_Id(Account B)>>:role/aws-service-role/redshift.amazonaws.com/AWSServiceRoleForRedshift",
                "arn:aws:iam::<<Data_Producer_Acct_Id(Account B)>>:role/<<Redshift_Cluster_IAM_Role_Name>>",
                "arn:aws:iam::<<Data_Producer_Acct_Id(Account B)>>:role/datazone*",
                "arn:aws:iam::<<Data_Producer_Acct_Id(Account B)>>:role/service-role/AmazonSageMaker*"
            ]
        }
    }
},
{
    "Sid": "AllowSMUSRolesCreateGrant",
    "Effect": "Allow",
    "Principal": {
        "AWS": "*"
    },
    "Action": "kms:CreateGrant",
    "Resource": "*",
    "Condition": {
        "Bool": {
            "kms:GrantIsForAWSResource": "true"
        },
        "StringEquals": {
            "kms:ViaService": "secretsmanager.<<AWS_Region>>.amazonaws.com"
        },
        "StringLike": {
            "aws:PrincipalArn": [
                "arn:aws:iam::<<Data_Producer_Acct_Id(Account B)>>:role/aws-service-role/redshift.amazonaws.com/AWSServiceRoleForRedshift",
                "arn:aws:iam::<<Data_Producer_Acct_Id(Account B)>>:role/<<Redshift_Cluster_IAM_Role_Name>>"
            ]
        }
    }
}

Note: Enable automatic rotation. Configure Secrets Manager automatic rotation for this secret with a rotation interval appropriate to your security policy (for example, every 30 days). When implementing rotation, verify that the rotation Lambda function updates the credentials in both Secrets Manager and Amazon Redshift database users simultaneously. Note that Amazon SageMaker Unified Studio retrieves the secret at connection time, so rotation must produce credentials that are valid immediately upon storage: use the alternating-users rotation strategy if you need to avoid downtime during rotation. See the Secrets Manager rotation documentation for setup instructions.

Using Amazon Redshift Serverless?

Add the following Tags to the Amazon Redshift Serverless namespace and workgroup.

Tags added to the Amazon Redshift Serverless namespace and workgroup

In the Secrets Manager secret, verify the host points to your Serverless endpoint.

AWS Secrets Manager secret showing the host pointing to the Serverless endpoint

Add the following tags to the AWS Secrets Manager secret.

Tags added to the AWS Secrets Manager secret for the Redshift Serverless credentials

Publish Amazon Redshift data to the data mesh

With prerequisites complete, you can now register your Amazon Redshift cluster as a data source in Amazon SageMaker Unified Studio.

Step 1: Create an Amazon Redshift type connection

Sign in to Account B, navigate to your Amazon SageMaker Unified Studio associated domain, and open the Amazon SageMaker Unified Studio URL.

Amazon SageMaker Unified Studio associated domain sign-in page

Follow the instruction here to Add an Amazon Redshift Connection

Add an Amazon Redshift connection form in Amazon SageMaker Unified Studio

The newly created Amazon Redshift connection appears here.

Newly created Amazon Redshift connection listed in Amazon SageMaker Unified Studio

Step 2: Create the data source for your Amazon Redshift data warehouse

Follow the instruction here to Add an Amazon Redshift data source

Add an Amazon Redshift data source form in Amazon SageMaker Unified Studio

Amazon Redshift data source configuration in Amazon SageMaker Unified Studio

For Publishing settings, choose whether assets are immediately discoverable in Amazon SageMaker Catalog.

Publishing settings controlling asset discoverability in the Amazon SageMaker catalog

Using Amazon Redshift Serverless?

When creating the connection and data source, use your workgroupName instead of clusterName. The rest of the data source configuration remains the same.

Step 3: Run the data source and publish the data asset to the data mesh

After the Data source is created, run the Data source. The data source run adds the data.

Data source run configuration in Amazon SageMaker Unified Studio

Data source run results in Amazon SageMaker Unified Studio

During creation of data source if you choose Publishing settings such as assets are immediately discoverable, the Amazon Redshift tables and views appear in the catalog as Published, ready for discovery and subscription by data consumers.

Published Amazon Redshift tables and views in the Amazon SageMaker catalog

After the data sources are published, users can discover the published data and submit a subscription request. The data producer can approve or reject requests. After approval, users can consume the data by querying it in the Amazon Redshift query editor. The following screenshot illustrates data discovery in the Amazon SageMaker Unified Studio portal.

Data discovery view in the Amazon SageMaker Unified Studio portal

To complete the end-to-end test, you need to set up a consumer Amazon Redshift cluster in Account C.

Step 1: Setting up the consumer cluster

Follow the prerequisites from Steps 1 and 2 in the previous section, make sure the cluster and secret are properly tagged as in the following screenshots:
Amazon Redshift cluster tags:

Tags applied to the consumer Amazon Redshift cluster

Tags for the AWS Secrets Manager secret that stores the user credentials for the Amazon Redshift cluster:

Tags on the AWS Secrets Manager secret storing the consumer cluster credentials

Step 2: Connect the consumer cluster to the data mesh

Log into Amazon SageMaker Unified Studio and navigate to your consumer project.
In the Compute section of your project, choose Add compute, then choose Connect to existing compute resources.
Choose Amazon Redshift Provisioned.
Select your consumer Amazon Redshift cluster from the dropdown list and enter the Secrets Manager name.
Choose Add compute.
Your newly added Amazon Redshift cluster should now show as available.

Consumer Amazon Redshift cluster added as compute in Amazon SageMaker Unified Studio

The newly added Amazon Redshift cluster shows an Available state.

Consumer Amazon Redshift cluster showing an Available state

In the Data section you can see that objects (table/views) from Amazon Redshift cluster are visible and you can query them.

Data section showing Amazon Redshift tables and views available to query

Step 3: Creating a subscription target

Find the tooling environment ID: in your local terminal after obtaining correct credentials as a project member, run this command to find the tooling environment ID.

export REGION='<your-region>'
export SUBSCRIBER_PROJECT_ID='<your-project-id>'
export DOMAIN_ID='dzd-xxxxxxx'

aws datazone list-environments \
  --domain-identifier $DOMAIN_ID \
  --project-identifier $SUBSCRIBER_PROJECT_ID \
  --region $REGION

In the response, find and copy the tooling environment ID as shown in the following example.

{
    "items": [
        {
            "projectId": "<PROJECT_ID>",
            "id": "<ENVIRONMENT_ID>",
            "createdBy": "SYSTEM",
            "createdAt": "<TIMESTAMP>",
            "updatedAt": "<TIMESTAMP>",
            "name": "Tooling",
            "awsAccountId": "<AWS_ACCOUNT_ID>",
            "awsAccountRegion": "eu-west-1",
            "provider": "Amazon SageMaker",
            "status": "ACTIVE",
            "environmentConfigurationId": "<ENVIRONMENT_CONFIGURATION_ID>"
        }
    ]
}

Locate the Manage Access Role: In Account C, navigate to SageMaker Unified Studio and find the Tooling blueprint. In the Provisioning Tab you will find the Manage Access role and copy the value, as it is needed for the next CLI call.

Provisioning tab showing the Manage Access role for the Tooling blueprint

Create the Subscription Target.

With all the information collected, you can create the subscription target for the Amazon Redshift cluster as shown by the CLI call.

export TOOLING_ENV_ID='<tooling-environment-id>'
export AUTHORIZED_PRINCIPAL='datazone_env_<tooling-env-id>'
export MANAGE_ACCESS_ROLE='arn:aws:iam::<account-id>:role/service-role/AmazonSageMakerManageAccess-<domain-id>'

aws datazone create-subscription-target \
  --domain-identifier $DOMAIN_ID \
  --environment-identifier $TOOLING_ENV_ID \
  --name "RedshiftCluster-default-target" \
  --subscription-target-config '[{
    "formName": "RedshiftSubscriptionTargetConfigForm",
    "content": "{\"databaseName\":\"<db-name>\",\"secretManagerArn\":\"arn:aws:secretsmanager:<region>:<account>:secret:<secret-name>\",\"clusterIdentifier\":\"<cluster-id>\",\"schemaName\":\"<schema-name>"}"
  }]' \
  --applicable-asset-types RedshiftViewAssetType RedshiftTableAssetType \
  --manage-access-role $MANAGE_ACCESS_ROLE \
  --provider "Amazon SageMaker" \
  --type RedshiftSubscriptionTargetType \
  --authorized-principals $AUTHORIZED_PRINCIPAL

Using Amazon Redshift Serverless?

Use RedshiftServerlessSubscriptionTargetType as the --type and RedshiftServerlessSubscriptionTargetConfigForm as the formName in the subscription target config. Replace clusterIdentifier with workgroupName in the content JSON.

Verify the Subscription Target.

To verify that the subscription target was created successfully, make a last CLI call. You should find in the return a new subscription target with the name RedshiftCluster-default-target.

aws datazone list-subscription-targets \
  --environment-identifier $TOOLING_ENV_ID \
  --domain-identifier $DOMAIN_ID \
  --region $REGION

Step 4: Subscribing to data assets

Open the data catalog inside SageMaker Unified Studio and search for the assets you want to subscribe to.

Data catalog search for assets to subscribe to in Amazon SageMaker Unified Studio

Submit your subscription request
After approval, the subscribed assets appear as views in your consumer Amazon Redshift cluster.

Adding multiple databases and schemas

To publish assets from entirely different databases on the same Amazon Redshift cluster, you need to create a separate data source for each database, meaning repeating the steps mentioned in the section before. Each data source points to the same cluster connection but specifies a different database name. This approach gives you independent control over scheduling, publishing settings, and metadata generation for each database’s assets.

On the consumer side, each subscription target is bound to a specific database and schema combination. This is the target location where SageMaker Unified Studio will create views that give the consumer access to subscribed assets. To receive subscribed data in multiple databases or schemas, you create one subscription target per database-schema combination. For example, different teams within the consumer account might want the data materialized in their own schema. The following example shows this pattern:

# Subscription target for the sales schema
aws datazone create-subscription-target \
  --domain-identifier $DOMAIN_ID \
  --environment-identifier $TOOLING_ENV_ID \
  --name "RedshiftCluster-sales-target" \
  --subscription-target-config '[{ "formName": "RedshiftSubscriptionTargetConfigForm", "content": "{\"databaseName\":\"consumer_db\",\"secretManagerArn\":\"arn:aws:secretsmanager:<region>:<account>:secret:<secret-name>\",\"host\":\"<endpoint>\",\"port\":\"5439\",\"schemaName\":\"sales\"}" }]' \
  --applicable-asset-types RedshiftViewAssetType RedshiftTableAssetType \
  --manage-access-role $MANAGE_ACCESS_ROLE \
  --provider "Amazon SageMaker" \
  --type RedshiftSubscriptionTargetType \
  --authorized-principals $AUTHORIZED_PRINCIPAL

# Subscription target for the marketing schema
aws datazone create-subscription-target \
  --domain-identifier $DOMAIN_ID \
  --environment-identifier $TOOLING_ENV_ID \
  --name "RedshiftCluster-marketing-target" \
  --subscription-target-config '[{ "formName": "RedshiftSubscriptionTargetConfigForm", "content": "{\"databaseName\":\"consumer_db\",\"secretManagerArn\":\"arn:aws:secretsmanager:<region>:<account>:secret:<secret-name>\",\"host\":\"<endpoint>\",\"port\":\"5439\",\"schemaName\":\"marketing\"}" }]' \
  --applicable-asset-types RedshiftViewAssetType RedshiftTableAssetType \
  --manage-access-role $MANAGE_ACCESS_ROLE \
  --provider "Amazon SageMaker" \
  --type RedshiftSubscriptionTargetType \
  --authorized-principals $AUTHORIZED_PRINCIPAL

Verifying the audit trail

To substantiate the governance and traceability claims in this architecture, enable AWS CloudTrail in all three accounts with data events for Secrets Manager and KMS. Enable Amazon Redshift audit logging on clusters to capture connection and query activity through STL_CONNECTION_LOG and STL_QUERY. Subscription approvals and rejections are recorded by SageMaker Unified Studio and emitted to CloudTrail under the datazone.amazonaws.com event source. Look for CreateSubscriptionRequest, AcceptSubscriptionRequest, and RejectSubscriptionRequest events.

Clean up

If you deployed this solution for testing or evaluation purposes and no longer need the resources, we recommend cleaning up to avoid unnecessary costs. Amazon Redshift clusters, Secrets Manager secrets, and SageMaker Unified Studio projects all incur charges when left running. The following steps guide you through a structured teardown in the correct order: subscriptions first, then data assets, and finally the infrastructure itself. This order verifies that no orphaned resources remain.

Remove all subscriptions
- Revoke subscriptions from the data assets you have subscribed to in the consumer project.
Delete your data assets
- Remove the data assets you published on the producer side to avoid leaving unnecessary resources behind.
Delete the projects
- Delete the project within your SageMaker Unified Studio Domain after all subscriptions are removed. Make sure to delete both the consumer and producer projects.
Delete the SageMaker Unified Studio Domain in Account A.

Conclusion

In this post, we demonstrated how Amazon SageMaker Unified Studio simplifies cross-account data governance for Amazon Redshift. By implementing this solution, organizations can move away from ad-hoc, non-auditable data sharing processes to a secure, scalable, and fully governed approach. Amazon SageMaker Unified Studio serves as the central governance layer that data producers and consumers use to publish, discover, and subscribe to data products across AWS accounts. This turns a fragmented data landscape into a well-governed data mesh without the need for custom tooling or manual coordination.

With cross-account data sharing and governance in place, the natural next step is to use this well-governed data for machine learning and generative AI workloads. Because Amazon SageMaker Unified Studio brings together data, analytics, and AI capabilities in a single environment, teams can more efficiently transition from discovering and subscribing to data products to building ML models and generative AI applications, all within the same environment. This reduces the traditional friction between data engineering and data science, accelerating time to value. To get started with establishing your organization’s data mesh using Amazon SageMaker Unified Studio, follow the guidance for Setting up Amazon SageMaker Unified Studio.

About the authors

Prioritize your AWS Health alerts using AWS User Notifications

2026-07-16 Naga Bhargav

Post Syndicated from Naga Bhargav original https://aws.amazon.com/blogs/architecture/prioritize-your-aws-health-alerts-using-aws-user-notifications/

If you run critical workloads on AWS, such as a contact center on Amazon Connect Customer, database workloads on Amazon Relational Database Service (Amazon RDS), or hybrid connectivity through AWS Direct Connect, service health events demand your attention. But not all events are equal. An operational issue, a scheduled maintenance window, and a deprecation notice buried in your inbox have very different consequences. The problem is that they all arrive through the same channel, making their urgency difficult to determine.

AWS Health generates events for every service, every account, every Region. The service delivers ongoing issues, scheduled changes, account notifications, and deprecation notices in one undifferentiated stream. For operations teams, this creates a familiar problem: either you treat every notification as urgent with unwanted triage noise, or you start ignoring them and risk missing something that matters. Both paths lead to slower response times and unwanted escalations.

This post walks you through a lightweight approach to solving this problem using AWS User Notifications, a fully managed service for routing AWS events to your preferred delivery channels. This solution filters health events to only the services you want to be notified about, then separates what remains into two priority tiers. Critical events arrive immediately. Informational events arrive as batched summaries. In this post, we address this problem with a single AWS CloudFormation template with four deployment approaches that you can deploy in your AWS environment.

Solution overview

The design follows a simple principle: filter first, then separate by priority.

The first layer filters out noise. Event rules match health events only for the services your organization depends on, such as AWS Direct Connect, Amazon Connect Customer, and Amazon RDS. Everything else is silenced before it reaches your inbox.

The second layer separates what remains by urgency. Two notification configurations handle different priority tiers:

CRITICAL — Matches events where eventTypeCategory is issue or scheduledChange. These arrive immediately as individual notifications with no batching.
INFORMATIONAL — Matches everything else using an anything-but filter such as accountNotification. AWS User Notifications batches these within a five-minute window and delivers them as grouped summaries.

In this solution, a CloudFormation template supports four deployment modes through a DeploymentMode parameter:

Mode	Scope	What you get
Linked (default)	Single account	Email contacts + User Notifications event rules + channel associations
Payer	Entire organization or OU	Everything in Linked, plus organizational unit associations scoped to a root or OU
Combined	Single account	Everything in Linked, plus Amazon EventBridge rules and an Amazon Simple Notification Service (Amazon SNS) topic with [CRITICAL]/[INFORMATIONAL] prefixed custom email
PayerCombined	Entire organization or OU	Everything in Linked, plus org associations AND Amazon EventBridge rules with SNS custom email messages

The following diagram shows how health events flow through the solution:

Architecture diagram showing priority-based AWS Health alerting using AWS User Notifications

Figure 1: Architecture diagram showing priority-based AWS Health alerting using AWS User Notifications

What gets deployed

Once you deploy the CloudFormation stack, AWS provisions the following resources:

Prioritized AWS services — AWS Direct Connect, Amazon Connect Customer, and Amazon RDS are pre-configured as the monitored services. You can customize this list directly in the CloudFormation template parameters.
Two notification configurations on AWS User Notifications — one scoped for CRITICAL events (service issues and scheduled changes) and one for INFORMATIONAL events (account notifications), ensuring targeted alerting.
Email delivery channel — AWS automatically links both notification configurations to the email address you provide during stack deployment, so alerts reach the right contacts from day one.

How the notification flow works

User Notifications path (all deployment modes)

AWS Health emits an event and lands on the default Amazon EventBridge event bus.
User Notifications event rule filters by service + category → two priority tiers.
Notification configuration routes: Critical = immediate, Informational = 5-min batch.
Email contact receives AWS-standard formatted notification.

Amazon EventBridge + SNS path (Combined and PayerCombined modes only)

In parallel with the above, a second delivery path activates:

Same AWS Health events land on the default Amazon EventBridge event bus.
Custom Amazon EventBridge rules (deployed by the template) evaluate the events on the default event bus and filter by the same service and category criteria as the AWS User Notifications event rules.
InputTransformer reformats the event into a human-readable message with [CRITICAL] and [INFORMATIONAL] prefix.
Amazon SNS delivers the custom formatted email to all subscribers via the Amazon SNS topic.
Failed deliveries route to an Amazon Simple Queue Service dead letter queue and an Amazon CloudWatch alarm triggers if an Amazon SNS delivery fails.

Prerequisites

To follow along, you need:

An active AWS account.
Permissions to deploy AWS CloudFormation stacks and create AWS User Notifications resources.
For organization-wide deployment: access to the management (payer) account and the organization root ID or organizational unit (OU) ID.
(Optional) The AWS Command Line Interface (AWS CLI), installed and configured, for CLI-based deployment.

Deployment walkthrough

This section walks you through deploying, verifying, and testing the solution. Choose one of the four deployment modes based on your scope, then follow the remaining steps to confirm everything works.

Step 1: Deploy the AWS CloudFormation stack

Download and deploy the complete solution through this sample CloudFormation template

Select the deployment option that matches your requirements:

Option A: Single account (Linked mode)

Deploy using the AWS CLI:

aws cloudformation deploy \
    --template-file prioritize-aws-health-notifications.yaml \
    --stack-name prioritize-aws-health-notifications \
    --parameter-overrides \
    DeploymentMode=Linked \
    [email protected] \
    NotificationRegions=us-east-1,us-west-2
    NotificationHubAlreadyEnabled=No

Note : Set NotificationHubAlreadyEnabled=Yes if your AWS account already has a notification hub enabled in AWS User Notifications.

Or deploy through the AWS CloudFormation console:

Open the CloudFormation console and choose Create stack.
Upload the prioritize-aws-health-notifications.yaml template file.
For Stack name, enter health-notifications.
For DeploymentMode, select Linked.
For NotificationEmail, enter the email address for notifications.
For NotificationRegions, enter the Regions to monitor (comma-separated)
For NotificationHubAlreadyEnabled, select Yes/No
Choose Submit.

Option B: Organization-wide (Payer mode)

Before deploying, run the following command from the payer account to grant the AWS Health service access to your organization:

aws health enable-health-service-access-for-organization

Then deploy:

aws cloudformation deploy \
    --template-file prioritize-aws-health-notifications.yaml \
    --stack-name prioritize-aws-health-notifications-org \
    --parameter-overrides \
    DeploymentMode=Payer \
    [email protected] \
    NotificationRegions=us-east-1,us-west-2 \
    NotificationHubAlreadyEnabled=No
    OrgRootId=r-xxxx

Replace r-xxxx with your organization root ID to cover all accounts, or use an OU ID (for example, ou-xxxx-xxxxxxxx) to scope coverage to a specific unit.

aws cloudformation deploy \
    --template-file prioritize-aws-health-notifications.yaml \
    --stack-name prioritize-aws-health-notifications-combined \
    --parameter-overrides \
    DeploymentMode=Combined \
    [email protected] \
    NotificationRegions=us-east-1,us-west-2
    NotificationHubAlreadyEnabled=No

aws cloudformation deploy \
    --template-file prioritize-aws-health-notifications.yaml \
    --stack-name prioritize-aws-health-notifications-full \
    --parameter-overrides \
    DeploymentMode=PayerCombined \
    [email protected] \
    NotificationRegions=us-east-1,us-west-2 \
    NotificationHubAlreadyEnabled=No
    OrgRootId=r-xxxx


Expected result: Stack reaches CREATE_COMPLETE in 2–3 minutes.

CloudFormation console showing CREATE_COMPLETE status

Figure 2: CloudFormation console showing CREATE_COMPLETE status.

Step 2: Confirm the email subscription

After the stack deploys, check the email inbox you specified during deployment. You will receive a subscription confirmation from AWS User Notifications.

Open the confirmation email.
Choose Confirm subscription.

Important: Notifications will not be delivered until you confirm the email contact.

Expected result: The email contact shows Verified in the AWS User Notifications console

AWS User Notifications console showing verified email contact

Figure 3: AWS User Notifications console showing verified email contact.

Step 3: Verify the notification configurations

Open the AWS User Notifications console and confirm the following resources were created:

Navigate to Notification configurations — you should see two entries:.
- Health-Critical-Notifications — scoped to issue and scheduledChange event types.
- Health-Informational-Notifications — matches all event categories except issue and scheduledChange using an anything-but filter.
Choose each configuration and verify:.
- Event rules list your selected services (AWS Direct Connect, Amazon Connect Customer, Amazon RDS).
- Delivery channels show your confirmed email contact.

Expected result: Two notification configurations visible, each with event rules matching your monitored services and the email channel associated.

AWS User Notifications console showing two notification configurations

Figure 4: AWS User Notifications console showing two notification configurations.

Step 4: Test the solution

Validate the deployed resources via the AWS CLI:

aws notifications list-notification-configurations

Expected result: Returns two configurations with their ARNs and aggregation settings — CRITICAL with no aggregation (NONE) and INFORMATIONAL with a 5-minute aggregation window (SHORT).

To verify end-to-end delivery, check the AWS Health Dashboard for any active events in your monitored Regions. When a matching event occurs:

CRITICAL (issue or scheduled change): Email arrives immediately with event details, affected resources, and recommended actions.
INFORMATIONAL (account notification): Email arrives as a grouped summary within 5 minutes.

Expected result: Email notification received with the correct delivery pattern — standalone for critical, batched for informational.

What you receive

The pattern is simple: a standalone email means something needs attention now. A batched summary means routine updates you can review on your own schedule. The email format is controlled by AWS User Notifications and cannot be customized. The priority distinction comes from the delivery pattern, not from text labels in the email body.

For teams using AWS Chatbot in chat applications (Slack or Microsoft Teams) or the console Notification Center, the configuration names [CRITICAL] and [INFORMATIONAL] appear directly in the notification, providing explicit priority context.

Sample CRITICAL email notification from AWS User Notifications

Figure 5: Sample CRITICAL email notification from AWS User Notifications related to an ISSUE.

Sample INFORMATIONAL digest email from AWS User Notifications

Figure 6: Sample INFORMATIONAL digest email from AWS User Notifications related to an accountNotification.

Customizing the solution

You can tailor the solution to your environment by adjusting which services are monitored and which Regions are covered.

Adding or removing monitored services

This template monitors AWS Direct Connect, Amazon Connect Customer, and Amazon RDS by default. To monitor additional services, update the service array in the EventPattern of both event rules. For example, to add Amazon Elastic Compute Cloud (Amazon EC2):

"service": ["DIRECTCONNECT", "CONNECT", "RDS", "EC2"]

Update the stack, and the new services are covered immediately.

Multi-Region monitoring

To get notifications about other AWS Regions, pass multiple Regions in the NotificationRegions parameter:

NotificationRegions=us-east-1,us-west-2,eu-west-1

Always include us-east-1 regardless of where your workloads run. AWS Health global events — such as those for AWS Identity and Access Management (IAM), Amazon Route 53, and Amazon CloudFront — are delivered to us-east-1. If you exclude it, you miss global events.

Adding delivery channels

The solution starts with an email, but you can extend it without modifying the core event rules or notification configurations:

More email recipients: Create additional EmailContact resources and associate them with the existing CRITICAL and INFORMATIONAL configurations.
Slack or Microsoft Teams: Set up an AWS Chatbot in chat applications channel and create a ChannelAssociation linking it to the notification configurations.
Mobile push: Install the AWS Console Mobile App and sign in. User Notifications delivers to the mobile app automatically — no additional CloudFormation resources needed.
Team-based routing: Associate the network team’s email only with the CRITICAL configuration, and the general ops team with both CRITICAL and INFORMATIONAL. This is done through channel associations alone — no changes to event rules.

How this solution compares to existing approaches

Several tools exist for routing AWS Health events, each designed for different operational needs. This solution is not a replacement for all of them — it fills a specific gap.

Approach	What it does	Trade-offs
AWS Health Aware (AHA)	Open-source framework with Lambda, DynamoDB, Secrets Manager. Supports Slack, Teams, Chime, and email with event deduplication.	Requires Business or Enterprise Support plan. Ongoing maintenance of deployed components.
HEIDI / CID Health Events Dashboard	Historical analysis and trend visualization using Amazon QuickSight , Amazon Athena , and Amazon S3 .	Designed for operational planning and post-incident review — not real-time alerting. Requires Business or Enterprise Support plan.
Custom Amazon EventBridge + Lambda + SNS	Full flexibility for routing and transformation.	Requires writing, testing, and maintaining application code.
Third-party tools (PagerDuty, Datadog)	Escalation, on-call routing, and acknowledgment workflows.	Licensing costs and vendor dependencies.
This solution	Simplest path to priority-separated, real-time health alerting. One stack, no code, no compute, no support plan requirement.	No deduplication, no escalation/acknowledgment, no historical storage.

The approach in this post sits at a different point on the spectrum. It works well as a standalone solution for teams that need straightforward alerting, and it works equally well as a foundation layer that feeds into more advanced tools as operational needs grow.

You can start with this solution for immediate coverage, then consider adding PagerDuty by subscribing it to the Amazon SNS topic (Combined mode) for escalation and on-call routing, or pair it with HEIDI for historical trend analysis.

Things to consider

This solution is intentionally lightweight, and that comes with trade-offs worth understanding:

Email format: AWS User Notifications controls the email body and subject line. You cannot add custom text like ‘[CRITICAL]’ to the email itself by default. The priority signal is the delivery pattern — standalone means critical, batched means informational. For teams that need explicit priority labels in email, the Combined deployment mode adds an Amazon EventBridge + SNS layer with InputTransformer that prefixes the email body with ‘[CRITICAL]’ or ‘[INFORMATIONAL]’.
Delivery monitoring (Combined modes): The Amazon EventBridge + SNS layer includes built-in reliability. A dead letter queue (DLQ) retains failed deliveries for 14 days for troubleshooting, and a CloudWatch alarm fires if SNS fails to deliver notifications. This means you are alerted not just about AWS Health issues, but also about failures in the notification pipeline itself.
No deduplication: AWS Health events have a lifecycle — created, updated, resolved. Each update triggers a new notification. A single incident might generate 2–4 emails as the event progresses. For strict deduplication, consider pairing with AHA or adding a lightweight Lambda function.
No escalation or acknowledgment: This solution sends notifications but does not track whether anyone acted on them. For on-call routing and escalation chains, integrate with an incident management tool like PagerDuty or OpsGenie via the SNS topic.
No historical storage: Notifications are delivered in real time but not stored for later analysis. For post-incident review and trend reporting, pair with HEIDI or the CID Health Events Dashboard.

The advantage of this approach is that it does not lock you into a single path. The notification configurations and event rules remain in place as you layer on additional capabilities.

Cleanup

If you no longer need the health notification resources, delete the CloudFormation stack:

aws cloudformation delete-stack --stack-name prioritize-aws-health-notifications

Note: AWS CloudFormation preserves resources with DeletionPolicy: Retain (notification configurations, event rules, email contacts, and channel associations) after you delete the stack. To fully remove them, delete the resources manually through the AWS User Notifications console or the AWS CLI.

Expected result: Stack reaches DELETE_COMPLETE within 2–3 minutes.

Conclusion

In this post, we walked through how to set up priority-based AWS Health alerting using AWS User Notifications and a single CloudFormation template. The solution filters health events to only the services that matter to your organization, then separates what remains into immediate critical alerts and batched informational summaries.

The core value is simplicity. No Lambda functions to patch. No DynamoDB tables to manage. No code to maintain. One stack, deployed in minutes, covering a single account or an entire organization. Because it uses only native AWS services with no support plan requirement, any team can adopt it regardless of their current tooling or support tier.

This approach works as a standalone alerting solution. It also works as a starting point that you can extend with Slack and Microsoft Teams integration through AWS Chatbot in chat applications, escalation workflows through PagerDuty or OpsGenie, and historical analysis through HEIDI or CID.

To get started, download the CloudFormation templates from the GitHub repository. For more information, see the AWS User Notifications User Guide and the AWS Health User Guide.

If you have questions or want help implementing this solution for your organization, contact your AWS account team or visit the AWS Contact Us page.