Tag Archives: Technical How-to

Using Amazon Mail Manager SMTP to send email via Amazon Simple Email Service

Post Syndicated from Josephine Elea Schlage original https://aws.amazon.com/blogs/messaging-and-targeting/using-amazon-mail-manager-smtp-to-send-email-using-amazon-simple-email-service/

If you’re running applications or mail servers that need to send email over Simple Mail Transfer Protocol (SMTP), you may find that the classic Amazon Simple Email Service (Amazon SES) SMTP endpoint (email-smtp.<region>.amazonaws.com) is not available in every AWS Region.

This applies to some newer AWS Regions and partitions, including eusc-de-east-1 in the AWS European Sovereign Cloud (ESC). In these AWS Regions, services configured with a traditional SMTP hostname and credentials, such as Postfix relays, cannot use the classic SES SMTP integration pattern. Amazon SES Mail Manager provides an alternative: an authenticated SMTP ingress endpoint that accepts connections using a hostname, port, and credentials, just like any standard SMTP server.

In addition to SMTP connectivity, Mail Manager introduces a configurable email pipeline between acceptance and delivery. This pipeline gives you traffic filtering, message archiving, and rule-based routing that are not available with the classic SES SMTP endpoint.

In this post, you configure Amazon SES Mail Manager to send outbound email in a Region that does not offer the classic SES SMTP endpoint. This post uses eusc-de-east-1 (AWS European Sovereign Cloud) as an example, but the same steps apply to AWS Regions where Mail Manager is available and the classic SMTP endpoint is not. By the end, you have a working Mail Manager pipeline that can:

  • Control outbound email flow with traffic policies.
  • Archive outgoing messages for compliance.
  • Deliver messages to recipients through a managed SMTP pipeline.

This post walks through a practical setup in eusc-de-east-1 with step-by-step instructions for configuring each component.

Solution overview

In this walkthrough, you configure Amazon SES Mail Manager in eusc-de-east-1 with the following components:

  • Traffic policy: You create a traffic policy with a default action set to Deny. The policy includes two policy statements connected by an OR condition. Policy Statement 1 allows messages that use TLS protocol version 1.2 or higher. Policy Statement 2 allows messages where the recipient address ends with a specific domain, filtering outgoing mail to approved recipients only.
  • Rule set: You create a rule set containing a single rule with two actions that archive outgoing email and then deliver it to recipients.
  • Ingress endpoint: You create an authenticated Mail Manager ingress endpoint that receives, routes, and manages messages based on your configured traffic policy and rule set.

After setting up these components, you use sample Python code to send an email through the ingress endpoint. Optionally, you can integrate with Postfix for relay-based delivery. You also configure Amazon CloudWatch logging to monitor how each message flows through the pipeline. To verify functionality, you check the email archive to confirm that outgoing messages are stored and that the email is received in the intended inbox.

The following diagram shows the message flow: Application or Amazon Elastic Compute Cloud (Amazon EC2) instance → ingress endpoint → traffic policy (allow or deny) → rule set (archive, then send to internet) → recipient inbox.

Walkthrough

This walkthrough covers the prerequisites and the step-by-step setup. Before you create traffic policies and a rule set, you first set up email archiving and AWS Identity and Access Management (IAM) roles, which are needed when you create the traffic policies and rules.

Prerequisites

Before beginning, verify that you have completed domain verification in the eusc-de-east-1 (ESC) Region and moved out of the Amazon SES sandbox. Domain verification is a required first step that confirms your authority to send email through SES from your domain. In this tutorial, you use a sample Python program to send email programmatically through an ingress SMTP endpoint (ARecord). You can run this program on your local machine through the AWS Command Line Interface (AWS CLI).

  • An active AWS account in the AWS European Sovereign Cloud with access to the eusc-de-east-1 Region.
  • A domain to verify as a sending identity in Amazon SES.
  • The AWS CLI, installed and configured for eusc-de-east-1 (required for Amazon CloudWatch logging).
  • An AWS Secrets Manager secret to store ingress endpoint credentials.
  • (Optional) An Amazon Virtual Private Cloud (Amazon VPC) with at least two subnets and an Amazon EC2 instance, if you plan to configure VPC endpoint connectivity.
  • IAM permissions for Amazon SES, AWS Key Management Service (AWS KMS), AWS Secrets Manager, and CloudWatch for the user who is signed in to the AWS Management Console.

Step 1: Create and verify an identity

To create and verify a sending identity in Amazon SES:

  1. In the Amazon SES console, choose Configuration, and then select Identities.
  2. Create the identity (domain or email address). If you verify a domain identity, configure email authentication with Sender Policy Framework (SPF), DomainKeys Identified Mail (DKIM), and Domain-based Message Authentication and Reporting and Conformance (DMARC) to prevent email from being marked as spam or failing delivery. See the following guides:
    1. Authenticating Email with DKIM in Amazon SES.
    2. Authenticating Email with SPF in Amazon SES.
    3. Complying with DMARC authentication protocol in Amazon SES.

    If you verify an email address identity without also verifying the parent domain, your messages may be quarantined or rejected depending on the domain’s DMARC policy.

  3. Complete the verification process.

Note: For eusc-de-east-1, the Custom MAIL FROM Domain Name System (DNS) records use amazonses.eu instead of amazonses.com.

Step 2: Configure an email archive for compliance and retention

Create an email archive to store outgoing messages. You configure this archive as the first action in your rule. The archive serves as a repository for outgoing messages.

  1. In the Amazon SES console, choose Mail Manager, then Email Archiving.
  2. Under Manage archives, select Create archive.
    1. Enter a unique name in the Archive name field.
    2. (Optional) Select a retention period to override the default of 180 days (6 months).
    3. (Optional) Set up encryption by either entering your own AWS Key Management Service (AWS KMS) key in the AWS KMS key ARN field, or selecting Create new key.
  3. Choose Create archive.
  4. After it is created, this archive stores your email according to the rules you define in the next step.

Step 3: Create an IAM role permission policy for the send to internet rule action

Configure an IAM role that permits Mail Manager to send email to external domains. This role is referenced in the rule for the second action, “send to internet,” which delivers email to recipients.

  1. Go to the IAM console.
  2. Choose Roles, and then choose Create role.
  3. For trusted entity, select Custom trust policy and paste the following (replace XXXXXXXXXXX with your AWS EUSC account ID): 
    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Statement1",
      "Effect": "Allow",
      "Principal": {
        "Service": "ses.amazonaws.com"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "XXXXXXXXXXX"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws-eusc:ses:eusc-de-east-1:XXXXXXXX:mailmanager-rule-set/*"
        }
      }
    }
  ]
}

  4. Skip add permissions, name review, and create your role.
  5. Open your newly created role and select Add permissions.
  6. From the menu, choose Create inline policy.
  7. Select JSON in the policy editor and paste the following (replace example.com with your verified domain, XXXXXXXXXXX with your AWS account ID, and my-configuration-set with your configuration set name if applicable). This policy grants the necessary permissions to send email to recipients on the internet, which is used in rule 2 of your rule set. 
    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "ses:SendEmail",
        "ses:SendRawEmail"
      ],
      "Resource": [
        "arn:aws-eusc:ses:eusc-de-east-1:XXXXXXXXXXX:identity/example.com",
        "arn:aws-eusc:ses:eusc-de-east-1:XXXXXXXXXXX:configuration-set/my-configuration-set"
      ],
      "Condition": {
        "StringEquals": {
          "ses:FromAddress": "example.com"
        }
      }
    }
  ]
}

  8. Review and save the policy.

Your newly created role now has the custom trust policy in Trusted entities, and a customer-managed inline permission policy under Permissions.

Step 4: Create a traffic policy

Traffic policies act as security checkpoints for your email infrastructure. They control which messages can enter your system based on rules you define. To create a traffic policy that enforces security requirements for your email:

  1. Open the Amazon SES console.
  2. Go to Mail Manager and select Traffic policies.
  3. Choose Create traffic policy.
  4. Enter a unique name for your policy.
  5. Set Default action to Deny.
  6. In your traffic policy, select “add new policy statement.”
    1. For Allow or deny properties, select Allow.
    2. For Properties, select TLS protocol version.
    3. For Operator, select Minimum version or Is version.
    4. For Value, select TLS 1.2 (minimum) or TLS 1.3 (Is version).
  7. Now, add a second condition to the same policy statement to filter outgoing mail to *example.com domains:
    1. For Properties, select Recipient address.
    2. For Operator, select “Ends with” and for Value enter example.com.

    Configure your policy statements as you like.

  8. Choose Create traffic policy.

Traffic policies are evaluated in a specific sequence:

  1. Deny policy statements are evaluated in order. If any match, the email is immediately blocked and no further evaluation occurs.
  2. If no Deny statements match, all Allow policy statements are evaluated in order. Multiple statements within a policy are connected by OR logic. If any statement matches, the email is allowed.
  3. Within each individual policy statement, multiple conditions are connected by AND logic. Each condition must be true for the statement to match.
  4. If no policy statements match (neither Deny nor Allow), the default action of the traffic policy (either Allow or Deny) is applied.

This policy denies traffic by default and allows only messages that meet the TLS 1.2 minimum requirement and are addressed to approved recipient domains.

Default action: Deny by default. Email traffic is initially blocked unless explicitly allowed by the following policy statements.

Policy statement 1: Allows messages to be sent if the recipient’s address ends with *example.com AND meets the minimum TLS protocol version of TLS 1.2.

Step 5: Create a rule set

Rule sets define how your messages are processed after they pass through your traffic policy. In this example, the rule set establishes a sequential email processing workflow. First, you add the action for archiving outgoing messages, and then you add a second action to deliver messages to recipients.

To create a rule set:

  1. Open the Amazon SES console.
  2. Go to Mail Manager and select Rule sets.
  3. Choose Create rule set.
  4. Enter a unique name for your rule set.
  5. On the rule set’s overview page, select Edit, then select Create new rule.

Step 6: Create rules

In this step, you create rules within your rule set that define the actions performed on each email: archiving for compliance and delivering to recipients.

Email add-ons are optional: In your rule set, you can configure the Vade Advanced Email Security Add On for scanning or dropping messages, archiving for compliance, writing to Amazon Simple Storage Service (Amazon S3) for future analysis, and sending email out. Configure these rules accordingly. This guide covers email sending and archiving in the rule below.

  • Add conditions or exceptions as needed:
    • Select Add new condition to specify what messages the rule applies to.
    • Select EXCEPT in the case of and select Add new exception for exclusions.
  • Configure actions by choosing Add new action.
  • For multiple actions, use the up and down arrows to set the execution order.

Action 1: Archive outgoing email. Stores a copy of each outgoing email in a Mail Manager archive. Archived email can be searched and retrieved directly from the Amazon SES console under Email archiving, supporting compliance and audit requirements.

Action 2: Send to internet. Delivers the email to the intended recipient using Amazon SES.

After you create your rule set, add rules that define how email is processed. You create a rule set containing a single rule with two actions that execute in sequential order.

Follow these steps to create and configure your rules.

  1. In the created rule set’s overview page, select Edit, then choose Create new rule.
  2. In the Rule details sidebar, enter a unique name for your rule.
    1. In the rule details on the right side, select “add new action.”
    2. From the menu, choose “archive,” and choose the archive you created at Step 2.
    3. Then add another action: select “add new action” and from the menu, choose “Send to internet.”
    4. Choose the IAM role that you created in Step 3. This role grants SES Mail Manager access to your resource.
  3. When finished creating your rules, choose Save rule set to apply your changes.

Rule 1: Archive and send email to recipients

The rule processes messages that have successfully passed through the traffic policy. The archive action confirms that messages are archived and searchable. The send to internet action then forwards messages to their intended recipients, completing the email delivery workflow.

Step 7: Store password in AWS Secrets Manager for the ingress endpoint

Before you create an ingress endpoint, set up a password in AWS Secrets Manager and an AWS KMS customer managed key:

1. Create a customer managed key policy for your ingress endpoint.

  1. Open the AWS KMS console.
  2. Select Customer managed key (not AWS managed keys).
  3. Create the key.
  4. Define key administrative permissions.
  5. Define key usage permissions.
  6. In your key policy editor, when you review the key statements, paste the following (replace XXXXXXXXXXX with your AWS account ID): 
    {
  "Sid": "Allow use of the key",
  "Effect": "Allow",
  "Principal": {
    "Service": "ses.amazonaws.com"
  },
  "Action": "kms:Decrypt",
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "aws:SourceAccount": "XXXXXXX",
      "kms:ViaService": "secretsmanager.eusc-de-east-1.amazonaws.com"
    },
    "ArnLike": {
      "aws:SourceArn": "arn:aws-eusc:ses:eusc-de-east-1:XXXXXXXX:mailmanager-ingress-point/*"
    }
  }
}

2. Set up a password in AWS Secrets Manager.

  1. Go to the AWS Secrets Manager console and select Store a new secret.
  2. Choose Other type of secret.
  3. Enter password as the key and your chosen password as the value.
  4. For encryption key, choose the customer managed key you created above.
  5. Choose Next to proceed to Configure secret.
  6. Enter a secret name and choose Edit permissions, then update the resource policy (replace XXXXXXXXXXX with your AWS account ID). 
    {
  "Version": "2012-10-17",
  "Id": "Id",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ses.amazonaws.com"
      },
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "aws:SourceAccount": "XXXXXXXXXXX"
        },
        "ArnLike": {
          "aws:SourceArn": "arn:aws-eusc:ses:eusc-de-east-1:XXXXXXXXXXX:mailmanager-ingress-point/*"
        }
      }
    }
  ]
}

  7. Choose Next, then create and store your secret.

Step 8: Create an authenticated ingress endpoint (ARecord)

Now that you have created your traffic policy and rule set and stored your credentials, you can create the ingress endpoint:

  1. In the Amazon SES console, choose Mail Manager and then select Ingress endpoints.
  2. Choose Create ingress endpoint.
  3. Configure your endpoint:
    1. For type, select authenticated, then select the Secret ARN you created in Secrets Manager.
    2. Choose the traffic policy you created earlier.
    3. Choose the rule set you created earlier.
    4. Configure Network Type: Public Network (Standard Setup). If you select Private Network, follow Step 9 first in a new tab.
    5. Enter a unique name for your endpoint.
  4. Choose Create ingress endpoint.

After your ingress endpoint is created, note the following details from the General details section:

  • Amazon Resource Name (ARN): arn:aws-eusc:ses:eusc-de-east-1:XXXXXXXXXXX:mailmanager-ingress-point/inp-XXXXX
  • Username: inp-XXXXXXXXXXX
  • Host: XXXXXXXXXXX.mail-manager-smtp.eusc-de-east-1.amazonaws.eu (ARecord)

You need these details when configuring your email client or application to send email through this endpoint.

Step 9: Configure VPC endpoint for SES Mail Manager (optional enhanced security)

A VPC endpoint allows your Postfix EC2 instance to reach Mail Manager privately, without sending traffic over the public internet. To use this option, create the VPC endpoint in the same VPC as your Postfix instance. Configure security group rules to allow traffic on port 587.

  • VPC: The VPC endpoint must be created in the same VPC where your Postfix EC2 instance resides.
  • Security groups:
    • Postfix EC2 SG: Outbound rule to VPC endpoint SG on port 587.
    • VPC endpoint SG: Inbound rule from Postfix EC2 SG on port 587.
  • Subnets: The VPC endpoint should be in subnets that are routable from your EC2 instance’s subnet.

Create a security group for the VPC endpoint:

  1. Open the Amazon VPC console.
  2. Select Security groups.
  3. Choose Create security group.
    1. Name: mail-manager-vpce-sg (example).
    2. VPC: Choose the VPC where your Postfix EC2 instance resides.
  4. Add an inbound rule:
    1. Type: Custom TCP.
    2. Port: 587 (or 25 if using port 25).
    3. Source: Security Group ID of your Postfix EC2 instance (or create a placeholder, update later).
  5. Choose Create security group. Note the Security Group ID for the next step.
  6. Choose Endpoints in the VPC console.
  7. Choose Create endpoint.
    1. Name: mailmanager-ingress-endpoint (example).
    2. For Service category, select AWS services.
    3. For Service Name, select com.amazonaws.eusc-de-east-1.mail-manager-smtp.auth.
    4. For VPC, choose the VPC where your Postfix server resides.
    5. Subnets: Select at least 2 (private) subnets (for high availability).
    6. Security Groups: Choose the security group you created.
  8. Choose Create Endpoint.

Wait for the endpoint status to become Available. After the endpoint status becomes Available, note the DNS name from the endpoint details. Use the regional (non-AZ-specific) DNS name for your Postfix relay configuration:

auth.mail-manager-smtp.eusc-de-east-1.on.amazonwebservices.eu

Step 10: Mail Manager logging (AWS CLI)

Now that you have created your Mail Manager resources, you can configure log delivery through the AWS CLI to track message flow from ingress endpoints through rule set processing. After it is configured, you can view these logs in CloudWatch Log Groups.

  1. Open your terminal (CLI).
  2. Log in to your AWS account using the following command: 
    aws configure

  3. Create a CloudWatch Log Group: 
    aws logs create-log-group \
    --log-group-name /aws/mailmanager/ruleset-logs \
    --region eusc-de-east-1

    Before you proceed, copy the log group ARN for the step Create the Delivery Destination.

  4. Create the Log Delivery Source:Add your rule set ID to the resource-arn parameter below. You can find the resource ARN for the rule set when you click on the rule set name under rule sets in the SES console. 
    aws logs put-delivery-source \
    --name rs-default \
    --resource-arn arn:aws-eusc:ses:eusc-de-east-1:XXXXXX:mailmanager-ruleset/YOUR-RULESET-ID \
    --log-type APPLICATION_LOGS

  5. Create the Log Delivery Destination:Add your log group ARN to the destinationResourceArn parameter below: 
    aws logs put-delivery-destination \
    --name mailmanager-destination \
    --output-format json \
    --delivery-destination-configuration '{"destinationResourceArn":"arn:aws-eusc:logs:eusc-de-east-1:XXXXXX:log-group:/aws/mailmanager/ruleset-logs:*"}'

    Copy the delivery destination ARN for the step below.

  6. Link Log Delivery Source to Log Delivery Destination (Create Delivery): 
    aws logs create-delivery \
    --delivery-source-name rs-default \
    --delivery-destination-arn arn:aws-eusc:logs:eusc-de-east-1:XXXXXX:delivery-destination:mailmanager-destination

    Verification commands:

    aws logs describe-log-groups --region eusc-de-east-1
aws logs describe-delivery-sources --region eusc-de-east-1
aws logs describe-delivery-destinations --region eusc-de-east-1
aws logs describe-deliveries --region eusc-de-east-1

  7. Send an email and view your logs for your rule set:
    1. Open the Amazon CloudWatch console.
    2. Select Log Groups in the sidebar navigation.
    3. Select the log group you would like to view logs for.
    4. Select the log you would like to view under Log Streams.

Example output of an email that was sent successfully:

{
  "resource_arn": "arn:aws-eusc:ses:eusc-de-east-1:account-id:mailmanager-rule-set/ruleset-id",
  "event_timestamp": 3456789876,
  "message_id": "message-id",
  "rule_set_name": "send",
  "rule_name": "sendtointernet",
  "rule_index": 1,
  "recipients_matched": "[\"[email protected]\"]",
  "action_metadata": {
    "action_name": "SEND",
    "action_index": 1,
    "action_status": "SUCCESS"
  }
}

Step 11: Send email using an ingress endpoint

Code example with Python:

import smtplib
import ssl

# Your ingress endpoint and port
smtp_server = "*****.eusc-de-east-1.amazonaws.eu"
# Or for VPC: "vpce-xxxxx.mail-manager-smtp.auth.eusc-de-east-1.vpce.amazonaws.eu"
smtp_port = 587

# Your SMTP credentials retrieved from Secrets Manager
username = "****"
password = "[REDACTED_PASSWORD]"

sender_email = "[email protected]"  # Your verified identity
receiver_email = "[email protected]"

# Properly formatted email message with headers
message = f"""From: Firstname Lastname <{sender_email}>
To: Firstname Lastname <{receiver_email}>
Subject: Test Email from Python

This email was sent via the Mail Manager ingress endpoint and delivered
to the recipient through the "Send to Internet" rule action.
"""

server = None
try:
    print(f"Connecting to {smtp_server}:{smtp_port}...")
    server = smtplib.SMTP(smtp_server, smtp_port)
    server.set_debuglevel(1)

    print("\nStarting TLS...")
    context = ssl.create_default_context()
    server.starttls(context=context)

    print("\nLogging in...")
    server.login(username, password)

    print("\nSending email...")
    server.sendmail(sender_email, receiver_email, message)
    print("\nEmail sent successfully.")
except Exception as e:
    print(f"\nError: {e}")
    import traceback
    traceback.print_exc()
finally:
    if server:
        server.quit()

Step 12: Integrate with your existing email server

Use Postfix or SMTP clients on Amazon EC2 to relay outbound email through Mail Manager, which then forwards it through the “Send to internet” action configured in Step 3.

If you choose to integrate with Postfix in this guide, your relay host is the ingress endpoint or the VPC endpoint you created. Your port is typically 587.

relayhost = [<ARecord>]:<port>

Example with Postfix

/etc/postfix/main.cf:

relayhost = [xxxx.eusc-de-east-1.amazonaws.eu]:587 or 25
relayhost = [vpce-xxxxx.mail-manager-smtp.auth.eusc-de-east-1.vpce.amazonaws.eu]:587

Clean up

Clean up your AWS environment by removing all resources created during this walkthrough, including Mail Manager configurations, ingress endpoints, rule sets, traffic policies, archives, IAM roles, Secrets Manager secrets, AWS KMS keys, and CloudWatch log groups.

Conclusion

In this post, you configured Amazon SES Mail Manager in the eusc-de-east-1 Region of the AWS European Sovereign Cloud to send outbound email over SMTP. You created a traffic policy to enforce TLS and recipient filtering, a rule set to archive and deliver messages, and an authenticated ingress endpoint that serves as a compatible SMTP relay for your applications.

To learn more, see the Amazon SES Mail Manager documentation, open the Amazon SES console to start configuring your own pipeline, or visit the Amazon SES service page for additional features.

Additional references

For more information, see the following references:

About the authors

Schedule notebook runs in Amazon SageMaker Unified Studio

Post Syndicated from Shivani Mehendarge original https://aws.amazon.com/blogs/big-data/schedule-notebook-runs-in-amazon-sagemaker-unified-studio/

If you build notebooks for recurring tasks such as daily customer analysis, weekly report generation, or data quality checks in Amazon SageMaker Unified Studio, you’ve likely wanted to run them automatically on a schedule. Until now, there wasn’t a native way to do this. Teams had to manage orchestration separately, even though the interactive notebook experience was already in place. Now, notebook scheduling is available, so you can configure your production workloads to run automatically with minimal manual intervention.

In this post, we walk you through the new scheduling and orchestrating capabilities for notebooks in Amazon SageMaker Unified Studio. You will learn how to:

  • Trigger on-demand background runs, such as a model re-training job, without waiting at your desk.
  • Create recurring schedules for tasks such as nightly data freshness checks or weekly business reviews.
  • Parameterize notebooks so a single template can generate reports across different AWS Regions or customer segments.
  • Orchestrate multi-notebook workflows where one notebook’s output feeds into the next. For example, an extract, transform, and load (ETL) pipeline followed by a summary dashboard refresh.
  • Debug failed runs with AI-assisted troubleshooting.

Sample use case overview

In this walkthrough, you will take on the role of a logistics analyst who monitors shipping performance across carriers. The notebook loads shipping data from the ShippingLogs.csv dataset, identifies late deliveries, and generates a performance summary. You want to run this notebook every morning without manual intervention, reuse it across different carriers, and know when something goes wrong.

You will start by running a notebook in the background and viewing the results. Next, you will create a recurring schedule for daily runs, then parameterize the notebook to generate reports for different carriers. You will also orchestrate the notebook in a multi-step workflow and debug a failed run using AI-assisted troubleshooting.

Prerequisites

Before you begin, you need:

  • An Amazon SageMaker Unified Studio project with Notebooks enabled. See Set up IAM-based domains for permission requirements.
  • A sample dataset. We use the ShippingLogs.csv dataset, which contains shipping data including estimated and actual delivery times, carriers, and origins. You can download it from the Workshop Studio (the file is named ShippingLogs.csv on the linked page).

Setting up the notebook

Start by creating a new notebook in your SageMaker Unified Studio project. If you haven’t already, upload the ShippingLogs.csv file under the Shared tab in the Files panel.

SageMaker Unified Studio Notebook Files panel showing the Shared tab with the ShippingLogs.csv dataset uploaded

In the first cell, we load and explore the dataset. To reference the file in code, select the file in the Shared tab and copy the Amazon Simple Storage Service (Amazon S3) URI shown in the file details. Alternatively, you can reference it with this code:

import pandas as pd
from sagemaker_studio import Project

# Initialize the project
proj = Project()

# Get the S3 root path
s3_root = proj.s3.root

df = pd.read_csv(s3_root + '/ShippingLogs.csv')
df.head()

The dataset contains columns including Carrier, ActualShippingDays, ExpectedShippingDays, ShippingOrigin, ShippingPriority, and OnTimeDelivery. Add a second cell to analyze shipping performance for a single carrier:

import matplotlib.pyplot as plt

carrier_data = df[df['Carrier'] == 'GlobalFreight']
# Flag late deliveries
carrier_data['is_late'] = carrier_data['ActualShippingDays'] > carrier_data['ExpectedShippingDays']
late_pct = carrier_data['is_late'].mean() * 100
# Visualize actual vs expected shipping days
plt.figure(figsize=(12, 4))
plt.hist(carrier_data['ActualShippingDays'] - carrier_data['ExpectedShippingDays'], bins=20, edgecolor='black')
plt.axvline(x=0, color='red', linestyle='--', label='On time')
plt.title(f'Shipping Delay Distribution - GlobalFreight ({late_pct:.1f}% late)')
plt.xlabel('Days Over Expected')
plt.ylabel('Number of Shipments')
plt.legend()
plt.show()

With the notebook working interactively, you’re ready to automate it.

Running a notebook asynchronously

To trigger an asynchronous run, open your notebook. In the notebook header, choose the menu on the Run all button, and then choose Run in background.

Notebook header with the Run all menu expanded, showing the Run in background option

This captures a snapshot of the notebook in its current state and starts a run on a separate dedicated compute. You can continue working on other tasks or close the browser entirely. Your interactive session isn’t affected.

You will see a notification at the bottom of your screen confirming that the run started. To check the status of your run, choose View Run in the notification. This opens a view showing every background and scheduled run with its status, duration, and a link to view the full output.

Run history view showing background and scheduled runs with status, duration, and output links

You can choose to view the run details at any point to view results as cells run. The run details include three tabs:

  • Output: The notebook in read-only mode with cell results rendered, including dataframe outputs, visualizations, and print statements.
  • Parameters: The parameter values used for this run.
  • Logs: Run logs for debugging.

Run details view showing the Output, Parameters, and Logs tabs with rendered cell output

You can also access past runs by selecting the View Runs option in the notebook header.

Notebook header with the View Runs option highlighted

Stopping an in-progress run

If you need to cancel a run, open the run, and choose Stop. The run terminates, and its status updates to reflect the cancellation.

Run detail view with the Stop button selected to terminate an in-progress run

What to know about background runs

Compute: Each background run uses its own dedicated compute, separate from your interactive session. Your interactive work isn’t interrupted.

Packages: The packages that you install through the notebook’s package manager will be available in your background runs. When you use !pip install in code cells, the asynchronous run installs those packages as well.

Local files: Background runs can’t access files stored locally in your notebook environment. Reference data from your project’s shared storage (Amazon S3) or connected data sources instead.

Startup time: Expect a few minutes of startup time while compute is provisioned and your environment is prepared.

Creating a recurring schedule

Now that you’ve confirmed asynchronous runs work correctly, you can automate the notebook on a schedule. Choose the schedule icon in the notebook header to open the schedule creation form.

Schedule creation form opened from the notebook header schedule icon

Configure the following settings:

  • Schedule name: Enter a descriptive name, such as Daily Shipping Report.
  • Schedule type: Choose Recurring for repeated runs or One-time for a single future run.
  • Frequency: Define how often the notebook runs using a rate (for example, every one day) or a cron expression. Set the time zone and the start and end dates for the schedule. For example, set the schedule to run every day at 7:00 AM UTC starting tomorrow.
  • Flexible time window (optional): The number of minutes after the scheduled start time within which the run can be invoked. For example, with a 5-minute window, the notebook runs within 5 minutes of the start time.
  • Advanced settings:
    • Compute Instance: Keep the current settings or override with a different instance type for the asynchronous run to use.
    • Timeout: Set a maximum run duration to help prevent notebooks from running indefinitely. If left blank, it defaults to 60 minutes.

Choose Create.

Configured schedule form with name, recurring type, daily frequency, and advanced settings populated

The schedule appears in the Schedules tab of the activity panel. SageMaker Unified Studio creates an Amazon EventBridge Scheduler schedule for each schedule you configure.

Schedules tab in the activity panel listing the newly created Daily Shipping Report schedule

Viewing schedule run history

To view past runs for a schedule, choose the schedule name in the Schedules activity panel. This opens the schedule details view, where you can see the list of runs triggered by that schedule, the duration of each run, and a link to open the notebook output for an individual run.

Schedule details view showing the list of past runs with status, duration, and output links

Editing and deleting schedules

To modify a schedule, choose Edit next to it in the Schedules panel. You can change the frequency, instance type, timeout, and other configuration fields. To pause or resume a schedule, choose Pause or Resume from the same menu. To remove a schedule, choose Delete from that menu. Deleting a schedule stops future runs but preserves historical run outputs in Amazon S3 for auditing purposes.

Schedules panel with the Edit, Pause, Resume, and Delete options for a schedule

Parameterizing notebooks

With parameters, you can reuse a single notebook across different inputs without duplicating code. For example, you can run the same shipping performance report for each carrier by passing a different carrier name to each run.

Defining parameters

Open the Parameters activity panel and choose Add. Set the parameter name to carrier and the default value to GlobalFreight.

Parameters activity panel with the carrier parameter and GlobalFreight default value configured

Using parameters in code

In your notebook, replace the second cell with the following code. This retrieves the carrier parameter value using the SageMaker Unified Studio Python SDK instead of the hardcoded value:

import sagemaker_studio
import matplotlib.pyplot as plt

carrier = sagemaker_studio.nbutils.parameters.get("carrier")

carrier_data = df[df['Carrier'] == carrier].copy()
carrier_data['is_late'] = carrier_data['ActualShippingDays'] > carrier_data['ExpectedShippingDays']
late_pct = carrier_data['is_late'].mean() * 100

plt.figure(figsize=(12, 4))
plt.hist(carrier_data['ActualShippingDays'] - carrier_data['ExpectedShippingDays'], bins=20, edgecolor='black')
plt.axvline(x=0, color='red', linestyle='--', label='On time')
plt.title(f'Shipping Delay Distribution - {carrier} ({late_pct:.1f}% late)')
plt.xlabel('Days Over Expected')
plt.ylabel('Number of Shipments')
plt.legend()
plt.show()

Creating schedules with different parameter values

Now create three schedules for the same notebook, each targeting a different carrier:

  • “daily-shipping-gf” with carrier = GlobalFreight.
  • “daily-shipping-mc” with carrier = MicroCarrier.
  • “daily-shipping-shipper” with carrier = Shipper.

When you view a historical run, a separate Parameters tab in the run output displays the parameter values that were active for that run.

You can also override parameter values when triggering an on-demand background run. Choose the menu on the Run all button, then choose Run with settings. You can keep the defaults or provide custom values for that run.

Orchestrating with Workflows

To combine notebooks into a multi-step pipeline, such as running a data calculation notebook before the shipping log notebook, you can use the Notebook Operator in the Workflows tool to orchestrate them.

To do this, choose the Add to workflows button under the options menu of the notebook header.

Notebook header options menu with the Add to workflows button highlighted

This takes you to the Workflows tool, adding a new Notebook Operator task with prefilled properties from your notebook. When configuring the Operator task:

  • Select the target notebook from the notebook menu.
  • Use the Parameters widget to pass notebook parameters into the run of the notebook.
  • Specify optional arguments such as the compute instance and timeout configuration for the run.

Workflows canvas with a Notebook Operator task configured with notebook, parameters, and compute settings

Workflows also supports polling for the status of a notebook run for a particular notebook using Notebook Sensor. In Workflows, you can add a new Sensor task by hovering on the edge of the existing Operator task, where a plus (+) button is displayed.

Workflows canvas showing the plus button on the edge of an Operator task for adding a Sensor

You can then search for and add the Notebook Sensor to the canvas.

Task picker dialog with Notebook Sensor selected for adding to the workflow canvas

When configuring the Sensor task, specify the notebook run ID within the text field. The Operator’s form field contains Jinja templating to retrieve the notebook run. If the Sensor is used within the same workflow as the Operator, this template can be copied to use within a Sensor to poll the notebook run. Select the target notebook from the notebook menu.

Notebook Sensor configuration panel with the notebook run ID field populated using Jinja templating

Within Workflows, you can configure notebook runs to emit outputs and use those outputs as inputs for subsequent notebook runs.

Building off of the previous shipping log notebook example, we will pass the carrier parameter from an upstream notebook’s output. Your shipping-logs-analysis notebook should be already set up.

Because the notebook depends on the carrier parameter, you can specify it in the Parameters panel.

Parameters panel for the shipping-logs-analysis Operator with the carrier parameter dependency configured

Now, define a second notebook, calculate-best-carrier, which performs a calculation to determine our best carrier to use for shipping:

import pandas as pd
from sagemaker_studio import Project

# Initialize the project
proj = Project()

# Get the S3 root path
s3_root = proj.s3.root

df = pd.read_csv(s3_root + '/ShippingLogs.csv')
df.head()

carrier_stats = df.groupby('Carrier').agg(
    total=('OrderID', 'count'),
    late=('OnTimeDelivery', lambda x: (x == 'Late').sum())
).reset_index()
carrier_stats['late_pct'] = carrier_stats['late'] / carrier_stats['total'] * 100

best = carrier_stats.sort_values('late_pct', ascending=True).iloc[0]
best_carrier = best['Carrier']

print("Late % by carrier:")
print(carrier_stats.to_string(index=False))
print(f"\nBest carrier: {best_carrier} ({best['late_pct']:.1f}% late)")

To configure the calculate-best-carrier notebook’s outputs, you can choose the Variables panel. A new selector is available at the bottom of this panel which allows you to select variables to mark as outputs.

Variables panel with the selector at the bottom for marking notebook variables as outputs

We want this notebook to emit the best_carrier variable.

Variables panel showing best_carrier marked as an output variable for the calculate-best-carrier notebook

Now, use the Add to workflows button as previously demonstrated to quickly add this notebook within a workflow. Chain a second Notebook Operator that points to our shipping-logs-analysis notebook. Because we specified a parameter dependency on carrier for this notebook, it’s available as an option in the Parameters widget menu.

Parameters widget menu of a Notebook Operator showing carrier as a configurable parameter dependency

When they’re chained, the notebook tasks detect the outputs set in upstream notebook runs. These outputs can be selected as keys within the Parameters widget of the Operator to pass into the run. This can be done recursively for an arbitrary number of Operator tasks. We can select the emitted best_carrier output from the calculate-best-carrier notebook.

Parameters widget displaying best_carrier as a selectable upstream output to pass into the next Operator

You can now choose the Save button on the top left of the visual canvas and the Run button to start the workflow. When the workflow is completed, the specified notebook outputs are available in the Task Output panel and the notebook run result can be viewed in the Notebooks tool.

Task Output panel showing the emitted notebook outputs after a successful workflow run

Notebook run result rendered in the Notebooks tool after the chained workflow completes

In a similar manner, the Notebook Sensor will also emit the notebook outputs from a particular notebook’s run which can be used within other tasks. This is useful when you want to retrieve outputs from a notebook run in another workflow.

Debugging a failed run with AI assistance

When viewing your past runs, you notice that a run from earlier today has a Failed status. Choose the failed run to open the notebook output in read-only mode.

In this example, suppose you incorrectly referred to column name ActualShippingDays as DeliveryDays. The run would fail with a KeyError: 'DeliveryDays' in the cell that computes late deliveries.

At the top of the failed run output, choose Troubleshoot with AI. Choosing the Troubleshoot with AI button lands you in the notebook with the Agent chat panel open.

Failed run output with the Troubleshoot with AI button highlighted at the top of the page

The data agent analyzes the cell outputs, identifies the cell that errored, explains the root cause, and suggests a fix. In this case, it identifies that the column DeliveryDays doesn’t exist in the dataframe and suggests updating the code reference. You can review the change, then verify the fix by choosing Run in background from the Run all menu to trigger a test run before the next scheduled run.

Note: You can also use the Data Agent to create schedules and start notebook runs using natural language, without having to navigate.

Cleaning up

To avoid incurring future charges, delete the resources that you created in this walkthrough:

  • Delete any schedules that you created from the Schedules panel in your notebook.
  • Delete test notebooks if you don’t need them.
  • Navigate to the Workflows page and delete any workflows that you created during this walkthrough.
  • Your project’s Amazon S3 storage retains historical run outputs until you manually remove them.

Conclusion

In this post, we showed how to run notebooks in the background in Amazon SageMaker Unified Studio using background runs, schedules, parameterization, workflow orchestration, and AI-assisted debugging. Using a shipping logistics dataset, we demonstrated how a single notebook can be parameterized to generate performance reports for different carriers on independent schedules, all without duplicating code or managing extensive infrastructure.

To get started, open a notebook in your SageMaker Unified Studio project, choose the menu on the Run all button in the notebook header, and choose Run in background. For more advanced use cases, explore workflows in Amazon SageMaker Unified Studio to build multi-step data pipelines, or review the Amazon SageMaker Unified Studio User Guide for additional configuration options.

Learn more:

If you have feedback or questions, reach out on AWS re:Post for Amazon SageMaker Unified Studio.

About the authors

Shivani Mehendarge

Shivani Mehendarge

Shivani is a Software Development Engineer at Amazon Web Services, where she builds scalable infrastructure that helps data teams run and automate their workloads in Amazon SageMaker Unified Studio. She is passionate about solving complex distributed systems challenges and building reliable cloud services.

Regan Perk

Regan Perk

Regan is a Senior Software Development Engineer on the Amazon SageMaker Unified Studio team. She designs, implements, and maintains features that enable customers to manage schedules and workflows in SageMaker Unified Studio.

Qazi Ashikin

Qazi Ashikin

Qazi is a Software Development Engineer at Amazon Web Services, where he works on developing features that allow customers to orchestrate workflows and schedules in SageMaker Unified Studio. He also works on AWS Glue Studio, where he builds agentic systems and maintains services that enable data analytics.

Building highly available Oracle databases with Amazon FSx for NetApp ONTAP

Post Syndicated from Vignyanand Penumatcha original https://aws.amazon.com/blogs/architecture/building-highly-available-oracle-databases-with-amazon-fsx-for-netapp-ontap/

Oracle databases power mission-critical enterprise applications, making their continuous availability essential for business operations. Traditional Oracle high availability (HA) solutions require complex clustering software, expensive shared storage arrays, and specialized database administration teams. These conventional approaches often introduce single points of failure while demanding significant operational overhead.

Modern cloud architectures offer a transformative approach that combines Amazon FSx for NetApp ONTAP (FSxN) with Amazon EC2 Auto Scaling groups, automated AMI creation, AWS Lambda-driven orchestration, and AWS Systems Manager Parameter Store (SSM Parameter). This solution removes traditional Oracle HA complexities while delivering enterprise-grade availability, automated recovery, and makes sure new instances launch with the latest Oracle configuration.

This post shows how to build a highly available Oracle database architecture using FSxN shared storage, Auto Scaling groups with dynamic AMI updates, and serverless orchestration to help reduce recovery times with current configurations.

Solution overview

The solution uses multiple AWS services working together to create a comprehensive high availability architecture. FSxN Multi-AZ provides persistent shared storage spanning availability zones for Oracle database files, software, and configurations, so that data remains accessible when EC2 instances are replaced. Auto Scaling groups deliver automated instance lifecycle management with the latest AMI configurations, so failed instances are quickly replaced with identical configurations that can immediately access the existing Oracle database files on FSxN. AWS Backup creates AMIs that capture the latest Oracle host configurations including patches and settings, preserving the complete server state for consistent deployments. AWS Lambda extracts the AMI ID from backup recovery points and updates the SSM Parameter, orchestrating the entire configuration management workflow. Systems Manager Parameter Store stores the current AMI ID for Auto Scaling group launch templates, so new instances always launch with the most recent configuration and can immediately connect to the Oracle database on shared storage.

The following diagram shows the complete architecture with all AWS services and their interactions:

AWS architecture diagram showing Oracle Database disaster recovery across two Availability Zones using FSx for ONTAP synchronous replication, AWS Backup automation with EventBridge and Lambda, and Auto Scaling group with SSM Parameter Store for AMI management.

Key benefits include:

  • Recovery Time Objective (RTO): Can help achieve 2–5 minutes with latest Oracle configuration
  • Recovery Point Objective (RPO): Near-zero through synchronous Multi-AZ replication
  • Configuration consistency: New instances launch with identical Oracle host setup
  • Automated AMI management: Scheduled AMI creation with Parameter Store updates

Walkthrough

This walkthrough demonstrates implementing Oracle HA using Amazon FSx for NetApp ONTAP shared storage, AWS Backup-driven AMI creation, Lambda orchestration, and Auto Scaling groups with Parameter Store integration for configuration consistency and automated failover.

Prerequisites

For this walkthrough, you should have the following prerequisites:

  • An AWS account with appropriate permissions for Amazon FSx, Auto Scaling, EC2, Lambda, and Systems Manager
  • A VPC with subnets in at least two Availability Zones
  • Oracle database software

Keep in mind that customers are responsible for their own Oracle licensing compliance.

  • An EC2 instance with Oracle database installed and configured
  • AWS Identity and Access Management (IAM) roles for AMI creation and cross-service communication
  • Basic knowledge of Oracle database administration and AWS automation

Assumptions

This post is a conceptual illustration of the architecture. Your specific implementation will vary based on your VPC layout, Oracle version, storage requirements, and organizational security policies.

We assume the reader is familiar with:

  • Creating and configuring Amazon FSx for NetApp ONTAP file systems through the AWS console
  • iSCSI concepts including initiators, targets, and multipath I/O
  • Oracle database startup and shutdown procedures
  • AWS Backup, Lambda, and Auto Scaling group fundamentals

For detailed step-by-step instructions on specific AWS services, refer to the additional resources section.

Step 1: Create an Amazon FSx for NetApp ONTAP file system

FSxN Multi-AZ provides the persistent shared storage foundation for this architecture. Unlike Amazon Elastic Block Store (Amazon EBS) volumes, which are bound to a single AZ, FSxN Multi-AZ replicates data synchronously across two AZs with automatic failover. This means that when an EC2 instance is replaced (whether in the same AZ or a different one), the new instance can immediately access the existing Oracle database files without restoring from backup.

To create the file system, navigate to the Amazon FSx console and select Amazon FSx for NetApp ONTAP as the file system type.

The critical configuration choice is selecting Multi-AZ deployment, which places an active file server in one AZ and a standby in another.

Amazon FSx console showing oracle-fsxn-multi-az file system configuration with ONTAP Multi-AZ 1 deployment, 1024 GiB SSD storage, 512 MB/s throughput, spanning us-east-1a preferred and us-east-1b standby subnets.

FSxN console showing Multi-AZ deployment type selection with preferred and standby subnets in separate availability zones.

After the file system is created, you need to set up a Storage Virtual Machine (SVM), which acts as a logical storage container providing data access to your Oracle instances. The SVM creation is done from the FSx console under your file system’s details.With the SVM in place, the next step is configuring iSCSI access. FSxN exposes iSCSI endpoints—these are IP addresses (one per AZ) that your EC2 instances use to connect to the storage over the iSCSI protocol. You can find these endpoint addresses in the FSx console under your SVM’s Endpoints tab.

Amazon FSx Storage Virtual Machine configuration page showing oracle-svm with Created lifecycle state, NFS, iSCSI, and management endpoints for Oracle Database storage connectivity.

SVM Endpoints tab showing iSCSI endpoint IP addresses for each availability zone. These addresses are used in the EC2 instance’s iSCSI discovery configuration.

The iSCSI setup involves creating iGroups (which define which EC2 instances can access the storage) and LUNs (logical storage units mapped to those groups) through the NetApp ONTAP CLI. On the EC2 side, you configure the iSCSI initiator to discover and connect to the FSxN endpoints, then mount the resulting block devices. Using multipath I/O with both endpoints makes sure that Oracle data remains accessible even during an AZ failover. For detailed iSCSI configuration steps, see mounting iSCSI LUNs on Linux clients.

A dedicated security group is required for FSxN access. At minimum, the security group must allow inbound traffic on ports 111 (NFS portmapper), 635 (NFS mountd), 2049 (NFS), 3260 (iSCSI), 4045–4046 (NFS lock), 443 (HTTPS for management), and 22 (SSH for ONTAP CLI). Restrict the source to only your Oracle EC2 instances’ security group.

Step 2: Set up AWS Backup for EC2 instance protection

AWS Backup captures the complete state of your Oracle EC2 instance. The key design choice here is using tag-based resource selection rather than specifying instance IDs directly. Because Auto Scaling groups replace instances (and generate new instance IDs), tag-based selection makes sure that any new instance with the correct tags are automatically included in the backup plan.Configure a backup plan with a frequency appropriate for your environment and set the resource assignment to select EC2 instances matching your application tag (for example, ‘Application: Oracle’).

AWS Backup console showing blog-test backup plan with hourly backup rule targeting Oracle EC2 instances identified by the Application:oracle-db tag.

AWS Backup resource assignment configured with tag-based selection. Any EC2 instances tagged with the application tag are automatically included in the backup plan.

Step 3: Configure Lambda for AMI management

When AWS Backup completes an EC2 backup, it creates an AMI as the recovery point. An Amazon EventBridge rule detects this completion event and triggers a Lambda function. The function extracts the AMI ID from the backup recovery point, updates the SSM Parameter Store parameter with the new AMI ID, and cleans up older AMIs to control storage costs.

AWS Lambda function configuration for oracle-backup-handler showing Python 3.11 runtime, EventBridge trigger, and description indicating it processes AWS Backup completion events and updates AMI in SSM.

Lambda function overview showing the EventBridge trigger, Python 3.11 runtime, and function description indicating its role in processing backup completions and updating AMI references in SSM.

This event-driven approach means the latest AMI is available without manual intervention. The Lambda function needs IAM permissions for EC2 (to manage AMIs), SSM (to update the parameter), and Backup (to read recovery point metadata).

Amazon EventBridge rule oracle-backup-completion configured to trigger the oracle-backup-handler Lambda function when AWS Backup completes an EC2 backup job, with event pattern filtering for COMPLETED state.

EventBridge rule configured to match AWS Backup job completion events for EC2 resources, with the Lambda function as the target.

Step 4: Configure the Systems Manager Parameter Store

The SSM Parameter Store holds the current AMI ID that the Auto Scaling group’s launch template references. The parameter is created with the aws:ec2:image data type, which enables the launch template’s resolve:ssm: functionality, a feature that allows the launch template to dynamically resolve the AMI ID at instance launch time without requiring a template version update.

AWS Systems Manager Parameter Store showing /oracle/ec2/ami-id parameter with AMI value ami-0a705a7d5523c555, version 857, last modified by the oracle-backup-lambda-role on April 25, 2026.

SSM Parameter Store showing the /oracle/ec2/ami-id parameter with aws:ec2:image data type. The “Last modified user” confirms the Lambda function is automatically updating this parameter after each backup cycle.

When Lambda updates this parameter after each backup cycle, the next instance launched by the Auto Scaling group will automatically use the latest AMI. This removes the operational burden of manually updating launch template versions.

Step 5: Set up an Auto Scaling Group with dynamic AMI

The launch template references the SSM parameter using the resolve:ssm: prefix for the AMI ID field. This is the mechanism that ties the entire automation pipeline together. The mechanism backups trigger AMI creation, AMI IDs flow into Parameter Store, and the launch template resolves the latest AMI at launch time.

EC2 Launch Template oracle-db-launch-template version 75 showing AMI ID resolved from SSM parameter resolve:ssm:/oracle/ec2/ami-id with r7i.large instance type for Oracle Database deployment.

Launch template AMI configuration showing the ‘resolve:ssm:’ prefix, which dynamically retrieves the latest AMI ID from Parameter Store at instance launch time.

The Auto Scaling group is configured with minimum, maximum, and desired capacity all set to 1. This is not traditional auto-scaling, it’s a self-healing pattern. The sole purpose is to detect when the Oracle instance becomes unhealthy and automatically launch a replacement. The health check grace period should be set to at least 300 seconds (5 minutes) to allow Oracle sufficient time to start before health checks begin evaluating the new instance.

The launch template also includes a User Data script that runs on each new instance. This script configures the iSCSI initiator, discovers and connects to the FSxN endpoints, mounts the Oracle data volumes, and starts the Oracle database through a systemd service. This automation makes sure that a replacement instance is fully operational without manual intervention.

EC2 Auto Scaling group oracle-db-asg configuration showing desired capacity of 1, scaling limits 1-1, r7i.large instance type, oracle-db-launch-template with Latest version, spanning two availability zone subnets.

Auto Scaling group configured with min=max=desired=1 across two availability zones, providing self-healing capability.

Test the complete workflow

To validate the architecture, simulate an instance failure by terminating the current Oracle EC2 instance.

The expected sequence is:

  1. The Auto Scaling group detects the instance is unhealthy (within approximately 30 seconds)
  2. A new instance launches from the latest AMI resolved from Parameter Store (approximately 2 minutes)
  3. The User Data script connects to FSxN using iSCSI and starts Oracle (approximately 2–3 minutes)
  4. The Oracle database is available and accepting connections (total elapsed: approximately 5 minutes)

Auto Scaling group Activity History showing the self-healing sequence — the unhealthy instance is terminated, and a replacement is launched automatically within seconds.

The new instance automatically inherits the application tags from the Auto Scaling group, which means AWS Backup includes it in the next backup cycle without manual configuration.

Cleaning up

To avoid incurring future charges, delete the resources:

  • Delete Lambda functions and EventBridge rules
  • Remove Parameters from Systems Manager Parameter Store
  • Delete AWS Backup plans and backup vault
  • Deregister created AMIs
  • Terminate Auto Scaling group instances
  • Delete the Amazon FSx for NetApp ONTAP file system

Conclusion

This architecture facilitates Oracle high availability with configuration consistency by combining FSxN persistent shared storage with automated AMI management and AWS Backup protection. The Lambda-driven AMI management from backup recovery points and Parameter Store integration helps make sure that replacement instances launched by Auto Scaling groups always use the latest Oracle host configuration and can immediately connect to the existing Oracle database files stored on FSxN. Replacements occur only when health checks fail. Organizations can target high availability while maintaining configuration consistency across instance replacements. The automated AMI management alleviates configuration drift and makes sure that disaster recovery scenarios restore Oracle instances with identical host-level configurations that can immediately access the persistent Oracle database on shared storage. Healthy instances continue running unchanged, with replacements occurring only, when necessary, because of health check failures.Next steps include implementing cross-Region AMI replication, adding AMI validation testing, and developing custom health checks that verify both Oracle database and host configuration consistency.

Additional resources

Amazon OpenSearch Service: Mechanisms to secure your domain

Post Syndicated from Imtiaz Sayed original https://aws.amazon.com/blogs/big-data/amazon-opensearch-service-mechanisms-to-secure-your-domain/

Imagine you’re building a product search feature for your website or storing customer records in Amazon OpenSearch Service to power full-text search. The moment that real user data enters your domain, security becomes essential.

Whether your workload is a public-facing website search, an internal application querying sensitive data, or a pipeline handling personally identifiable information (PII), the questions you face are the same:

  • Who should be allowed to connect to my domain?
  • How do I authenticate users and services?
  • How do I make sure that even authenticated users only see data they are entitled to see?
  • How do I satisfy regulatory requirements such as HIPAA, PCI DSS, or SOC 2?

This post offers an overview of the security mechanisms available for Amazon OpenSearch Service, spanning authentication and authorization, encryption, and network access controls. You learn how to implement fine-grained access control, manage AWS Identity and Access Management (IAM) roles, and secure data both in transit and at rest for both public and virtual private cloud (VPC) access domains.

Scope: This post covers security for Amazon OpenSearch Service managed clusters only. It doesn’t cover Amazon OpenSearch Serverless, which uses a different security model. For serverless security, see Amazon OpenSearch Serverless security in the AWS documentation.

To begin, let’s look at the security layers in Amazon OpenSearch Service.

Amazon OpenSearch Service security layers

Amazon OpenSearch Service has multi-layer security. The following diagram illustrates the multi-layer security in Amazon OpenSearch Service.

Diagram showing the three security layers of Amazon OpenSearch Service: Network, Domain access policy, and Fine-grained access control

Figure 1: Multi-layer security.

The three main layers of security are network, domain access policy, and fine-grained access control.

Network – The first security layer is the network, which determines whether requests reach an OpenSearch Service domain. If you choose Public access when you create a domain, requests from any internet-connected client can reach the domain endpoint. If you choose VPC access, clients must connect to the Amazon Virtual Private Cloud (Amazon VPC) (and the associated security groups must permit it) for a request to reach the endpoint.

Domain access policy – The second security layer is the domain access policy. After a request reaches a domain endpoint, the resource-based access policy allows or denies the request access to a given URI. The access policy accepts or rejects requests at the edge of the domain, before they reach data or indexes in OpenSearch itself.

Fine-grained access control – The third and final security layer is fine-grained access control. After a resource-based access policy allows a request to reach a domain endpoint, fine-grained access control evaluates the user credentials and either authenticates the user or denies the request. If fine-grained access control authenticates the user, it fetches all OpenSearch internal roles mapped to that user and uses the full set of permissions to determine how to handle the request.

With fine-grained access control, you can control access to your data in Amazon OpenSearch Service. For example, depending on who makes the request, you might want to hide certain fields in your documents or exclude certain documents altogether. With fine-grained access control, you can:

  • Define role-based access control to determine who can perform which actions on which indexes, documents, and fields.
  • Define security at the index, document, and field level to allow access to only required data.

Fine-grained access control requires OpenSearch or Elasticsearch 6.7 or later. It also requires HTTPS for all traffic to the domain, encryption of data at rest, and node-to-node encryption. Depending on how you configure the advanced features of fine-grained access control, more processing of your requests might require compute and memory resources on individual data nodes. After you turn on fine-grained access control, you can’t turn it off. For more details, see Fine-grained access control in Amazon OpenSearch Service in the AWS documentation.

To learn more about security features in an OpenSearch Service domain, let’s start by configuring a new public access domain. We discuss a VPC access domain later in the post.

Public access domain

With a public access domain, you can configure an OpenSearch Service domain so that the domain endpoint is accessible from the internet.

The AWS console for Amazon OpenSearch Service provides a guided wizard that you can use to configure and reconfigure your provisioned Amazon OpenSearch Service domains. Follow the Tutorial: Configure a domain with the internal user database and HTTP basic authentication in the AWS documentation to configure a domain with basic authentication and validate fine-grained access control.

Let’s review some important configuration attributes for a public access domain.

Network:

Public access. To simplify the network access configurations, you can use Public access, but for production workloads, we recommend VPC access.

With the domain in public access, you have several options to secure access. While you can use a resource-based access policy to restrict access to specific IAM principals or IP addresses, the recommended approach is to turn on fine-grained access control (FGAC) and use it as the primary mechanism for securing your domain. With FGAC turned on, you can set an open access policy (allowing all traffic to reach the domain) and let FGAC handle authentication and authorization at the index, document, and field level.

When using IAM-based authentication with FGAC, you should map IAM roles to backend roles in OpenSearch. You can use backend roles to assign permissions to groups of users based on their IAM role, rather than managing individual user mappings. This is especially important because if your IAM federation or authentication mechanism changes, the backend role mappings make sure of consistent access control within OpenSearch.

Amazon OpenSearch Service network configuration screen with Public access selected

Figure 2: Use public access domain.

Fine-grained access control: Fine-grained access control provides numerous features to help you keep your data secure, such as document-level security, field-level security, read-only users, and OpenSearch Dashboards/Kibana tenants. Fine-grained access control requires a primary user, which is the administrator identity we discuss through the rest of this post.

The primary user is the administrator identity for your OpenSearch domain. This user can set up additional users in Amazon OpenSearch Service, assign roles to them, and assign permissions for those roles. You can choose username and password authentication for the primary user or use an IAM identity. You use these credentials to log in to OpenSearch Dashboards. Following the best practices on choosing your primary user, you should move to an IAM primary user for production workloads.

Fine-grained access control can be applied regardless of how you log in. You can follow your organization’s suggested authentication mechanism and apply fine-grained access control on top of it.

FGAC provides security at multiple levels to meet your security needs:

  • Index-level security – Controls who can create, search, read, write, update, or delete within specific indexes.
  • Document-level security – Restricts which documents within an index a user can see, using OpenSearch query filters (for example, only show documents where department: “sales”).
  • Field-level security – Controls which fields within documents are visible (include or exclude specific fields).
  • Field masking – Anonymizes sensitive field data (for example, hash a release_date or SSN field) rather than hiding it entirely.

Fine-grained access control supports several authentication mechanisms, including HTTP basic authentication using an internal user database, Amazon Cognito for web-based Dashboards access, SAML for enterprise identity provider integration, JSON Web Tokens (JWT) for token-based authentication, and AWS Identity and Access Management with SigV4 signing for IAM users and roles.

Encryption:

Amazon OpenSearch Service encrypts data both in transit and at rest. When you turn on fine-grained access control, encryption is required—the corresponding settings are automatically turned on and can’t be changed. These include Transport Layer Security (TLS 1.2 or later) for requests to the domain and for traffic between nodes in the domain, and encryption of data at rest through AWS Key Management Service (AWS KMS).

For encryption at rest, OpenSearch Service supports three key types: AWS owned keys, AWS managed keys, and customer managed keys. While AWS owned keys provide a quick-start option with no additional configuration, customer managed keys are the recommended best practice. Customer managed keys give you full control over the encryption key lifecycle, including key rotation policies, granular access control through key policies, and the ability to audit key usage through AWS CloudTrail. To use a customer managed key, create a symmetric encryption key in AWS KMS and select it when configuring your domain’s encryption settings.

For a basic public access domain with FGAC, all traffic reaches the domain freely (no VPC restriction), and an open access policy is used so no SigV4 signing is needed. FGAC then takes over, authenticating users through the internal user database (username/password) and enforcing role-based permissions at the index, document, and field level.

The public access configuration we discussed is useful for development and testing, but for production workloads, a best practices deployment combines VPC access, IAM-based authentication, and fine-grained access control. This approach layers all three security mechanisms—network isolation, identity verification, and granular permissions—to protect your domain end to end.

VPC access domain

Placing your OpenSearch Service domain inside a VPC restricts network-level access to resources within the VPC or connected networks. Traffic between your applications and the OpenSearch endpoint doesn’t traverse the public internet, and you can use security groups to further limit which entities can communicate with the domain. OpenSearch Service places a VPC endpoint (VPCe) using AWS PrivateLink into one, two, or three subnets of your VPC depending on your Availability Zone configuration. For high availability (HA), turn on multiple Availability Zones with each subnet in a different zone within the same AWS Region. For more details, see Launching your Amazon OpenSearch Service domains within a VPC.

For this best practices deployment, we use an IAM primary user with Amazon Cognito authentication for OpenSearch Dashboards and for fine-grained access control. We configure a primary IAM role and a limited IAM role, associate them with users in Amazon Cognito through a user pool and identity pool, and then use fine-grained access control to manage permissions. The primary user can then sign in to OpenSearch Dashboards, create backend roles, map the limited user to a restricted role, and enforce granular access at the index, document, and field level. For more details, see Tutorial: Configure a domain with an IAM master user and Amazon Cognito authentication in the AWS documentation.

The following high-level steps detail what’s needed to configure a VPC access domain with Amazon Cognito users. These steps use the Amazon Cognito user pool for authentication. The same basic process works for any Cognito authentication provider that lets you assign different IAM roles to different users.

  • Create an Amazon Cognito user pool.
  • Add users in the user pool for the primary user and a limited-access user.
  • Create an Amazon Cognito identity pool.
  • Update the IAM role for the primary user to allow access to OpenSearch Dashboards.
  • Create an IAM role for the limited user.
  • Create the domain.

You can follow Creating and managing Amazon OpenSearch Service domains in the AWS documentation to provision a domain. The following sections describe some important attributes for the domain.

Network:

VPC access. Public access isn’t recommended for production workloads. We recommend that you use VPC access for all production workloads. Pick the VPC, subnets, and security group that you have created for the OpenSearch domain.

Amazon OpenSearch Service network configuration screen with VPC access selected and VPC, subnet, and security group fields populated

Figure 4: Use VPC access.

Fine-grained access control:

Turn on fine-grained access control with OS[MasterUserRole] as the primary user. You can follow steps in Tutorial: Configure a domain with an IAM master user and Amazon Cognito authentication to create OS[MasterUserRole].

Amazon OpenSearch Service fine-grained access control configuration with an IAM ARN selected as the primary user

Figure 5: Turn on fine-grained access control with an IAM role.

Fine-grained access control provides numerous features to help you keep your data secure, such as document-level security, field-level security, read-only users, and OpenSearch Dashboards/Kibana tenants. Fine-grained access control requires a primary user.

The primary user is the administrator identity for your OpenSearch domain. This user can set up additional users in Amazon OpenSearch Service, assign roles to them, and assign permissions for those roles. You can choose username and password authentication for the primary user or use an IAM identity. You use these credentials to log in to OpenSearch Dashboards. Following the best practices on choosing your primary user, you should choose an IAM primary user for production workloads.

Fine-grained access control can be applied regardless of how you log in. You can follow your organization’s suggested authentication mechanism and apply fine-grained access control on top of it.

Amazon Cognito authentication:

To turn on Amazon Cognito authentication, select Enable Amazon Cognito authentication and choose the Amazon Cognito user pool and Amazon Cognito identity pool for your OpenSearch Dashboards.

Amazon OpenSearch Service authentication configuration with Amazon Cognito enabled and a user pool and identity pool selected

Figure 6: Turn on Amazon Cognito authentication.

Access policy:

The access policy controls whether a request is accepted or rejected when it reaches the Amazon OpenSearch Service domain. You can configure a domain-level access policy to allow access to your Amazon OpenSearch Service domain.

Amazon OpenSearch Service domain access policy editor showing a JSON policy granting access to the configured IAM principals

Figure 7: Configure domain-level access to the domain.

Encryption:

Amazon OpenSearch Service encrypts data both in transit and at rest. When you turn on fine-grained access control, encryption is required—the corresponding settings are automatically turned on and can’t be changed. These include Transport Layer Security (TLS 1.2 or later) for requests to the domain and for traffic between nodes in the domain, and encryption of data at rest through AWS KMS.

For encryption at rest, OpenSearch Service supports three key types: AWS owned keys, AWS managed keys, and customer managed keys. While AWS owned keys provide a quick-start option with no additional configuration, customer managed keys are the recommended best practice. Customer managed keys give you full control over the encryption key lifecycle, including key rotation policies, granular access control through key policies, and the ability to audit key usage through AWS CloudTrail. To use a customer managed key, create a symmetric encryption key in AWS KMS and select it when configuring your domain’s encryption settings.

With these configurations, you can configure your Amazon OpenSearch domain and OpenSearch Service Dashboards so that they’re accessible only within the chosen VPC. For your production scenario, you can follow your organization’s approved mechanism to access the resources in a VPC. You can access OpenSearch Service Dashboards with a primary user to create a limited-access role and map it to the IAM role with limited access to validate fine-grained access control.

Conclusion

In this post, we looked at the important security configurations for a public and a VPC-based Amazon OpenSearch domain. You can examine more settings for fine-grained access control in the OpenSearch Dashboards Security section.

If you have feedback about this post, submit comments in the Comments section. If you have questions about this post, start a new thread on the Amazon OpenSearch Service forum or contact AWS Support.

About the author

Imtiaz (Taz) Sayed

Imtiaz (Taz) Sayed

Imtiaz (Taz) Sayed is the WW Tech Leader for Analytics at AWS. He enjoys engaging with the community on all things data and analytics. He can be reached through LinkedIn.

Narendra Gupta

Narendra Gupta

Narendra is a Specialist Solutions Architect at AWS, helping customers on their cloud journey with a focus on AWS analytics services. Outside of work, Narendra enjoys learning new technologies, watching movies, and visiting new places.

Akhilesh Dube

Akhilesh Dube

Akhilesh is a Senior Analytics Solutions Architect at AWS. He possesses more than two decades of expertise in working with databases and analytics products. His primary role involves collaborating with enterprise clients to design robust data analytics solutions while offering comprehensive technical guidance on a wide range of AWS Analytics and AI/ML services.

Nishchai JM

Nishchai JM

Nishchai is an Analytics and GenAI Specialist Solutions Architect at Amazon Web services. He specializes in building larger scale distributed applications and help customer to modernize their workload on Cloud. He thinks Data is new oil and spends most of his time in deriving insights out of the Data.

Automating contract intelligence with Doczy.ai™ on AWS

Post Syndicated from Sanket Nasre original https://aws.amazon.com/blogs/architecture/automating-contract-intelligence-with-doczy-ai-on-aws/

Extracting actionable insights from thousands of contracts and legal documents remains a challenge. For organizations, critical business information is locked in unstructured documents such as contracts, legal agreements, provider arrangements, and vendor invoices. Extracting and operationalizing this information has traditionally been a manual, error-prone, and resource-intensive process. This leads to missed savings opportunities, costly delays, and significant inefficiencies across the enterprise.

AArete, a global management and technology consulting firm specializing in healthcare, recognized this challenge and developed Doczy.ai™, an intelligent contract interpretation solution powered by generative AI on Amazon Web Services (AWS).

In this post, we show you how Doczy.ai™ uses generative AI on AWS to automate contract intelligence at scale, transforming unstructured documents into structured, actionable insights, so organizations can automate critical business processes and unlock the full value of their data.

The challenge: Data trapped in documents

For healthcare organizations, managing and interpreting contracts and documents represents a major operational bottleneck. Manual review processes require deploying teams to extract data from thousands of documents. This is an approach that is neither scalable nor sustainable, highly prone to error, and costly. Organizations relying on institutional knowledge face additional risks: critical information resides with a few key individuals, creating knowledge silos and succession planning challenges. Existing Contract Lifecycle Management (CLM) systems often prove inadequate for capturing the nuanced and complex terms unique to each agreement. These legacy systems can only configure predefined fields, missing the rich detail and contextual information that distinguishes contracts. The downstream impact is substantial: in healthcare, reimbursement terms must be manually translated into claims systems—a slow, error-prone process. Similarly, verifying vendor invoices against contract terms often requires manual effort, leading to payment processing delays and missed contractual savings opportunities. These inefficiencies ultimately leave significant value on the table.

This is where Doczy.ai™ provides significant value.

Doczy.ai™: An intelligent contract interpretation solution

Doczy.ai™ directly addresses these challenges using advanced AI and scalability on AWS. Developed by AArete, Doczy.ai™ pushes the boundaries of document intelligence. The solution automatically interprets complex documents and converts them into a structured, queryable information repository that allows organizations to unlock the full value of their data and drive smarter decisions.The evolution of Doczy.ai™ reflects rapid AI advancement. Prior to 2020, document processing required manual effort, with individuals processing approximately 100 documents per week. Between 2020–2023, the firm implemented rules-based contract processing, achieving approximately 55% accuracy. The breakthrough came in 2024 with an AI-based processing built on AWS achieved 99% accuracy—a dramatic improvement over the 55% accuracy of traditional rules-based systems.

Doczy.ai™ architecture

Doczy.ai™ is built on a comprehensive AWS architecture designed to handle the entire document processing lifecycle: from the moment a file enters the system to the moment it generates actionable business intelligence.

Doczy.ai is built on a comprehensive AWS architecture designed to handle the entire document processing lifecycle: from the moment a file enters the system to the moment it generates actionable business intelligence.

Architecture of Doczy.ai™

External users access the platform through a secure Next.js frontend, with Amazon Cognito managing authentication and authorization behind the scenes. After authentication, users upload documents directly to Amazon Simple Storage Service (Amazon S3), where durable, scalable object storage ensures nothing is lost and everything is accessible at scale. From there, the real intelligence begins.

An AWS Lambda function triggers Amazon Textract to extract text and metadata from documents in various formats. What sets Doczy.ai™ apart at this stage is its patented “smart chunking” algorithm, a proprietary approach that goes far beyond pulling words off a page. Rather than treating a document as a flat sequence of text, smart chunking preserves hierarchical structure and one-to-many relationships within documents. It uses a combination of semantic and keyword search to decompose text into meaningful, context-aware chunks, applying dynamic parameters to maintain logical relationships throughout. Sequential identifiers and metadata-driven grouping organize these chunks into field groups, detecting overlaps and removing duplications while keeping the document’s natural flow intact.

After chunking, the document enters the dual clustering engine of Doczy.ai™. This two-lens methodology analyzes every contract simultaneously from both a semantic and a structural perspective. On the semantic side, extracted text is converted into embeddings, numerical representations of meaning, and similar ideas are grouped together even when they’re expressed in different words. On the structural side, pattern-recognition algorithms identify clause types, formatting conventions, table layouts, and hierarchical organization, understanding. For example, that a three-nested-level exhibit carries fundamentally different implications than a straightforward attached schedule.These two analyses don’t operate in isolation. Projection algorithms compare the semantic and structural clusters side by side, synthesizing them into a unified, enriched document model that captures both meaning and context. It’s this convergence that drives the 99% accuracy rate of Doczy.ai™. The system doesn’t just read the words, it understands the contract. Advanced large language models (LLMs) then generate structured output grounded in this dual-clustered intelligence.Before output is finalized, the system determines each document’s file class and generates prompts tailored to the extracted text, cluster classification, and domain context. Through few-shot and multi-shot prompting, the platform continuously edits the prompt on domain-specific examples and based on real outputs, creating a feedback loop that compounds accuracy improvements over time.

The resulting structured data flows into Snowflake, forming a centralized repository that powers intelligent dashboards with actionable insights and visualizations. Throughout the entire pipeline, Amazon CloudWatch monitors performance in real time and proactively surfaces issues before they escalate, while AWS Secrets Manager safeguards sensitive information, ensuring that security is not an afterthought, but a foundational layer woven into every stage of the system.

The transformative impact of Doczy.ai™

The results of this AI-powered approach are transformative and measurable. By automating contract interpretation and document processing, Doczy.ai™ has demonstrated significant impact at scale for multiple organizations across healthcare and financial services. The scale of operations over the last 22 months demonstrates the maturity and production readiness of Doczy.ai™. This solution has processed 2.5 million contract documents (50 million pages) with 137 million API calls to Amazon Bedrock and 442 billion tokens—a level of automation and accuracy previously unattainable through manual or traditional document processing approaches. Over this same period, Doczy.ai™ has helped clients achieve approximately 330 million dollars in cumulative direct and indirect savings.The 99% accuracy rate represents significant improvement over the approximately 55% accuracy of rules-based systems and far exceeds manual processing, which is typically affected by fatigue and human error. The 97% reduction in manual processing time translates directly to cost savings and enables organizations to reallocate human resources to higher-value activities that require judgment and strategic thinking.

A use case in action: Business process automation for health plans

For health plans, Doczy.ai™ provides a powerful solution to automate and improve contract management across the entire lifecycle. It ingests existing contracts in both paper and digital formats, integrates with contract management systems such as Coupa and Icertis, and processes new contracts and amendments as they’re executed. It then creates a centralized metadata repository that feeds directly into downstream systems, enabling end-to-end business process automation.This automation unlocks critical capabilities: Organizations can continuously analyze and improve contract terms, identifying opportunities to improve financial performance and operational efficiency. The architecture feeds accurate, up-to-date contract data directly into claims systems, automating the configuration process that previously required manual translation of reimbursement terms and removing manual data entry, configuration errors, and delays. Additionally, the platform helps maintain claim payment accuracy by assessing payments against contract terms, identifying discrepancies, and flagging potential overpayments or underpayments before they occur.By automating manual processes, health plans can adapt quickly to new contract terms and regulatory requirements. The intelligent dashboards and actionable insights provided by Doczy.ai™ enable decision-makers to understand contract performance, identify trends, and take proactive action to optimize financial outcomes.

Getting started with Doczy.ai™

Organizations interested in using Doczy.ai™ to transform document processing and contract management can engage with AArete to discuss their specific use cases and requirements. AArete offers the platform as a Software as a Service (SaaS) solution, enabling rapid deployment without significant infrastructure investment. AArete’s team of experts will configure this solution for your specific document types, domain terminology, and business processes, supporting maximum value from day one.

Conclusion

The challenge of unlocking data from unstructured documents is a major hurdle for many businesses, particularly in healthcare and financial services where contracts and agreements govern critical operational and financial relationships. By embracing intelligent document intelligence on AWS, organizations can solve this long-standing operational challenge and unlock a new frontier of strategic advantage, turning their data into their most valuable asset.

Built on a sophisticated architecture that orchestrates Amazon Cognito, Amazon S3, AWS Lambda, Amazon Textract, Amazon Elastic Container Service (Amazon ECS), Amazon Bedrock, Amazon CloudWatch, and AWS Secrets Manager, Doczy.ai™ demonstrates how modern cloud services can solve complex document-heavy business problems. Its advanced hybrid smart chunking, dual clustering, and prompt optimization techniques form the core of a patented contract intelligence engine.

Doczy.ai™ delivers tangible impact, processing up to 250,000 contract documents per week with 99% accuracy, reducing manual processing time by 97%, and helping clients unlock roughly 330 million dollars in cumulative savings over 22 months. By embracing this intelligent document processing, organizations can turn contracts into a strategic data asset, improving efficiency, accuracy, and profitability while freeing teams to focus on higher-value work.

To learn more about how AArete and Doczy.ai™ can help your organization transform document processing and unlock the value of your unstructured data, visit the AArete website.

About the authors

Multi-Region event-driven failover architecture with Amazon EventBridge and Route 53

Post Syndicated from Napoleone Capasso original https://aws.amazon.com/blogs/compute/multi-region-event-driven-failover-architecture-with-amazon-eventbridge-and-route-53/

Multi-Region Event-Driven Failover Architecture with Amazon EventBridge and Route 53

Event-driven architectures enable applications to respond to events in real-time, providing scalability and loose coupling between components. However, ensuring high availability across multiple AWS regions requires careful design of failover mechanisms. This post demonstrates how to build a resilient multi-region event-driven architecture using Amazon EventBridge, Amazon API Gateway, and Amazon Route 53 health-based failover.

Overview

Organizations building event-driven applications need to achieve high availability and disaster recovery capabilities. This architecture provides automatic failover between AWS regions while maintaining regional independence for event processing. The solution uses Amazon Route 53 health checks to monitor regional Amazon API Gateway endpoints and automatically routes traffic to healthy regions without manual intervention.

The architecture delivers several key benefits. Regional independence reduces latency by processing events in the same region where they originate. Amazon DynamoDB global tables provide automatic data replication across regions, ensuring data availability during regional failures. The solution provides robust failover capabilities while maintaining architectural simplicity.

Organizations with strict availability requirements can find this solution particularly valuable. All event processing remains within AWS regions, and failover occurs automatically based on health check results. The architecture supports both planned maintenance windows and unplanned regional outages, providing flexibility for operational needs.

Solution overview

The solution implements an active-passive multi-region architecture where events flow through Amazon API Gateway to regional Amazon EventBridge buses. Amazon Route 53 health checks monitor the primary region and automatically route traffic to the secondary region during failures. Each region processes events independently, while Amazon DynamoDB Global Tables replicate data across regions.

The following diagram provides an overview of the solution:

The above diagram depicts the multi-region architecture running across two AWS regions. The Route 53 DNS service serves as the main entry point for the application, with health checks monitoring both regions. Each region contains an identical stack with Amazon API Gateway, Amazon EventBridge, Amazon SQS, and AWS Lambda. The Amazon DynamoDB Global Table replicates data between regions automatically.

Solution deployment

To deploy this solution, follow the instructions in the GitHub repository and clone the repository. The solution deploys in two AWS regions. Ensure valid SSL certificates exist in AWS Certificate Manager (ACM) in both regions for the custom domain.

Prerequisites

For this walkthrough, the following resources are needed:

  • AWS Account: An AWS account with permissions to create and manage Amazon API Gateway, Amazon EventBridge, Amazon SQS, AWS Lambda, Amazon DynamoDB, Amazon Route 53, AWS IAM, and AWS CloudFormation resources
  • AWS Serverless Application Model (SAM): The AWS SAM CLI installed, as the templates use the SAM transform for Lambda and API Gateway resource definitions
  • Domain Name: A registered domain with a Route 53 hosted zone- SSL Certificates: ACM certificates for the custom domain in both deployment regions
  • AWS CLI: The AWS CLI installed and configured with credentials for the target AWS account
  • Region Selection: Two AWS regions for deployment

Walkthrough

The AWS CloudFormation templates from the sample GitHub repository create a secure, multi-region architecture that provides automatic failover for event-driven applications. The templates provision regional API Gateway endpoints, EventBridge buses, SQS queues, Lambda functions, and an Amazon DynamoDB Global Table. The solution establishes health monitoring through Route 53 health checks and configures DNS failover routing. The templates use AWS Serverless Application Model (SAM) transform to simplify Lambda and API Gateway resource definitions.

Step 1: Deploy the primary stack

The primary stack creates the foundational resources in the primary region. This includes the Amazon EventBridge bus, Amazon API Gateway with custom domain, health check, AWS Lambda function, Amazon SQS queue, and Amazon DynamoDB Global Table. The stack creates an EventBridge bus that receives events from API Gateway:

EventBus: 
Type: AWS::Events::EventBus 
Properties: 
Name: !Ref EventBusName

The API Gateway uses AWS service integration to forward events directly to EventBridge:

x-amazon-apigateway-integration: 
type: "aws" 
uri: !Sub "arn:aws:apigateway:${AWS::Region}:events:path//" 
credentials: !GetAtt ApiGatewayEventBridgeRole.Arn 
httpMethod: "POST"

The health check monitors the API Gateway endpoint to determine regional availability:

DomainHealthCheck: 
Type: AWS::Route53::HealthCheck 
Properties: 
HealthCheckConfig: 
Type: HTTPS 
ResourcePath: /Prod/health FullyQualified
DomainName: !Sub ${Api}.execute-api.${AWS::Region}.amazonaws.com 
Port: 443 
RequestInterval: 30 
FailureThreshold: 3

The Route 53 DNS record configures failover routing with the PRIMARY designation:

ApiDnsRecord:
Type: AWS::Route53::RecordSet
Properties:
HostedZoneId: !Ref HostedZoneId
Name: !Ref CustomDomainName
Type: A
SetIdentifier: primary-region
Failover: PRIMARY
HealthCheckId: !Ref DomainHealthCheck

The DynamoDB Global Table creates replicas in both regions:

DataTable: 
Type: AWS::DynamoDB::GlobalTable 
Properties: 
BillingMode: PAY_PER_REQUEST 
Replicas: 
- Region: !Ref AWS::Region 
- Region: !Ref SecondaryRegion

Note the `DataTableName` output value for use in the secondary stack deployment. The `CustomDomainURL` output provides the endpoint to invoke the solution.

Step 2: Deploy the secondary stack

The secondary stack creates identical resources in the secondary region , except for the Amazon DynamoDB table which references the existing Global Table. The secondary stack creates its own Amazon EventBridge bus, Amazon API Gateway, health check, AWS Lambda function, and Amazon SQS queue. The Route 53 DNS record uses the SECONDARY designation

Step 3: Event processing flow

Events flow through the processing pipeline in each region. API Gateway receives events and forwards them to EventBridge using the PutEvents API. EventBridge evaluates event rules and routes matching events to SQS queues. Lambda functions poll the SQS queues and process events in batches. AWS Lambda writes processed data to the DynamoDB Global Table, which replicates across regions.

The Lambda function processes events from the queue and writes to DynamoDB:

def handler(event, context): 
for record in event.get('Records', []): 
body = json.loads(record['body']) 
detail = body.get('detail', {}) 
event_id = body.get('id', '') 
item = { 'id': event_id, 'detail': detail, 'timestamp': datetime.utcnow().isoformat() } 
table.put_item(Item=item)

Testing

Fetch the custom domain URL and test it by sending an event:

curl -X POST https://api.example.com \-H "Content-Type: application/json" \ -d '{ "Detail": { "IsHelloWorldExample": "true" }, "DetailType": "POSTED", "Source": "demo.event" }' -v

The response includes an `X-Region` header indicating which region processed the request. Under normal conditions, this shows the primary region.

To test failover:

  1. Remove the base path mapping for the primary region:
aws apigateway delete-base-path-mapping \ --domain-name api.example.com \ --base-path '(none)' \ --region {primary-region}
  1. Delete the primary API Gateway stage:

aws apigateway delete-stage \ --rest-api-id <primary-api-id> \ --stage-name Prod \ --region {primary-region}

  1. Wait 2-3 minutes for the health check to fail. The Route 53 health check performs checks every 30 seconds with a failure threshold of 3, requiring 90 seconds to detect the failure.
  2. Send another request to the API endpoint:
curl -X POST https://api.example.com \-H "Content-Type: application/json" \ -d '{ "Detail": { "IsHelloWorldExample": "true" }, "DetailType": "POSTED", "Source": "demo.event" }' -v
  1. Verify the failover: The `X-Region` header now shows the secondary region, confirming successful failover.

Verify event processing in the secondary region:

  1. Check the Lambda logs for successful processing:

aws logs tail /aws/lambda/<secondary-lambda-name> --region {secondary region}

You should see log entries similar to:

Processing message: 
{"version":"0",
"id":"abc12345-...",
"source":"demo.event",
"detail-type":"POSTED",...} 
Event Source: demo.event
Detail Type: POSTED
Successfully wrote item to DynamoDB: abc12345-... 
Successfully read item from DynamoDB: 
{'id': 'abc12345-...', 
'source': 'demo.event', 
'detailType': 'POSTED', 
'detail': 
{'data': {'IsHelloWorldExample': 'true'}, 
...}, 
'timestamp': '2025-01-15T18:30:00.000000', 
'processed': True}
  1. Verify the data in Amazon DynamoDB:

aws dynamodb scan \ --table-name <table-name> \ --region {secondary region}```

The scan results should include items with the event details:

{ "Items": 
[ { "id": {"S": "abc12345-..."}, 
"source": {"S": "demo.event"}, 
"detailType": {"S": "POSTED"},
"detail": 
{"M": {"data": 
{"M": 
{"IsHelloWorldExample": 
{"S": "true"}}}}}, 
"timestamp": {"S": "2025-01-15T18:30:00.000000"},
"processed": {"BOOL": true} } ], 
"Count": 1 }
  1. Restore the primary region – recreate the stage:

aws apigateway create-stage \ --rest-api-id <primary-api-id> \ --stage-name Prod \ --deployment-id <deployment-id> \ --region {primary region}

  1. Restore the primary region – recreate the base path mapping:

aws apigateway create-base-path-mapping \ --domain-name api.example.com \ --rest-api-id <primary-api-id> \ --stage Prod \ --region {primary region}

You can find the “deployment-id” by running: aws apigateway get-deployments \ --rest-api-id <primary-api-id> \ --region {primary region}

After 2-3 minutes, the health check passes and Route 53 routes traffic back to the primary region.

Cleanup

To remove the solution and avoid ongoing charges, delete the CloudFormation stacks in the correct order. Delete the secondary stack first, then the primary stack. This order is important because the Amazon DynamoDB Global Table is owned by the primary stack. Warning: Deleting these stacks permanently removes all resources including the Amazon DynamoDB global table and any event data stored in it. Back up any data you need before proceeding. This action cannot be undone. The following resources incur costs while deployed:

  • Amazon API Gateway (REST API)
  • Amazon Route 53 health checks and DNS records
  • Amazon DynamoDB global table (with cross-region replication)
  • AWS Lambda function invocations and duration
  • Amazon SQS queue operations
  • Amazon CloudWatch Logs storage

Delete the secondary stack:

aws cloudformation delete-stack --stack-name secondary-stack --region {secondary region}

Wait for the secondary stack deletion to complete:

aws cloudformation wait stack-delete-complete --stack-name secondary-stack --region {secondary region}

Delete the primary stack:

aws cloudformation delete-stack --stack-name primary-stack --region {primary region}

Wait for the primary stack deletion to complete:

aws cloudformation wait stack-delete-complete --stack-name primary-stack --region {primary region}

This removes all resources including the Amazon EventBridge buses, Amazon API Gateways, AWS Lambda functions, Amazon SQS queues, Amazon DynamoDB Global Table, Amazon Route 53 health checks, DNS records and IAM roles.

Conclusion

This post demonstrates how to establish a resilient multi-region architecture for event-driven applications using Amazon EventBridge, Amazon API Gateway, and Amazon Route 53. The solution uses Route 53 health-based failover, a powerful capability that automatically routes traffic to healthy regions based on health check results. This architecture significantly enhances application availability by providing automatic failover during regional outages while maintaining regional independence for event processing.

Building a scalable user search layer on top of Amazon Cognito

Post Syndicated from Philip Chen original https://aws.amazon.com/blogs/architecture/building-a-scalable-user-search-layer-on-top-of-amazon-cognito/

Imagine a teammate who needs to find a user across thousands of accounts with only a partial email address, a last name, and a known access level. How quickly can your team respond? If your use case involves straightforward searches on standard Amazon Cognito attributes, the built-in ListUsers API is likely all you need. But for advanced scenarios involving custom attributes, fuzzy matching, complex filtering, and sub-second response times, a dedicated search layer is the right investment.

Amazon Cognito provides robust user authentication and management capabilities for modern applications. As applications scale, development teams typically implement advanced search functionality to find users by partial email match, segment group membership, or audit across multiple custom attributes.

In this post, we show how to build a comprehensive scalable user search layer on top of Amazon Cognito using AWS Lambda, Amazon DynamoDB, and Amazon OpenSearch Service.

Solution overview

This solution extends Amazon Cognito with advanced search capabilities using AWS Lambda, Amazon DynamoDB, and Amazon OpenSearch Serverless.

Key capabilities:

  • Multiple search types: Exact match, prefix match, and fuzzy search
  • Complex filtering: Query across email, phone, groups, and registration date simultaneously
  • High performance: Sub-second response times at any scale
  • Automatic synchronization: Real-time updates as users authenticate or update profiles
  • API-driven: RESTful API with pagination support

The architecture uses Cognito Lambda triggers to capture user data during authentication, stores it in DynamoDB, and indexes it in OpenSearch Serverless through DynamoDB Streams. The following architecture diagram illustrates how these components work together.

Figure 1: Solution architecture for Searchable Cognito Users

Walkthrough

The solution architecture demonstrates two flows: Ingestion flow and Search flow.

Ingestion flow

The ingestion flow captures and indexes user data through two paths: Cognito Lambda triggers and AWS CloudTrail. Together, these paths maintain synchronization between the search index and Cognito without requiring manual intervention or scheduled batch jobs.

1. Cognito Lambda triggers

This path captures user data during authentication events using a Cognito trigger Lambda function that handles two trigger types: Post-confirmation and Pre-token generation. The post-confirmation trigger creates the initial user record on sign-up, while the pre-token generation trigger tracks login activity and app client information on each subsequent authentication. The pre-token generation trigger also provides access to the user’s group membership in the event payload, which is indexed as a searchable field. The flow operates through the following steps:

  1. Client initiates sign-up or login — User submits authentication request to Amazon Cognito.
  2. Post-confirmation trigger — On sign-up, Cognito invokes the Cognito trigger Lambda which creates the initial user record in the DynamoDB user table with profile attributes (email, name, groups).
  3. Pre-token generation trigger — On each login, Cognito invokes the Cognito trigger Lambda which updates the user’s login timestamp and app client information in the DynamoDB user table.
  4. Stream processing — DynamoDB Streams detects the new or updated record and triggers the OSS ingest Lambda.
  5. Index updated — OSS ingest Lambda processes the stream event and indexes the user data in OpenSearch Serverless.

Note: The Cognito Lambda triggers are deployed in a VPC. Cognito enforces a 5-second timeout on trigger functions. If you’re extending these triggers with additional functionality or already using post-confirmation or pre-token generation triggers, ensure the combined execution time stays well within this limit. Consider provisioned concurrency if cold starts are a concern.

Figure 2: User Data Ingestion via Cognito Lambda Triggers

2. CloudTrail

This path captures admin-initiated user changes that occur outside the authentication flow, such as creating users using the Cognito console or CLI. These actions don’t trigger Cognito Lambda triggers, so CloudTrail and EventBridge bridge the gap. The flow operates through the following steps:

  1. Admin action performed — User performs an admin action in Amazon Cognito (for example, create user, update attributes, add to group, disable user).
  2. API call logged — AWS CloudTrail captures the Cognito admin API call.
  3. EventBridge rule matched — An Amazon EventBridge rule matches the Cognito admin event.
  4. CloudTrail event Lambda invoked — EventBridge invokes the CloudTrail event consumption Lambda, which reads the current user state from Cognito and upserts the profile in the DynamoDB user table.
  5. Stream change event — DynamoDB Streams emits the change event.
  6. Invoke OSS Lambda — The stream event triggers the OSS ingest Lambda.
  7. Index user data — OSS ingest Lambda indexes the updated user data in OpenSearch Serverless.

Figure 3: User Data Ingestion via CloudTrail

Figure 4: Data model for indexed user attributes in Amazon DynamoDB

Search flow

With the search flow, authorized users can query the indexed user directory:

  1. Query submission — Authenticated user submits search query through the UI.
  2. Request validation — API Gateway receives the request with the Cognito JWT token and validates it using the Cognito authorizer.
  3. Search execution — Upon successful validation, the search Lambda function is invoked with the search parameters.
  4. OpenSearch query — Lambda assumes a read-only role for OpenSearch Service access and executes the query against the OpenSearch Serverless index.
  5. Results returned — Lambda formats and returns the query results to the frontend, where the UI displays them in a paginated format.

Figure 5: Search Flow Sequence Diagram

Figure 6: Demo UI user search integration on multiple properties

Figure 7: Demo UI user search integration on auto-suggest

Try it yourself

Ready to see this solution in action? The repository includes everything you need to deploy a complete working implementation in your own AWS environment.

The source code for this solution is available on GitHub at: https://github.com/aws-samples/sample-user-search-layer-for-cognito.

The repository includes everything you need: AWS CDK infrastructure code, Lambda function implementations, a React frontend, and documentation. You can have a fully functional searchable user directory running in your account in under 20 minutes. When you’re finished testing, clean up all resources to avoid ongoing charges.

Conclusion

In this post, you learned how to extend Amazon Cognito with advanced search capabilities. By combining OpenSearch Serverless, DynamoDB Streams, and Lambda functions, you can build a scalable, event-driven architecture that automatically maintains a searchable user directory with sub-second query performance.

This pattern unlocks powerful use cases: support teams can quickly locate users across thousands of accounts, administrators can segment users by group membership for targeted communications, and compliance teams can audit user attributes with complex filtering.

To dive deeper into the AWS services powering this solution:

About the authors

Simplifying policy management with URL and Domain Category filtering on AWS Network Firewall

Post Syndicated from Lawton Pittenger original https://aws.amazon.com/blogs/security/simplifying-policy-management-with-url-and-domain-category-filtering-on-aws-network-firewall/

Network administrators face a persistent challenge: maintaining domain blocklists and allowlists that keep pace with the internet. New websites and services emerge daily, and keeping these lists current requires constant manual updates that leave gaps in coverage. This challenge intensifies when managing access to rapidly evolving categories like AI services, where new tools launch on a regular basis.

AWS Network Firewall is a managed, stateful network firewall and intrusion detection and prevention service for fine-grained control of your virtual private cloud (VPC) network traffic. With URL and domain category filtering, security teams can use predefined categories to control access instead of managing individual domains. AWS-managed URL and domain categories stay current automatically as new domains are registered, removing the need for manual list maintenance.

This feature is especially useful for organizations navigating AI governance. Instead of manually tracking every new AI service, you can control access to the entire Artificial Intelligence and Machine Learning category while creating exceptions for approved services. The same approach works for social media, streaming sites, gambling, and dozens of other categories, all with built-in audit trails for compliance reporting.

In this post, we walk through URL and domain category filtering configurations for AWS Network Firewall, from basic rules to exception handling and monitoring strategies that give you visibility into how your workloads interact with external services.

Streamlined policy management with predefined categories

With URL and domain category filtering, you control website access using predefined categories instead of individually specifying sites in a domain list rule group. You can select from AWS-managed categories such as Social Networking, Gambling, or Artificial Intelligence and Machine Learning to implement and maintain filtering policies. AWS keeps these categories current automatically, so you don’t need to update firewall policies when new domains are registered.

Network Firewall offers two category filtering options. Domain category filters by domain name using the TLS Server Name Indication (SNI) field, with no decryption required. URL category filters by the full URL path, which requires TLS inspection for HTTPS traffic. To keep things straightforward, this post focuses on domain category filtering. To set up URL category filtering with TLS inspection, see Creating a TLS inspection configuration in Network Firewall.

Prerequisites

To follow the steps in this post, start by making sure that you have the following prerequisites in place:

  1. An existing Network Firewall deployment: This walkthrough assumes you have an existing Network Firewall deployment to filter egress traffic flows from your Amazon Virtual Private Cloud (Amazon VPC) in place. If you aren’t already using Network Firewall, see Getting started with AWS Network Firewall to set up your firewall before proceeding.
  2. The HOME_NET variable set correctly at the firewall policy level: The rules in this post use the $HOME_NET variable to scope traffic to your internal network. In the AWS Management Console for Amazon VPC, select your firewall policy under the Firewall policies tab, select the Details tab, and check the policy variables section under HOME_NET variable override values. We recommend setting this to all RFC 1918 private IP address ranges: 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16. When you set $HOME_NET at the policy level, all rule groups associated with that policy inherit the value automatically. Network Firewall automatically maps $EXTERNAL_NET to the inverse of $HOME_NET, so configuring HOME_NET correctly also configures $EXTERNAL_NET.
Figure 1: Firewall policy details tab showing the HOME_NET variable override values set to RFC 1918 private IP address ranges

Figure 1: Firewall policy details tab showing the HOME_NET variable override values set to RFC 1918 private IP address ranges

Create a category rule using the console rule builder

To get started quickly, you can create a domain category rule using the console’s built-in rule builder. In this example, we create a single alert rule for the Artificial Intelligence and Machine Learning category.

  1. Open the AWS Management Console, search for and open the Amazon VPC console.
  2. In the left navigation, scroll to Network Firewall and select Rule groups.
  3. Choose Create rule group.
  4. For Rule group type, select Stateful rule group.
  5. For Rule group format, select Standard stateful rules.
  6. For Rule evaluation order, select Strict order. Choose Next.
    Figure 2: Create Network Firewall rule group page showing Stateful rule group type, Standard stateful rules format, and Strict order evaluation selected

    Figure 2: Create Network Firewall rule group page showing Stateful rule group type, Standard stateful rules format, and Strict order evaluation selected

  7. Enter Domain-Category-Rules for the Name, Domain Category Rules for the Description, and 50 for the Capacity. Choose Next.
  8. In the rule group editor, select the Category Matching radio button.
  9. Under Category Matching, select Match all selected categories.
  10. Under AWS category type, select Domain Category from the dropdown.
  11. Under Categories, select Artificial Intelligence and Machine Learning.
  12. For Protocol, select TLS.
  13. For Source, select Custom, then enter $HOME_NET in the dialog box.
  14. Set the Destination IP to Any.
  15. For Action, select Alert.
  16. Choose Add rule to add this rule to the rule group. Choose Next.
    Figure 3: Completed category matching rule showing TLS protocol, $HOME_NET source, Any destination, and Alert action added to the rule group

    Figure 3: Completed category matching rule showing TLS protocol, $HOME_NET source, Any destination, and Alert action added to the rule group

  17. Under Customer managed key, leave the default setting (Customize encryption settings should remain unchecked).
  18. Under Add tags – optional, leave the default setting of no tags.
  19. Choose Next, then Create rule group.

This rule generates an alert log entry each time a connection matches a domain in the Artificial Intelligence and Machine Learning category. It doesn’t block traffic. To block traffic, change the action to Drop or Reject in step 15.

Creating the same rule using Suricata compatible rule strings

The console rule builder is a quick way to get started, but we recommend using Suricata compatible rule strings for production deployments. Suricata rules give you full control over rule options, make rules straightforward to copy, edit, share, and back up, and support the majority of the Suricata engine. For more information, see Limitations and caveats for stateful rules in AWS Network Firewall.

The following walkthrough creates the same alert rule you built with the console rule builder, this time using a Suricata rule string.

In the Amazon VPC console, navigate to Network Firewall, then select Network Firewall rule groups.

  1. Choose Create rule group.
  2. For Rule group type, select Stateful rule group.
  3. For Rule group format, select Suricata compatible rule string.
  4. For Rule evaluation order, select Strict order. Choose Next.
    Figure 4: Create Network Firewall rule group page showing Stateful rule group type, Suricata compatible rule string format, and Strict order evaluation selected

    Figure 4: Create Network Firewall rule group page showing Stateful rule group type, Suricata compatible rule string format, and Strict order evaluation selected

  5. Enter Suricata-Domain-Category-Rules for the Name, Suricata Domain Category Rules for the Description, and 50 for the Capacity. Choose Next.
  6. Leave the Rule variables section empty. The $HOME_NET variable is inherited from the firewall policy, as configured in the prerequisites.
  7. Leave IP set references empty.
  8. Paste the following rule into the Suricata compatible rule string editor: 
    alert tls $HOME_NET any -> $EXTERNAL_NET any (msg:"Artificial Intelligence and Machine Learning Category"; aws_domain_category:Artificial Intelligence and Machine Learning; sid:1000001;)

  9. Choose Next.
    Figure 5: Suricata compatible rule string editor with the domain category alert rule pasted in and the rule variables section left empty

    Figure 5: Suricata compatible rule string editor with the domain category alert rule pasted in and the rule variables section left empty

  10. Under Customer managed key, leave the default setting (Customize encryption settings should remain unchecked).
  11. Under Add tags – optional, leave the default setting of no tags. Choose Next.
  12. Choose Create rule group.
  13. After creating the rule group, return to your firewall policy and add it under Stateful rule groups. We recommend associating new rule groups in a development or test environment first to validate behavior before deploying to production.

The following table explains each component of this rule:

alert Action: generate an alert log entry when the rule matches. Other actions include pass, drop, and reject.
tls Protocol: inspect TLS traffic, matching against the SNI field in the TLS Client Hello.
$HOME_NET any -> $EXTERNAL_NET any Source and destination: match traffic from any internal IP address (HOME_NET) and port to any external IP address (EXTERNAL_NET) and port. The HOME_NET variable defines your internal network ranges, and the EXTERNAL_NET variable is automatically set to the inverse.
msg:”Artificial Intelligence and Machine Learning Category” The message written to the alert log when this rule is triggered.
aws_domain_category:Artificial Intelligence and Machine Learning The AWS-managed domain category to match against. The firewall looks up the destination domain in the category database and matches if the domain belongs to this category.
sid:1000001 A unique signature ID for this rule. Each rule in a rule group must have a unique SID.

Managing exceptions for approved services

You can manage exceptions to keep business-critical websites accessible. For example, say you need to allow access to OpenAI while blocking all other AI and ML traffic. To do this, return to the Suricata-Domain-Category-Rules rule group you created earlier and replace the basic alert rule with the following ruleset. Select the Suricata-Domain-Category-Rules rule group, under the Rules section, choose Edit.

Figure 6: Selecting Suricata-Domain-Category-Rules rule group to edit with new rules

Figure 6: Selecting Suricata-Domain-Category-Rules rule group to edit with new rules

Paste in the following rules and choose Save rule group.

# Allow OpenAI (TLS)
pass tls $HOME_NET any -> $EXTERNAL_NET any (tls.sni; dotprefix; content:".openai.com"; nocase; endswith; flow:to_server; alert; msg:"Allow OpenAI over TLS"; sid:1000001;)

# Allow OpenAI (HTTP)
pass http $HOME_NET any -> $EXTERNAL_NET any (http.host; dotprefix; content:".openai.com"; nocase; endswith; flow:to_server; alert; msg:"Allow OpenAI over HTTP"; sid:1000002;)

# Block all other AI/ML category traffic (TLS)
reject tls $HOME_NET any -> $EXTERNAL_NET any (msg:"Block non-approved AI/ML sites over TLS"; aws_domain_category:Artificial Intelligence and Machine Learning; flow:to_server; alert; sid:1000003;)

# Block all other AI/ML category traffic (HTTP)
reject http $HOME_NET any -> $EXTERNAL_NET any (msg:"Block non-approved AI/ML sites over HTTP"; aws_url_category:Artificial Intelligence and Machine Learning; flow:to_server; alert; sid:1000004;)

Figure 7: Suricata compatible rule string editor with the exception-based ruleset containing pass rules for OpenAI and reject rules for the AI/ML category

Figure 7: Suricata compatible rule string editor with the exception-based ruleset containing pass rules for OpenAI and reject rules for the AI/ML category

With strict order evaluation, the firewall evaluates rules in the order you define them. The pass rules for OpenAI appear first, so matching traffic is allowed before the broader category block rules run.

To verify the rules are working as expected, test from a host that routes traffic through your network firewall. These commands suppress the response body and check the exit code of the curl request. If curl completes a TCP connection, it prints CONNECTION ALLOWED. If the firewall resets the connection, curl exits with a non-zero code and prints CONNECTION BLOCKED.

A request to openai.com should succeed because it matches the pass rule:

curl -s -o /dev/null https://openai.com && echo "CONNECTION ALLOWED" || echo "CONNECTION BLOCKED"

Result: CONNECTION ALLOWED

A request to chat.mistral.ai should be rejected because it matches the broader AI/ML category block rule:

curl -s -o /dev/null https://chat.mistral.ai && echo "CONNECTION ALLOWED" || echo "CONNECTION BLOCKED"

Result: CONNECTION BLOCKED

How to monitor category usage

When you add a domain category rule to your firewall policy, Network Firewall performs a category lookup for every connection that matches the rule’s protocol and IP specifications. The rules in this post match on $HOME_NET any -> $EXTERNAL_NET any, which means the firewall looks up the category for all outbound traffic originating from your internal network. This is why it’s important to have the $HOME_NET variable configured correctly at the firewall policy level. With this configuration, a single category rule is enough for category metadata to appear in your firewall logs across all matching connections, not just connections that match the specific category in your rule.

Each log entry includes an aws_category field containing a JSON array of all categories the destination domain belongs to. A single domain can map to multiple categories. For example, a request to chat.mistral.ai produces a log entry with “aws_category": "[\"Social Networking\",\"Artificial Intelligence and Machine Learning\"]” because that domain belongs to both categories.

You can access firewall logs through Amazon CloudWatch, Amazon Simple Storage Service (Amazon S3), and Amazon Data Firehose. These logs show which categorized websites your workloads access, helping you track usage patterns and enforce acceptable use policies.

The following sample log entry shows what a blocked request to chat.mistral.ai looks like using the exception-based rules from the previous section. The alert.signature field contains the rule’s msg value, and the aws_category field lists all categories the destination domain belongs to:

{ 

     "firewall_name": "egress-and-east-west-firewall", 

     "availability_zone": "us-east-1a", 

     "event_timestamp": "1775599146", 

     "event": { 

          "aws_category": "[\"Social Networking\",\"Artificial Intelligence and Machine Learning\"]", 

          "tx_id": 0, 

          "app_proto": "tls", 

          "src_ip": "10.1.1.100", 

          "src_port": 58664, 

          "event_type": "alert", 

          "alert": { 

                    "severity": 3, 

                    "signature_id": 1000003, 

                    "rev": 1, "signature": 

                    "Block non-approved AI/ML sites over TLS", 

                    "action": "blocked", 

                    "category": "" 

          }, 

          "flow_id": 763153567844057, 

          "dest_ip": "172.66.2.203", 

          "proto": "TCP", 

          "verdict": { 

                    "action": "drop", 

                    "reject-target": "to_client", 

                    "reject": [ 

                         "tcp-reset" 

                    ] 

          }, 

          "tls": { 

               "sni": "chat.mistral.ai", 

               "version": "UNDETERMINED" 

          }, 

          "dest_port": 443, 

          "pkt_src": "geneve encapsulation", 

          "timestamp": "2026-04-07T21:59:06.906761+0000", 

          "direction": "to_server" 

     } 

}

The aws_category field shows the domain belongs to both the “Social Networking” and “Artificial Intelligence and Machine Learning” categories. The verdict field confirms the connection was dropped with a TCP reset sent to the client.

Traffic that matches a pass rule with the alert keyword also generates a log entry with the aws_category field populated. For example, a connection to chat.openai.com that matches the OpenAI exception rule from the earlier section produces a log entry with alert.action set to “allowed” and the same category metadata. This means your queries capture both blocked and allowed traffic.

Querying logs with CloudWatch Logs Insights

If you send your firewall logs to Amazon CloudWatch Logs, you can use CloudWatch Logs Insights to analyze category traffic patterns. A single connection can generate multiple log entries (for example, a reject rule log and a default action log for the same flow), so the following queries deduplicate by flow_id to count each connection only once. Because a single domain can belong to multiple categories, results are grouped by category combination. For example, traffic to a domain categorized as both “Social Networking” and “Artificial Intelligence and Machine Learning” appears as a single combined entry.

To get started, navigate to the CloudWatch console. In the left navigation pane under Logs, select Logs Insights. Under Query scope, leave Log group name selected, then select your AWS Network Firewall alert logs log group. For the time window, we recommend starting with the default of 1 hour to keep the queries light. Enter each of the following queries into the editor and choose Run query to review the results. Note that CloudWatch Logs Insights queries incur charges based on the amount of data scanned. See Amazon CloudWatch pricing for details.

Most accessed categories

This query shows which category combinations your workloads connect to most frequently:

fields @timestamp, event.aws_category, event.flow_id
| filter ispresent(event.aws_category) and event.aws_category != "[]"
| stats latest(event.aws_category) as categories by event.flow_id
| stats count(*) as connections by categories
| sort connections desc
| limit 20

Figure 8: CloudWatch Logs Insights query results showing the most frequently accessed category combinations sorted by connection count

Figure 8: CloudWatch Logs Insights query results showing the most frequently accessed category combinations sorted by connection count

Least accessed categories

This query reverses the sort order to surface category combinations with the fewest connections, helping you identify categories that might not be relevant to your environment or that warrant further investigation:

fields @timestamp, event.aws_category, event.flow_id
| filter ispresent(event.aws_category) and event.aws_category != "[]"
| stats latest(event.aws_category) as categories by event.flow_id
| stats count(*) as connections by categories
| sort connections asc
| limit 20

Figure 9: CloudWatch Logs Insights query results showing the least frequently accessed category combinations sorted by connection count ascending

Figure 9: CloudWatch Logs Insights query results showing the least frequently accessed category combinations sorted by connection count ascending

Most accessed categories, allowed traffic only

The event.verdict.action field indicates the actual outcome of each connection:drop for blocked traffic and alert for allowed traffic. This query shows which category combinations have the most allowed connections:

fields @timestamp, event.aws_category, event.flow_id, event.verdict.action
| filter ispresent(event.aws_category) and event.aws_category != "[]"
| stats latest(event.aws_category) as categories, latest(event.verdict.action) as verdict by event.flow_id
| filter verdict = "alert"
| stats count(*) as connections by categories
| sort connections desc
| limit 20

Figure 10: CloudWatch Logs Insights query results showing the most accessed category combinations filtered to allowed traffic only

Figure 10: CloudWatch Logs Insights query results showing the most accessed category combinations filtered to allowed traffic only

Most accessed categories, blocked traffic only

The same query filtered to blocked connections. Change the verdict filter to drop:

fields @timestamp, event.aws_category, event.flow_id, event.verdict.action
| filter ispresent(event.aws_category) and event.aws_category != "[]"
| stats latest(event.aws_category) as categories, latest(event.verdict.action) as verdict by event.flow_id
| filter verdict = "drop"
| stats count(*) as connections by categories
| sort connections desc
| limit 20

Figure 11: CloudWatch Logs Insights query results showing the most accessed category combinations filtered to blocked traffic only

Figure 11: CloudWatch Logs Insights query results showing the most accessed category combinations filtered to blocked traffic only

Drill down into a specific category

This query uses a like filter to find all traffic where the aws_category field contains a specific category, regardless of what other categories the domain also belongs to. In this example, the query returns all domains your workloads have connected to that map to the Artificial Intelligence and Machine Learning category, broken down by domain and verdict. Replace the category name in the like filter to investigate any category.

fields @timestamp, event.tls.sni, event.aws_category, event.verdict.action, event.flow_id
| filter ispresent(event.aws_category) and event.aws_category like /Artificial Intelligence and Machine Learning/
| stats latest(event.tls.sni) as sni, latest(event.verdict.action) as verdict by event.flow_id
| stats count(*) as connections by sni, verdict
| sort connections desc
| limit 20

Figure 12: CloudWatch Logs Insights query results showing a drill down into the Artificial Intelligence and Machine Learning category with connections broken down by domain and verdict

Figure 12: CloudWatch Logs Insights query results showing a drill down into the Artificial Intelligence and Machine Learning category with connections broken down by domain and verdict

Bandwidth consumption by category

This query shows which category combinations consume the most egress bandwidth. It correlates flow logs (which contain byte counts) with alert logs (which contain category data) using the shared flow_id field. To run this query, select both your alert log group and your flow log group in CloudWatch Logs Insights.

fields @timestamp
| filter ispresent(event.netflow.bytes) or ispresent(event.aws_category)
| stats sum(event.netflow.bytes) as flowBytes, latest(event.aws_category) as categories by event.flow_id
| filter ispresent(categories) and categories != "[]"
| stats sum(flowBytes) as totalBytes by categories
| sort totalBytes desc
| limit 20

Figure 13: CloudWatch Logs Insights query results showing bandwidth consumption by category combination sorted by total bytes descending

Figure 13: CloudWatch Logs Insights query results showing bandwidth consumption by category combination sorted by total bytes descending

These queries help you identify which categories your workloads access by volume, surface blocked and allowed traffic patterns, and pinpoint where the bulk of your egress bandwidth is going.

Conclusion

In this post, you walked through how to set up URL and domain category filtering on AWS Network Firewall, from creating your first category rule using both the console rule builder and Suricata compatible rule strings, to managing exceptions for approved services and monitoring category traffic patterns with CloudWatch Logs Insights. With AWS-managed categories that stay current automatically, you can control access to broad classes of websites without maintaining individual domain lists, and the built-in aws_category log field gives you the visibility to track how your workloads interact with external services.

This feature is available in all AWS commercial regions where AWS Network Firewall is supported.

To learn more, visit the AWS Network Firewall product page and the feature documentation.

Lawton Pittenger

Lawton Pittenger

Lawton is a Worldwide Security Specialist Solutions Architect at AWS, based in New York City. He specializes in helping customers design and implement effective network security controls. At AWS, he works with customers at scale and collaborates closely with service teams to drive continuous improvement in security services based on customer needs and feedback. Outside of work, his interests include skateboarding, snowboarding, and spending time in nature.

Sofia Aluma

Sofia Aluma-Santos

Sofía is a Sr. Security Specialist leading Network Security Go-To-Market and strategy. She helps customers build scalable, secure, resilient networks.

Eric Fortenbery

Eric Fortenbery

Eric is an AWS Solutions Architect based in Atlanta, GA who helps EdTech customers architect secure, scalable platforms.

Mostafa Elkhouly

Mostafa Elkhouly

With over a decade of experience in networking technologies and security, I’m your go-to tech enthusiast! When I’m not jet-setting or tinkering with the latest gadgets, I thrive on empowering customers to harness the full potential of AWS services.

Migrating your Java applications to AWS Graviton using AWS Transform custom

Post Syndicated from Hahnara Hyun original https://aws.amazon.com/blogs/compute/migrating-your-java-applications-to-aws-graviton-using-aws-transform-custom/

For Java applications, modern JVMs like Amazon Corretto and OpenJDK are highly optimized for Arm64 and modern applications that are pure Java often require zero changes to run on Graviton. In many cases, applications aren’t fully modernized or purely Java and have a range of dependencies. When you’re responsible for migrating workloads, it’s helpful to use a systematic approach that surfaces issues, proposes solutions, and does the transformation work for you at scale.

That’s why we built the Java x86 to Graviton Migration transformation for AWS Transform custom (ATX). This is an AI-powered agent that analyzes your Java codebase, creates a migration plan, and executes the transformation—complete with version-controlled commits at every step. With ATX you can efficiently assess hundreds of Java applications simultaneously and quickly learn which applications require no changes and which ones need modifications. This streamlines the process of estimating the scope of effort, while also having suggested code updates before you even start.

ATX is available as a Kiro power, a VS Code extension, and an Agent Skill if you’d like to use it directly within other AI assistants to reduce context switching. While we will be using ATX to highlight how you can rapidly accelerate a Graviton migration, we have also published an open source Graviton universal skill based on the Agent Skills open standard so that you have the flexibility to use the skill natively within Kiro, Claude Code, Codex, or the platform of your choice.

AWS Graviton processors, based on the Arm64 architecture, can provide up to 40% better price performance over comparable x86-based instances for a wide variety of workloads. Now customers can use AI tools to quickly migrate workloads to Graviton.

The Java x86 to Graviton migration transformation

At a high level, we recommend customers finish any major version Java updates prior to migrating to Graviton and there’s a separate Java Version Upgrade transformation available for this use case. The Java x86 to Graviton Migration transformation requires a minimum of Java 8 and won’t incorporate Java version updates into the code changes.

The Java x86 to Graviton Migration completes multiple steps with work divided across multiple AI agents within the AWS Transform service, covering things like:

  • Native library analysis – Identifies Java Native Interface (JNI) dependencies and finds Arm64-compatible alternatives
  • Dependency updates – Updates libraries to versions with Arm64 support
  • Build configuration – Modifies Maven/Gradle configs for multi-architecture builds
  • Architecture-specific code – Refactors hard-coded x86 assumptions
  • Unit Test – Verifies compatibility at runtime given unit tests are in the project
  • Documentation – Creates migration notes and runbooks for your team

The agent automatically detects your Java version, manages runtime switching as needed during analysis, and handles much of the environment complexity for you such as multi-module project detection or Maven or Gradle auto-detection. Transformation completion times vary, but for many applications you can expect it to take roughly an hour (ATX works well with repos under 300K lines of code).

In this post, we:

  • Walk through the requirements for running the Java x86 to Graviton Migration transformation.
  • Help you familiarize yourself with ATX using a single Java application with Interactive Mode
  • Outline how to assess Graviton compatibility across the Java applications that you want to migrate to Graviton in a single batch and summarize the results with Campaign Mode.

By the end, you should have a good idea of how Java x86 to Graviton Migration transformation functions and have a summary of the expected code changes and dependency updates needed for each of your Java applications, along with version-controlled code updates.

Graviton transformation requirement

The Java x86 to Graviton migration transformation should run on an Arm64 machine.

The agent doesn’t just read your code, it builds, loads native libraries, and validates your application’s runtime behavior on Arm64. If you run the transformation on an x86 machine, the agent can identify compatibility issues but can’t execute build validation or run tests.

If you try to run on x86, you will see the following error message:

⚠  This transformation requires Arm64 architecture.    
Detected: x86_64        
Please run ATX on an Arm64 environment. See documentation for options.

To get started you need a Graviton instance or Apple Mac laptop running Arm64 with the ATX CLI, build tools, and Java JDKs that your project requires. The project source code should also be loaded locally onto the machine running the ATX CLI. Because Apple silicon is Arm64-based, it’s possible to build, load, and verify Arm64 based dependencies for a quick proof-of-concept. However, we recommend running the transformation in an environment that reflects what you plan to deploy in production to surface any potential OS level incompatibilities.

Requirements

Requirement Details
AWS Transform custom permissions AWS Identity and Access Management (IAM) policies for the Transform service (see Authentication docs)
Arm64 execution environment Amazon Elastic Compute Cloud (Amazon EC2) Graviton instance or Apple Silicon Mac. Running on x86 limits validation to static analysis only. Phase 3 (build/test) requires Arm64.
Node.js 20+ Required by the AWS Transform CLI. Use the official installer at nodejs.org/en/download. Package managers (dnf, yum) can install an older version.
Git AWS Transform custom uses local Git for version control during the transformation.
AWS Transform CLI Installed using the setup script (see Client Setup for the curl command).
Java build tooling A JDK (Arm64 build, e.g. Amazon Corretto or OpenJDK), Maven and/or Gradle as required by the target project. These are not optional for Java transformations. The agent needs them for dependency analysis, native library scanning, and build validation.

Running the Graviton transformation with Interactive Mode

With your code on an Arm64 environment and all the prerequisites for the transformation, we can begin the transformation.

Step 1: Navigate to Your Project and create or clone a git repo

cd /home/developer/workspace # Docker 
# or 
cd ~/workspace # AMI
git init

We recommend not pointing to the main branch of the repository of your application. You can work in a local git environment or create a separate branch. ATX needs the ability to commit changes as it iteratively transforms your code. The final decision on which commits are pushed is up to the developer.

Step 2: Launch ATX Interactive Mode

Enter the following command to launch ATX interactive mode.

atx

ATX starts in interactive mode:


To view available transformations, in a separate terminal enter:

atx custom def list > custom_list.txt

The AWS Managed transformations will be listed first, followed by User-created transformations that you’ve developed.

Step 3: Select the Graviton transformation

Enter the following into atx cli:

>AWS/early-access-java-x86-to-graviton

ATX will prompt you for next steps and your project details:

> Would you like to:+ c to abort or provide feedback)
1. View the entire transformation definition
2. View specific sections of the transformation definition
3. Apply this transformation to your code
4. Modify this transformation
> 3
> What is the file system path to the code repository where you want to apply this transformation?
> .

Note that because this is an AWS Managed Transformation, you can’t view the complete transformation definition or modify it. However, you can provide additional context customized to your use case. Keep in mind that the Transformation won’t make permanent changes to your code through the transformation process.

Step 4: Provide additional context

ATX might ask clarifying questions to tailor the transformation:

Step 5: Review the analysis

ATX analyzes your codebase:

Analyzing your codebase...[████████████████████████████████████████] 100%
✓ Parsed 847 Java files
✓ Analyzed 234 dependencies  
✓ Identified 12 native library usages
✓ Scanned build configuration
✓ Evaluated test coverage
Analysis complete. Preparing migration plan...

Step 6: Review the migration plan

ATX presents a detailed plan before making changes:

═══════════════════════════════════════════════════════════════
                    MIGRATION PLAN
═══════════════════════════════════════════════════════════════

I've analyzed your project and here's what I'll do:

PHASE 1: Dependency Updates (12 changes)
────────────────────────────────────────
 • Update netty-transport-native-epoll 4.1.85 → 4.1.100
   Reason: Arm64 support added in 4.1.94
   
 • Update snappy-java 1.1.8.4 → 1.1.10.5  
   Reason: Current version lacks Arm64 native binaries
   
 • Update lz4-java 1.7.1 → 1.8.0
   Reason: Arm64 optimization improvements
   
 • ... (9 more dependency updates)

PHASE 2: Build Configuration (3 changes)
────────────────────────────────────────
 • Add Arm64 classifier to native dependency declarations
 • Configure multi-architecture Docker build
 • Add Graviton-optimized JVM flags to startup scripts

PHASE 3: Code Modifications (2 changes)
────────────────────────────────────────
 • Refactor ProcessorArchitecture.java
   Current: Hard-coded x86_64 assumption
   Change: Add Arm64 detection logic
   
 • Update NativeLibraryLoader.java
   Current: Loads x86-specific .so files
   Change: Architecture-aware library loading

PHASE 4: Testing & Validation
────────────────────────────────────────
 • Run build and tests on Arm64
 • Generate migration report and documentation

═══════════════════════════════════════════════════════════════

? How would you like to proceed?
  ❯ Execute the full plan
    Execute phase by phase (review each)
    Modify the plan
    Ask questions about the plan
    Export plan and exit

Step 7: Ask questions, request changes

This is your chance to understand and adjust:

? Ask questions about the plan:

You: Why are you updating netty to 4.1.100? 

ATX: Good concern. Netty 4.1.94 was the first version with Arm64 
     native transport support, which is required for optimal 
     performance on Graviton. 
     
     However, I can adjust this:
     
     Option 1: Use 4.1.94 (minimum Arm64 support)
     Option 2: Use 4.1.97 (includes Arm64 bug fixes)
     Option 3: Keep 4.1.100 (latest stable with Arm64)
     
     Which would you prefer?

You: Let's go with 4.1.100.

Step 8: Execute the Transformation

When you’re ready:

? How would you like to proceed?
  ❯ Execute the full plan

Starting transformation...

Phase 1: Dependency Updates
───────────────────────────
[1/12] Updating netty-transport-native-epoll...
       ✓ Modified pom.xml
       ✓ Committed: "chore(deps): update netty to 4.1.97 for Arm64 support"

[2/12] Updating snappy-java...
       ✓ Modified pom.xml  
       ✓ Committed: "chore(deps): update snappy-java to 1.1.10.5 for Arm64"

... (progress continues)

Phase 2: Build Configuration
───────────────────────────
[1/3] Adding Arm64 classifiers...
      ✓ Modified pom.xml
      ✓ Committed: "build: add Arm64 native classifiers"

... (progress continues)

Phase 3: Code Modifications
───────────────────────────
[1/2] Refactoring ProcessorArchitecture.java...
      ✓ Modified src/main/java/com/example/util/ProcessorArchitecture.java
      ✓ Committed: "feat: add Arm64 architecture detection"

... (progress continues)

Phase 4: Validation
───────────────────
Running build... ✓
Running tests... ✓ (847 passed, 0 failed)

═══════════════════════════════════════════════════════════════
                 TRANSFORMATION COMPLETE
═══════════════════════════════════════════════════════════════

Summary:
 • 17 files modified
 • 3 files created  
 • 14 commits made
 • All tests passing

Generated Artifacts:
 • GRAVITON_MIGRATION_REPORT.html  - Full migration report
 • MIGRATION_RUNBOOK.md            - Deployment guide for your team
 • commit-log.txt                  - All commits with descriptions

Your code is now Graviton-ready!

After the transformation is complete, you can now performance test and load test on Graviton instances to configure your scaling policies or target thresholds to maximize price/performance on Graviton. For more guidance on performance testing, see the AWS Graviton Technical Guide.

What you get after transformation

Version-controlled history

Every logical change is a separate commit:

$ git log --oneline -10

a3f2b1c (HEAD) docs: add Graviton migration runbook
b82d4e5 test: add Arm64 architecture verification tests
c9a1f3d feat: add Arm64 architecture detection
d4e7c2a build: configure multi-arch Docker build
e5f8d1b build: add Arm64 native classifiers
f6a9e2c chore(deps): update lz4-java to 1.8.0
g7b0f3d chore(deps): update snappy-java to 1.1.10.5
h8c1a4e chore(deps): update netty to 4.1.97 for Arm64 support
...

Each commit is atomic and revertible. If something doesn’t work, you can git revert specific changes.

Migration report

A comprehensive markdown report covering:

  • What was changed and why
  • Dependencies that were updated
  • Code modifications with before and after diffs
  • Performance optimization recommendations

Migration runbook

A deployment guide for your team:

  • Pre-deployment checklist
  • JVM flags designed for Graviton
  • Monitoring and rollback procedures

Additional resources on migrating to Graviton on an infrastructure level can be found in the Transition Guide.

Assessing Graviton compatibility for multiple Java applications with Campaign Mode

When you’re ready to start migrating multiple applications, you might want to opt for an automated process that removes the manual effort of going back and forth with the transformation agent after each transformation step with campaign mode. The following command allows ATX CLI to go through a full transformation that you can check back in with after it’s completed. This limits the additional customization and context that you might want to provide the agent.

As mentioned in the first step of running a Graviton Transformation, the environment that the code is transformed in and decision of which commits are pulled into the main repo is up to the developer. Running in campaign mode across several applications doesn’t require accepting and pushing code changes. Therefore, this automated method is most useful when you want to gauge a high-level overview of effort required to migrate across several or even hundreds of applications.

atx custom def exec \
--code-repository-path /path/to/myapp \
--non-interactive \
--trust-all-tools \
--campaign  \
--repo-name myapp \
--add-repo

This command can be added into scripts, allowing further automations to be built into continuous integration and delivery (CI/CD) pipelines or scaling transformation jobs across several repos without manually entering prompts as previously shown through interactive mode.

The status of transformations running with campaign mode will be displayed in the AWS Transform Web UI. Setting up the Web UI is a prerequisite to running a transformation in campaign mode.

In addition to this view, if you run the transformation across multiple applications, you can generate a consolidated dashboard with an agent of your choice. Gather the transformation results into a centralized directory, then use the following prompt for example:

Analyze all Java application Graviton transformation summaries in <directory>/<path>/ and create a comprehensive dashboard that includes: 
 
1. Executive summary with key metrics (total apps, compatibility rate, code changes required) 
2. Application summary table with columns: Application name, Type, Java version, Dependencies count, Code changes, Compatibility %, Status 
3. Code changes analysis - which apps needed changes and why 
4. Dependency transformation analysis - common dependencies and their ARM64 status, any upgrades required 
5. Native library analysis - which apps use native libs and their compatibility 
6. Performance expectations - JWT/crypto improvements, general performance gains, cost-performance ratios 
7. JVM optimization patterns - common flags used across applications 
8. Build system patterns - Maven/Gradle usage, Docker multi-arch support 
9. Test results summary - pass/fail rates, pre-existing vs ARM64 issues 
10. Common libraries requiring changes (or note if none) 
11. Deployment readiness assessment 
12. Risk assessment with mitigation strategies 
13. Migration recommendations with phased approach 
14. Documentation summary - total docs created and their coverage 
 
Read graviton-validation/00-summary.md from each application subdirectory. Consolidate findings into a single comprehensive markdown dashboard with tables, metrics, and actionable insights. 
 
Focus on: compatibility rates, code change requirements, dependency issues, performance expectations, and migration readiness.

Keep in mind that agents might output outcomes of the migration that aren’t sourced from the transformation summaries. As a result, we recommend that you use the summary as a high-level estimate of the technical effort required for migrating to Graviton.

Conclusion

The AWS Transform custom Java x86 to Graviton Migration transformation alleviates the guesswork in Graviton migrations by using AI for dependency analysis, compatibility assessment, code refactoring, and runtime validation. Development teams can evaluate hundreds of Java applications simultaneously, with each transformation providing atomic version-controlled commits for straightforward rollback and clear change tracking. The tool offers two modes: 1) interactive mode for hands-on, application-by-application migration with developer review at each step, or 2) campaign mode for automated assessment across multiple applications. ATX converts unknown Graviton migration effort into defined requirements through automated compilation and runtime testing. This provides a more efficient way to evaluate workload compatibility and migrate to Graviton.

The Java x86 to Graviton Migration transformation is one of a range of pre-built AWS Managed Transformations but you can also create custom transformations unique to your own use case that can be scaled to drive migrations across your organization. Learn more on the AWS Transform custom website or documentation.

Resources

Automate data discovery and centralized management with AWS Glue Data Catalog

Post Syndicated from Ramakrishna Natarajan original https://aws.amazon.com/blogs/big-data/automate-data-discovery-and-centralized-management-with-aws-glue-data-catalog/

Managing sensitive data across sprawling data environments is hard. In this post, we show you how to tackle data discovery, classification, and governance across your databases, data warehouses, and object storage to regain visibility and control over your data landscape. As you build new features, products, and services, your data naturally spreads across multiple systems to meet immediate application and business needs. Different teams spin up their own data stores, and before long, you’re dealing with a complex web of repositories—often with limited visibility into what exists where. This data sprawl becomes most challenging when you must understand and protect your sensitive data. Security teams often struggle to maintain accurate inventories of data categorization and classification. Stakeholders demand comprehensive insights into data classification and processing activities, usually on tight deadlines, and keeping up-to-date data inventories becomes increasingly daunting as your data grows. Without automation, you’re left with manual processes that stretch over weeks, leave room for human error, and create unnecessary business risk.

The need for automation

In a typical manual scenario, creating a new database triggers a chain of time-consuming events. The governance team reviews the new data source, documents its contents, and scans for sensitive data. The security team assesses its configuration and access controls. Days or weeks pass before you fully understand this new asset’s sensitivity.

With automation, creating a new database triggers immediate action. The system detects the new source, catalogs its structure, identifies sensitive data, and updates a central inventory within minutes, supporting proper governance from the moment you create it. Here’s how it works on AWS: When you create an Amazon Simple Storage Service (Amazon S3) bucket for customer orders, you add tags such as Business Function, Data Owner, and Purpose. After the bucket is in use, the system detects it, creates catalog entries, analyzes data patterns, identifies sensitive information, and updates governance records without additional input from you. This gives your organization real-time visibility. Security teams instantly see which repositories contain sensitive information. Governance teams generate up-to-date inventory reports on demand, and data teams immediately understand sensitivity levels, helping them use data responsibly.

Solution overview

The solution uses key AWS services across three layers that work together for comprehensive data visibility and categorization.

Detection Layer: Continuously monitors your AWS environment for new resource creation. When you provision an Amazon S3 bucket, Amazon Relational Database Service (Amazon RDS) database, or Amazon DynamoDB table, Amazon EventBridge rules capture this activity and initiates the governance workflow, so no data source goes unnoticed.

Architecture: EventBridge triggers Lambda and SQS to create Glue crawlers and ETL jobs for new S3 data sources
Figure 1 Automated data source discovery (S3 example) workflow using EventBridge Rules and Lambda functions

Processing Layer: After a new source is detected, AWS Glue crawlers analyze its schema while specialized jobs scan for sensitive data patterns. The system also extracts metadata from resource tags, enriching your understanding of each repository’s purpose and ownership.

Architecture: Glue PII Detection jobs scan S3, DynamoDB, and Aurora; Lambda updates Glue Data Catalog
Figure 2 PII detection and processing workflow using AWS Glue jobs and DynamoDB staging

Management Layer: Maintains a central source of truth about your data assets. AWS Glue Data Catalog provides a unified view across your organization, tracking schema changes and sensitivity levels. This layer also manages the processing workflow state and generates insights for stakeholders.

Architecture: Lambda extracts S3 bucket metadata via EventBridge, stores in DynamoDB, updates Glue Data Catalog
Figure 3 Tag-based metadata capture and Data Catalog update workflow

Setting up the solution

This solution uses AWS Cloud Development Kit (AWS CDK) for deployment, organized into four stacks that build upon each other.PrerequisitesBefore deployment, verify that you have:

  • Access to an AWS account with permissions to create resources in Amazon S3, AWS Lambda, Amazon DynamoDB, AWS Glue, and Amazon EventBridge
  • Node.js (version 18 or later) and npm installed
  • Access to a terminal to run AWS CDK CLI commands
  • Basic familiarity with AWS Console navigation

Step 1: Infrastructure deployment

Deploy four stacks using AWS CDK. Each establishes components for data discovery, cataloging, and PII detection.

  1. BaseInfraStack: Deploys core infrastructure—Amazon Virtual Private Cloud (Amazon VPC), DynamoDB tables for state management, EventBridge rules for monitoring, and Lambda functions for orchestration.
  2. GlueAssetsStack: Sets up S3 buckets for AWS Glue ETL scripts and deploys PySpark code for PII detection.
  3. GlueJobCreationStack: Creates Data Catalog databases and deploys Lambda functions that automate the creation of AWS Glue crawlers and PII detection jobs for newly discovered data sources.
  4. ReportingStack: Deploys Lambda functions that process PII detection results and tag metadata, updating the Data Catalog accordingly.

To deploy these stacks, you will use the AWS CDK CLI, running the following commands:

# Clone and prepare repository
git clone https://github.com/aws-samples/automated-datastore-discovery-with-aws-glue.git
cd automated-datastore-discovery-with-aws-glue
npm install
npx cdk bootstrap

# Deploy infrastructure stacks sequentially
npx cdk deploy BaseInfraStack
npx cdk deploy GlueAssetsStack
npx cdk deploy GlueJobCreationStack
npx cdk deploy ReportingStack
CloudFormation Stacks console showing four stacks with CREATE_COMPLETE status including BaseInfraStack

Figure 4 CloudFormation console showing successful stack deployment

Step 2: Verify initial setup

In the AWS Management Console, open DynamoDB and find the glueJobTracker table. This table is a critical component of the framework:

  • Purpose: Central state management – tracks processing states and configurations for discovered data sources.
  • Current state: The table should be empty because no discovery processes have been triggered yet.
  • Structure: Tracks states such as Data Catalog entry creation and PII detection job setup for each data source.

By verifying this table, you confirm that the infrastructure is ready to begin tracking new data sources.

DynamoDB glueJobTracker table scan returning zero items, showing empty table before pipeline execution

Figure 5 Empty DynamoDB glueJobTracker table before execution

Solution in action

This solution runs automatically in production through EventBridge triggers and scheduled AWS Glue crawlers. The following walkthrough executes each step manually so you can observe the workflow.You follow the journey of a newly created S3 bucket containing sensitive data, seeing how the solution discovers, catalog, and processes it through each stage.

Step 3: Create a new S3 bucket

  1. Open the Amazon S3 console.
  2. Choose Create bucket.
  3. Enter a unique name for your bucket (for example, demo-customer-data-20250819).
  4. In the Tags section, add the following tags:
    1. Key: gdpr-scan, Value: true
    2. Key: Business Function, Value: Sales – US
    3. Key: Data Classification, Value: Confidential
  5. Keep other settings as default and choose Create bucket.
S3 bucket properties with versioning disabled and data classification tags

Figure 6 S3 console showing new bucket creation with tags

Step 4: Upload sample data

  1. In the S3 console, open your newly created bucket.
  2. Choose Upload.
  3. Create a new file named customer_orders.csv with the below content.
  4. Upload this file to a folder named orders/ in your bucket.
order_id,customer_id,email,ssn
ORD001,CUST1001,[email protected],***-**-****
ORD002,CUST1002,[email protected],***-**-****

S3 upload succeeded showing customer_orders.csv uploaded

Figure 7: S3 console showing uploaded CSV file in the orders folder

Step 5: Verify automated detection

  1. Open the DynamoDB console.
  2. Navigate to the glueJobTracker table.
  3. Choose the Items tab.
  4. You should see a new item with an s3_location matching your bucket name.
40918.pngDynamoDB glueJobTracker table scan returning 1 item with S3 data source entry

Figure 8 DynamoDB console showing detected bucket entry in glueJobTracker table

Step 6: Initiate catalog creation

  1. Open the AWS Lambda console.
  2. Find the function with a name containing s3GlueCatalogCreator.
  3. Choose the function name to open its details.
  4. Choose the Test tab.
  5. Create a new test event with an empty JSON object {}.
  6. Choose Test to invoke the function.
  7. Check the execution result for a successful response.
Lambda function execution succeeded with logs showing Glue table, crawler, and DynamoDB item creation

Figure 9 Lambda console showing successful function execution

Step 7: Run the AWS Glue crawler

  1. Navigate to the AWS Glue console.
  2. In the left sidebar, choose Crawlers.
  3. Find the crawler with a name related to your S3 bucket.
  4. Select the crawler and choose Run crawler.
  5. Wait for the crawler to complete (typically 3–5 minutes).
AWS Glue crawler running

Figure 10 Glue console showing crawler in “Running” state

Step 8: Verify schema discovery

  1. In the AWS Glue console, go to Databases in the left sidebar.
  2. Choose the s3_source_db database.
  3. You should see a new table corresponding to your uploaded data.
  4. Choose the table name to view its schema.
Glue Data Catalog table Version 3 showing 4-column CSV schema with no sensitive data annotations yet

Figure 11 Glue console showing detected table schema

Step 9: Execute PII detection

  1. Return to the Lambda console.
  2. Find and open the function with a name containing s3GlueCreator.
  3. Use the Test tab to invoke this function with an empty JSON object {}.
  4. After successful execution, go to the AWS Glue console.
  5. Navigate to Jobs in the left sidebar.
  6. Find the newly created PII detection job (it should contain your bucket name).
  7. Select the job and choose Run job.
  8. Monitor the job execution in the Glue console.
AWS Glue PII detection job running with 10 DPUs, G.1X worker type, Glue version 4.0, showing run details

Figure 12 Glue console showing PII detection job in “Running” state

Step 10: Review PII detection results

  1. Open the DynamoDB console.
  2. Navigate to the piiDetectionOutputTable.
  3. In the Items tab, you should see new entries related to your data.
  4. These entries will show detected PII types and confidence scores.
DynamoDB piiDetectionOutputTable scan showing 2 PII detection results identifying USA_SSN and EMAIL types

Figure 13 DynamoDB console showing PII detection results in piiDetectionOutputTable

Step 11: Verify Data Catalog updates

  1. Open the AWS Lambda console.
  2. Find the function with a name containing ReportingStack-PIIReportS3.
  3. Choose the function name to open its details.
  4. Choose the Test tab.
  5. Create a new test event with an empty JSON object {}.
  6. Choose Test to invoke the function.
  7. Check the execution result for a successful response.
  8. Return to the AWS Glue console.
  9. Go to Databases > s3_source_db > Your table.
  10. Review the schema. PII columns should now have comments indicating their classification.
Glue Data Catalog table Version 7 with schema showing sensitive data element comments for EMAIL and USA_SSN

Figure 14 Glue console showing updated table schema with PII classifications

Note: While we focus on S3 data sources in this walkthrough, the framework extends to other data stores, offering a unified approach for PII detection and compliance management, so organizations can automatically discover, catalog, and monitor sensitive data elements across your entire data ecosystem. For more information, see aws-samples/automated-datastore-discovery-with-aws-glue.

Best practices and operational excellence

As you implement this solution, consider these key practices for effective results:

  • Design your tagging strategy to capture essential business context about each data source. Implement automated tag enforcement through AWS Organizations for consistency across teams.
  • Monitor automated workflows regularly and configure retention policies for processed data to manage costs.
  • For enhanced security, configure VPC endpoints for services such as Amazon S3, DynamoDB, and other data sources. This keeps traffic within the AWS network, which is especially important when processing sensitive data. Verify that server-side encryption (SSE) is enabled on your data stores. This solution uses AWS Key Management Service (AWS KMS) keys for DynamoDB tables and SSE-S3 for S3 buckets by default, aligning with data-at-rest encryption best practices.
  • For teams with multiple AWS accounts, implement cross-account discovery and cataloging to maintain a comprehensive view of your data landscape.
Multi-account EventBridge buses aggregate events to central account with Lambda and Glue Data Catalog

Figure 15 Centralized Storage of Glue PII Detection Results in AWS Data Catalog

Clean up

To avoid ongoing charges and remove the resources created by this solution, follow these steps:

  1. Empty and delete the S3 buckets created for sample data and AWS Glue assets.
  2. Delete the AWS CloudFormation stacks in reverse order of creation:
    1. ReportingStack
    2. GlueJobCreationStack
    3. GlueAssetsStack
    4. BaseInfraStack
  3. Manually delete any remaining resources:
    1. DynamoDB tables (glueJobTracker, piiDetectionOutput, tagCaptureTable)
    2. AWS Glue databases and crawlers
    3. Lambda functions
    4. EventBridge rules
  4. Review your AWS account to ensure that all related resources have been removed.

Remember, deleting these resources will remove all data and configurations associated with this solution. Make sure that you have saved any important information before proceeding with the clean-up.

Conclusion

In this post, you learned how to build an automated data governance framework using AWS Glue Data Catalog. You set up detection, processing, and management layers that automatically discover, catalog, and classify your data sources.This approach improves how you manage sensitive data assets. Teams spend less time on manual discovery and categorization, freeing them to derive value from data. The system gives you current insights into your data landscape and automatically identifies sensitive data, creating a trusted source of truth that helps teams work efficiently while maintaining controls.You can extend this framework with custom sensitivity patterns for your industry. Its modular design supports continuous improvement and integrates with existing workflows. This turns data governance from a manual burden into an efficient process that scales with your organization.

About the authors

Ramakrishna Natarajan

Ramakrishna Natarajan

Ramakrishna is a Senior Partner Solutions Architect at Amazon Web Services. He is based out of London and helps AWS Partners find optimal solutions on AWS for their customers. He specialises in Telecommunications OSS/BSS and has a keen interest in evolving domains such as Data Analytics, AI/ML, Security and Modernisation. He enjoys playing squash, going on long hikes and learning new languages.

Capture data lineage of Amazon EMR spark jobs into Amazon SageMaker Unified Studio

Post Syndicated from Jose Romero original https://aws.amazon.com/blogs/big-data/capture-data-lineage-of-amazon-emr-spark-jobs-into-amazon-sagemaker-unified-studio/

Data engineers running Apache Spark jobs on Amazon EMR face a persistent challenge: understanding how data moves through Spark pipelines as it’s transformed, joined, and written to downstream tables . Tracking these transformations manually requires examining job logs, reviewing code, and piecing together transformation logic across multiple sources. As pipelines scale, this process becomes complex. The visibility gap affects key business activities: troubleshooting data quality issues takes longer – impact analysis for schema changes requires more effort – and compliance audits need extensive documentation of data provenance.

Amazon SageMaker is the center for all your data and analytics where you can find and access all the data in your organization and act on it using tools across various use case. This unified platform addresses the data visibility challenge by bringing together data governance, collaboration, and discovery into a single interface. At the heart of this platform is Amazon SageMaker Catalog, a centralized hub that enables organizations to catalog, govern, and discover all their data assets with complete visibility into lineage. By capturing data lineage across your entire data ecosystem from raw sources through transformations to final outputs, SageMaker Catalog enables you to track data provenance across your entire platform, enable collaboration with clear visibility into data ownership and quality metrics, build trust through comprehensive data lineage that supports compliance and confident decision-making, and accelerate discovery of trustworthy, governance-ready data assets. You can access and visualize this lineage directly in Amazon SageMaker Unified Studio, which serves as the unified interface to explore data relationships and collaborate across your analytics workflows.

Amazon EMR, starting from version 7.11, now includes native OpenLineage support that automates lineage capture. OpenLineage is an open-source framework for data lineage that automatically emits lineage metadata from your data transformation jobs directly into Amazon SageMaker Catalog, or other data governance solutions, without requiring customizations.

This EMR native support of OpenLineage is part of a growing set of integrations across AWS analytics services including AWS Glue, Amazon EMR Serverless, and Amazon Redshift. The complete list of services with native OpenLineage integration can be found in the data lineage support matrix.

In this post, you’ll walk through a practical, step-by-step example that shows how to capture and track data lineage from Spark jobs running on Amazon EMR directly into Amazon SageMaker Catalog using OpenLineage. You’ll see how lineage metadata flows automatically and explore data relationships and dependencies across your workflows in Amazon SageMaker Unified Studio.

Solution overview

Imagine you’re part of a large enterprise that relies on HR analytics to optimize workforce planning, compensation strategies, and talent retention practices. Your data engineering team owns the delivery of these analytical products by processing raw HR datasets (including employee records, attendance logs, and compensation details), with Spark jobs running on your Amazon EMR infrastructure.

With time, Spark jobs have grown in complexity. Your team now struggles to maintain visibility into how data moves through pipelines, who modified it, and how to map dependencies between datasets and final analytical products.

The following solution demonstrates how you can address these challenges by automatically capturing data lineage end-to-end from Spark jobs running on your EMR infrastructure and visualizing it in Amazon SageMaker Unified Studio so that you and the business understand data provenance of the final analytical products.

AWS cloud data pipeline architecture diagram showing data flowing from Amazon S3 CSV files (employees.csv, attendance.csv) through Amazon EMR with Apache Spark processing, AWS Glue Data Catalog metadata management, and Amazon SageMaker Catalog integration, producing salary_adjustments.csv and bonus_payments.csv output files stored in Amazon S3.

The architecture includes a Data Layer with CSV files containing employee, attendance, salary, and bonus data stored in Amazon S3 (Simple Storage Service), representing typical HR and payroll source systems.

The Processing Layer uses Amazon EMR cluster running Apache Spark jobs that transform raw data into analytical tables. The first Spark job joins employee and attendance data while the second Spark job combines attendance with compensation data. Both jobs use Apache Iceberg table format to provide ACID (Atomic, Consistent, Isolated, and Durable) transactions and time travel capabilities.

The Metadata Layer uses AWS Glue Data Catalog to store Iceberg table metadata, making tables discoverable and accessible across AWS analytics services. A Lineage Layer uses the OpenLineage integration in EMR to automatically track input/output datasets (CSV files and Iceberg tables), transformation logic at column level (joins, filters, aggregations), and job execution metadata.

Finally, the Data Governance Layer uses Amazon SageMaker Catalog to capture and process OpenLineage events posted by the EMR Spark jobs and automatically build a comprehensive lineage graph that shows complete data provenance from CSV source files through Spark transformations to Iceberg analytical tables.

Before you deploy this solution, make sure you have the following resources in place.

Prerequisites

For this walkthrough, you should have the following prerequisites:

  • An AWS account.
  • Your assumed role should have full access to Amazon EMR serverless, Amazon S3, Amazon Identity and Access Management (IAM) and AWS Lambda. Note that for production workloads, minimum permissions are recommended.
  • A Amazon VPC (Virtual Private Cloud) with at least one subnet with internet access. You can provision this VPC as you create the Amazon SageMaker domain next.
  • An existing Amazon SageMaker Unified Studio domain and project. To get started, use the quick setup option as explained here. To create a project, follow the instructions here.
  • An S3 bucket with the sample data files and Spark scripts uploaded (see Prepare Your Source Data below)
  • Default EMR service roles — if this is your first time using EMR in this account, run `aws emr create-default-roles` from the AWS CLI or CloudShell to create them.

With these prerequisites in place, let’s examine what the AWS CloudFormation template will deploy to your AWS environment.

Architecture components

The deployment creates several interconnected components that work together to capture and visualize lineage:

  • An S3 bucket to store all data and artifacts for the solution.
  • An EMR cluster (v 7.12.0) with Apache Iceberg support enabled and OpenLineage integration pre-installed, ready to run Spark jobs with lineage tracking.
  • A set of IAM policies that grant the necessary permissions to the EMR cluster to post lineage events to your SageMaker Unified Studio domain.
  • A set of AWS Lake Formation permissions that grant the EMR cluster to create, alter, and drop Iceberg tables in your specified Glue database.

With an understanding of what will be deployed, you’re ready to launch the CloudFormation stack.

Deploy the solution

Note: While this walkthrough uses the AWS EMR console and AWS CLI to verify the cluster and run Spark jobs, you can also perform these steps directly from Amazon SageMaker Unified Studio. SMUS provides a unified interface to create and manage EMR clusters, submit Spark jobs, and monitor execution — all within the same environment where you’ll later explore the lineage captured in Amazon SageMaker Catalog.

Prepare your source data

Before deploying the CloudFormation stack, clone or download the following git repository.PutHereGitRepo

Upload the CSV files downloaded from git to the input/ prefix and the spark scripts in scripts/ prefix. You can run the following command to upload the files:

aws s3 cp employees.csv s3://YOUR-BUCKET/input/
aws s3 cp attendance.csv s3://YOUR-BUCKET/input/
aws s3 cp salary_adjustments.csv s3://YOUR-BUCKET/input/
aws s3 cp bonus_payments.csv s3://YOUR-BUCKET/input/
aws s3 cp emr-lineage-spark-job.py s3://YOUR-BUCKET/scripts/
aws s3 cp emr-lineage-compensation-job.py s3://YOUR-BUCKET/scripts/

To deploy the solution, complete the following steps in CloudFormation console:

  1. Create new stack by specifying the CloudFormation yaml file previously download from git repository PutHereThe YMLFileName
  2. Enter a stack name (e.g., emr-lineage-demo) and provide the following parameters:
    • SourceS3BucketName: S3 bucket containing your CSV files and Spark scripts
    • SourceCSVPrefix: S3 prefix where CSV files are located
    • SourceScriptsPrefix: S3 prefix where Spark scripts are located
    • GlueDatabaseName: The name of the Glue database associated to your Amazon SageMaker Unified Studio project.
    • DataZoneDomainId: Your SageMaker Unified Studio domain ID.
    • VpcId: The id of the VPC that was deployed as part of the prerequisites.
    • For EMRReleaseLabel, MasterInstanceType, CoreInstanceType and CoreInstanceCount, keep the default values.
  3. Acknowledge IAM resource creation, choose Next and then Submit. The CloudFormation stack takes approximately 10 to 15 minutes to complete.
  4. In the EMR console, wait for the cluster status to show as WAITING before moving to the next step.

Screenshot of the Amazon EMR on EC2 Clusters management console showing a list of 14 clusters, with the cluster "EMR-Lineage-Demo-emr-ec2-lineage-demo-stack" (ID: j-3APWOTUDNYO2T) highlighted in a "Waiting – Ready to run steps" status with a green badge.

Now that the EMR cluster is running with OpenLineage enabled, let’s examine how the Spark jobs are configured to capture lineage metadata.

Explore data lineage configuration in EMR

When submitting Spark jobs to EMR, specific configurations enable OpenLineage to create and post lineage events to SageMaker Unified Studio as the job runs:

  • spark.hadoop.hive.metastore.client.factory.class – Configures Spark to use AWS Glue as the Hive metastore.
  • spark.jars – Path to the pre-installed OpenLineage library (available on EMR 7.11+).
  • spark.extraListeners – Registers an OpenLineage listener to capture metadata of input / output datasets and transformations.
  • spark.openlineage.transport.type – Uses the OpenLineage DataZone transport option to send lineage events directly into SageMaker Catalog.
  • spark.openlineage.transport.domainId – The ID of your SageMaker Unified Studio domain, that serves as the target for lineage events.
  • spark.glue.accountId – Your AWS account ID for Glue data catalog operations.

Now that you understand the configuration that enables automatic lineage capture, you’re ready to run the data pipeline.

When running this two-step pipeline, you will calculate the total employee compensation by combining salary adjustments, bonuses, and attendance data. The final analytical asset will serve payroll processing and budgeting.

Run employee attendance analysis job

The first job reads employee details (in employees.csv dataset) and attendance records (in attendance.csv dataset), joins the datasets on EmployeeID and creates a unified dataset (employee_attendance Iceberg table) in your Glue database.

Follow the steps below to run this first job:

  1. In the CloudFormation console, navigate to the stack’s Outputs tab
  2. Copy the value of the Job1SubmitCommand output key. Note that this is the command you’ll use to submit the first job in EMR with the right configuration.

AWS CloudFormation console screenshot showing the Outputs tab for the "emr-ec2-lineage-demo-stack" stack, displaying 9 outputs including the Job1SubmitCommand — an AWS EMR add-steps command with Apache Spark configuration for the EMR Lineage Demo Job targeting cluster j-3APWOTUDNYO2T.

  1. Run the command in your terminal or AWS CloudShell.
  2. Monitor the job in the Amazon EMR console under Steps.

Screenshot of the Amazon EMR console Steps tab for the cluster "EMR-Lineage-Demo-emr-ec2-lineage-demo-stack," showing one completed step named "EMR-Lineage-Demo-Job" with Step ID s-0270631D8DHBCJZKBAZ and a green "Completed" status checkmark.

Run employee compensation analysis job

Now, you will calculate the total employee compensation (Iceberg table) by combining salary adjustments (salary_adjustments.csv dataset), bonuses (bonus_payments.csv dataset), and attendance (calculated in the last step):

  1. Repeat the steps 1 to 4 to run Job 2.
  2. After completion, open the AWS Glue console.
  3. Navigate to Data Catalog, then Tables and select your SageMaker project’s database.
  4. Confirm that employee_attendance and employee_compensation tables are listed.

With both Spark jobs complete, you can now visualize the complete data lineage graph in Amazon SageMaker Unified Studio.

Visualizing lineage in SageMaker Unified Studio

SageMaker Unified Studio provides a graph-based data lineage visualization that helps data engineers, analysts, and data scientists clearly understand which source datasets (files or tables) feed into each dataset, what transformations and logic are applied at every step, which downstream analytics assets consume the data, and how changes to upstream data or transformations may impact the rest of the data pipeline.

Now that the data pipeline run successfully, let’s review the captured lineage for the HR data in SageMaker Unified Studio:

  1. Navigate to the SageMaker Unified Studio console, sign in to your domain.
  2. Open your project and go to Data Sources
  3. Find your AWS Glue Data Catalog source

Screenshot of the Amazon SageMaker project catalog Data Sources page listing three configured data sources: a Redshift Serverless source, an AWS Glue Lakehouse source named "AwsDataCatalog-emr_ec2_lineage_blogpost_glue_db-default-datasource" (highlighted), and a Tooling SageMaker model package group source — all scheduled MTWTFSS and in Ready or Running status.

  1. Click RUN. Two new assets will be created.

Screenshot of the AWS Glue Data Catalog interface showing run activities for the data source "AwsDataCatalog-emr_ec2_lineage_blogpost_glue_db-default-datasource," with two completed on-demand runs and a highlighted asset table showing employee_attendance and employee_compensation successfully created in the emr_ec2_lineage_blogpost_glue_db database.

  1. Navigate to Assets and Click on employee_compensation. Under the LINEAGE tab you’ll find the lineage graph view that SageMaker builds based on the OpenLineage metadata captured from the EMR Spark jobs as they run.

AWS Glue data lineage visualization showing the flow of the employee_compensation dataset from an Apache Spark job (default.emr_lineage_compensa, COMPLETE, Dec 22 2025 11:42:47 AM) through an AWS Glue Iceberg table (20 columns) to an AWS Glue Inventory destination table, with a right sidebar displaying lineage metadata including the dataset ARN, OpenLineage producer URL, Iceberg snapshot ID, and projected field names EmployeeID, Name, and Department.

    • You’ll first see three lineage nodes from left to right: one representing the EMR Spark job that created the final Iceberg table, a second one representing the actual Iceberg table in the Glue catalog, and a third one representing the data asset in the SageMaker Catalog inventory that maps to the Glue table.
    • Click on any lineage node to view its underlying metadata in the details pane, including dataset names, S3 locations, schema, data types, job execution details and more.
  1. Expand the lineage to the left by clicking on the double arrow next to the first lineage node. Keep expanding until you hit the originating datasets.

Data pipeline lineage diagram showing the complete ETL flow from Amazon S3 source files (input/attendance.csv with 6 columns, input/employees.csv with 5 columns) through two Apache Spark jobs to intermediate tables (input/salary_adjustments.csv, iceberg/employee.csv, AWS Glue employee_attendance with 14 columns) and final destination tables (AWS Glue iceberg/employee_compensation with 29 columns, AWS Glue Inventory employee_compensation_hive with 30 columns), all timestamped Dec 22, 2025.

    • Expanding the graph to the left reveals the complete data pipeline back to original CSV source files. You can see how compensation data depends on upstream attendance analytics.
    • Note how each lineage node represents an element in the data pipeline you run, including both Spark jobs and even the intermediate employee_attendance Iceberg table that connects them.
  1. You can expand column-level lineage by clicking on the column section of a lineage node of a dataset or data asset. This allows you to understand how data changes at a column level as it goes downstream your data pipeline.

Data lineage diagram showing the employee compensation ETL pipeline with four Amazon S3 source tables (employee.csv with 5 columns, input/attendance.csv with 6 columns, input/salary_adjustments.csv with 4 columns, output/employee_attendance.csv with 14 columns) processed by two Apache Spark jobs to produce a final s3://employee_compensation table with 20 columns, all dated Dec 22, 2025.

Cleanup

To avoid ongoing charges, clean up the resources:

  1. First, empty the destination bucket by running the following command in your terminal or with AWS CloudShell.

aws s3 rm s3://${DEST_BUCKET}/ --recursive

  1. Delete the CloudFormation stack.
    • On the AWS CloudFormation console, choose Stacks in the navigation pane.
    • Choose the stack you created, then choose Delete and then Delete stack when prompted.

Conclusion

In this post, you explore how to capture data lineage from Spark jobs in Amazon EMR (v7.11+) directly into Amazon SageMaker Unified Studio. You learned how to set up an Amazon EMR cluster with native OpenLineage support to automatically track lineage metadata from Spark jobs processing your data. You also configured the integration between EMR and Amazon SageMaker Catalog to ensure lineage information flows seamlessly into your governance platform. Finally, you explored the resulting lineage graph in SageMaker Unified Studio and saw how it provides comprehensive visibility into data transformations, from source CSV files through Spark processing jobs to final analytical tables using Apache Iceberg format.

We encourage you to now test these capabilities with your own data pipelines running on EMR. By implementing automated lineage tracking, many customers have strengthened their governance frameworks while gaining valuable insights into data dependencies, impact analysis, and compliance requirements. This approach enables data teams to build trust in their analytics outputs while maintaining the agility needed to derive business value from their data assets.

About the authors

Yanick Houngbedji is a Solutions Architect for Independent Software Vendors (ISV) at Amazon Web Services (AWS), based in Montréal, Canada. He specializes in helping customers architect and implement highly scalable, performant, and secure cloud solutions on AWS. Before joining AWS, he spent over 8 years providing technical leadership in data engineering, big data analytics, business intelligence, and data science solutions.

Jose Romero is a Senior Solutions Architect for Startups at Amazon Web Services (AWS) based in Austin, TX, US. He is passionate about helping customers architect modern platforms at scale for data, AI, and ML. As a former senior architect in AWS Professional Services, he enjoys building and sharing solutions for common complex problems so that customers can accelerate their cloud journey and adopt best practices. Connect with him on LinkedIn.

Automating identity lifecycle and security with AWS Directory Service APIs

Post Syndicated from Ali Alzand original https://aws.amazon.com/blogs/security/automating-identity-lifecycle-and-security-with-aws-directory-service-apis/

Managing identities and access across complex environments has become more critical than ever. AWS Directory Service for Managed Microsoft Active Directory, also known as AWS Managed Microsoft AD, has added new capabilities to manage users and groups. Now, you can perform create, read, update, and delete (CRUD) operations on users and groups directly through AWS Command Line Interface (AWS CLI), APIs, and the AWS Management Console. You can use this powerful capability to automate identity lifecycle management and enhance security in your AWS environment. By using these APIs, collectively known as the Directory Service Data APIs, you can perform operations such as:

  • Listing users and groups
  • Retrieving user and group details
  • Disabling and enabling user accounts
  • Resetting user passwords
  • Managing group memberships

These APIs provide new possibilities for automating identity management tasks and integrating Active Directory management into your existing workflows and applications.

The introduction of these APIs brings several key benefits:

  • Automation of the identity lifecycle: You can now programmatically manage user accounts throughout their lifecycle—from creation to deletion—enabling streamlined onboarding and offboarding processes.
  • Enhanced security: By integrating these APIs with security services like Amazon GuardDuty, you can create automated responses to potential security threats, such as disabling accounts with inappropriate access.
  • Improved compliance: You can use automated user management to help enforce consistent policies and help maintain compliance with various regulatory requirements.
  • Operational efficiency: You can automate routine tasks such as user provisioning, deprovisioning, and group management, reducing manual effort and the potential for human error.
  • Integration capabilities: By using these APIs, you can seamlessly integrate with existing identity management systems, custom applications, and third-party tools.
  • Cost optimization: By automating processes and reducing manual intervention, you can potentially help your organization optimize operational costs associated with identity management.

In this post, we explore these new APIs and demonstrate how you can use them to create an automated solution for detecting and responding to unexpected behavior by Active Directory users. We walk through a practical example that combines GuardDuty, AWS Step Functions, Amazon EventBridge, and the new AWS Directory Service APIs to create a robust security automation workflow.

Solution overview

To demonstrate the power of these new APIs, let’s explore a practical solution that automates the detection and response to unexpected behavior by Active Directory users. This solution combines several AWS services to create a robust security automation workflow:

    1. GuardDuty continuously monitors for unexplained behavior of Active Directory users from AWS Managed Microsoft AD. For the example in this post, we’re using Backdoor:Runtime/C&CActivity.B!DNS
    2. An EventBridge rule detects GuardDuty findings related to these users and triggers a Step Functions workflow. 
      {
  "detail-type": ["GuardDuty Finding"],
  "source": ["aws.guardduty"],
  "detail": {
    "type": ["Backdoor:Runtime/C&CActivity.B!DNS"]
  }
}

    3. The Step Functions workflow will:
      1. Extract the Active Directory username from the instance using a run command.
      2. Start an automation that will disable the account using the DisableUser API.
Figure 1: Diagram of the Step Functions workflow showing the process of Systems Manager finding the username and starting the automation to disable the account

Figure 1: Diagram of the Step Functions workflow showing the process of Systems Manager finding the username and starting the automation to disable the account

  1. Finally, another EventBridge rule will monitor the DisableUser API call. It will send an email to the user using Amazon Simple Notification Service (Amazon SNS) notifications. 
    {
  "detail-type": ["AWS API Call via CloudTrail"],
  "source": ["aws.ds"],
  "detail": {
    "eventSource": ["ds.amazonaws.com"],
    "eventName": ["DisableUser"]
  }
}

This solution delivers automated, near real-time remediation of potential security threats — significantly reducing exposure windows and containing the impact of unauthorized account access.

The following figure shows a high-level architecture diagram of the solution.

Figure 2: Diagram showing the workflow of what happens when potentially damaging activity is detected

Figure 2: Diagram showing the workflow of what happens when potentially damaging activity is detected

Note: The solution must be deployed in the primary AWS Region of your directory.

Prerequisites

To complete the walkthrough in this post, you must have the following prerequisites in place.

GuardDuty

GuardDuty is an automated threat detection service that continuously monitors for unexpected activity and unauthorized behavior to protect your AWS accounts, workloads, and data stored in Amazon Simple Storage Service (Amazon S3).

To activate GuardDuty:

  1. Go to the GuardDuty console.
    1. If you’re activating GuardDuty for the first time, under Try threat detection with GuardDuty, select All Features and then choose Get Started.
    2. If you’ve used GuardDuty before, select Runtime Monitoring and then choose Enable under Runtime Monitoring.
Figure 3: Runtime Monitoring enabled

Figure 3: Runtime Monitoring enabled

AWS Managed Microsoft AD

AWS Managed Microsoft AD provides a fully managed service for Microsoft Active Directory (AD) in the AWS Cloud. When you create your directory, AWS deploys two domain controllers that are exclusively yours in separate Availability Zones for high availability. For use cases that require even higher resilience and performance in a specific AWS Region or during specific hours, you can scale AWS Managed Microsoft AD by deploying additional domain controllers to meet your needs. These domain controllers can help load balance, increase overall performance, or provide additional nodes to protect against temporary availability issues. Using AWS Managed Microsoft AD, you can define the correct number of domain controllers for your directory based on your use case.

To deploy a new AWS Managed Microsoft AD:

  1. Go to the Directory Service console.
  2. Choose Set up directory and select AWS Managed Microsoft AD.
  3. Select Standard Edition and enter a directory DNS name and password.
  4. Select a virtual private cloud (VPC). For this example, use the Default VPC.
  5. Choose Create directory.

Create a test Active Directory user

You will use this test user account to sign in to an EC2 instance and initiate a command that simulates unexplained activity that results in this account being disabled.

To create the test user, you can use AWS CloudShell or the AWS CLI from your local machine. Run the following commands, replacing the --directory-id value with your own:

# Create the test user
aws ds-data create-user \
 --directory-id "your-directory-id" \
 --sam-account-name "TestUser" \
 --given-name "Test" \
 --surname "User"

Then

# Set a password for the test user 
aws ds reset-user-password \
 --directory-id "your-directory-id" \
 --user-name "TestUser" \
 --new-password "YourSecurePassword123!"

In this example, the password is set to YourSecurePassword123!. If you need to replace it with a password that meets your organization’s requirements, see Resetting and enabling an AWS Managed Microsoft AD user’s password. For more information on creating users, see Creating an AWS Managed Microsoft AD user in the AWS Directory Service documentation.

Test EC2 instance

To generate alerts on GuardDuty, you need a domain joined Linux EC2 instance. If you don’t have a domain joined EC2 Linux instance, follow these instructions for joining a Linux instance to an Active Directory domain. This instance will be used to simulate suspicious activity that triggers a GuardDuty finding and initiates the automated remediation workflow.

Implement the solution

Let’s walk through the steps to implement this solution in your AWS environment.

Deploy the solution

  1. Download the CloudFormation template
  2. Navigate to the CloudFormation console in the AWS account.
  3. For Create Stack, choose with new resources (standard).
  4. For Template source, choose Upload a template file. Choose Choose file and select the template you downloaded in step 1.
  5. Choose Next.
  6. For Stack name, enter a stack name (such as CRUD-API-MAD).
  7. In the Parameters area, do the following:
    1. For DirectoryID, enter the AWS Active Directory ID.
    2. For NotificationEmail, enter the email address to send the notification to.
  8. On the Configure stack options page, choose Next.
  9. Select I acknowledge that AWS CloudFormation might create IAM resources with custom names, then choose Submit.

After the page is refreshed, the status of your stack should be CREATE_IN_PROGRESS. When the status changes to CREATE_COMPLETE, proceed to the next section.

Test

To simulate a threat, use a GuardDuty test domain that GuardDuty will recognize as a command and control server.

  1. Go to the Amazon EC2 console.
  2. Choose Instances from the navigation pane.
  3. Select the test EC2 instance that you created earlier.
  4. Choose Connect, select the Session Manager tab, and choose Connect.
  5. Authenticate with your test user by entering su followed by the test user with the domain name that you created earlier. For example su [email protected], then enter the password.
  6. Enter the command curl guarddutyc2activityb.com.
    You will receive an error because the page won’t resolve, but GuardDuty will have detected concerning events.
  7. Go to the GuardDuty console and select Findings from the navigation pane.
  8. Within 3–5 minutes, you should see a high severity finding for Backdoor:Runtime/C&CActivity.B!DNS.
  9. This will then trigger the automation to disable the account.
    Figure 4: Account successfully disabled

    Figure 4: Account successfully disabled

  10. After the account is disabled, an email notification will be sent notifying an administrator that the account was disabled (it might take up to 5 minutes to receive the notification).

    Figure 5: AWS notification message showing the username has been disabled

    Figure 5: AWS notification message showing the username has been disabled

Note: You must archive the GuardDuty finding before running this test again, because the EventBridge rule only runs once against a GuardDuty finding with the same details. To archive the finding, select the check box next to the Backdoor:Runtime/C&CActivity.B!DNS finding, choose Actions (top right), and select Archive.

Conclusion

The new AWS Directory Service APIs for AWS Managed Microsoft AD provide powerful capabilities for programmatically managing Active Directory users and groups. By using these APIs in conjunction with services such as Amazon GuardDuty and AWS Step Functions, you can create sophisticated automation workflows that enhance your security posture and streamline identity management processes.

The solution we’ve explored in this post demonstrates just one of many possible use cases for these new APIs. As you integrate these capabilities into your own environments, you will probably discover numerous opportunities to improve efficiency, security, and compliance in your identity management practices.

For a solution that uses PowerShell Active Directory cmdlets with AWS Systems Manager Run Command to disable users, see How to automatically disable users in AWS Managed Microsoft AD based on GuardDuty findings.

For more information about AWS Directory Service and its APIs, visit the AWS Directory Service documentation.

We’re excited to see how you’ll use these new APIs to innovate and improve your identity management workflows. If you have any questions or want to share your own use cases, leave a comment below or reach out to AWS Support.

Remember, the cloud journey is all about continuous improvement and innovation. Keep exploring, keep learning, and keep pushing the boundaries of what’s possible with AWS.

Ali Alzand

Ali Alzand

Ali is a Senior Infrastructure Migration & Modernization Specialist Solutions Architect at AWS who helps enterprise customers migrate, modernize, and operate their Microsoft workloads on AWS. He specializes in Infrastructure as Code, automating at scale with AWS Systems Manager, EC2 Image Builder, and CloudFormation. He also designs event-driven architectures building responsive, loosely coupled solutions with EventBridge and Lambda. Outside of work, Ali enjoys grilling with friends and discovering new cuisines around town.

Kevin Sookhan

Kevin Sookhan

Kevin is a Specialist Solutions Architect at Amazon Web Services with over 20 years of experience working with Microsoft technologies. He has expertise in running Microsoft workloads on AWS with specialization in helping customers with their migrations, cost optimization, and infrastructure architecture.

Automate root cause analysis across Datadog and Elasticsearch with AWS DevOps Agent

Post Syndicated from Bhuvan Jain original https://aws.amazon.com/blogs/devops/automate-root-cause-analysis-across-datadog-and-elasticsearch-with-aws-devops-agent/

Modern distributed systems route business transactions through dozens of microservices, message queues, and event streams. When a message fails to process or processing exceeds SLA thresholds, troubleshooting requires correlating logs from tools like Elasticsearch, metrics from Datadog, and infrastructure change events in AWS CloudTrail. Correlating these signals manually across heterogeneous backends, each with different query languages, schemas, and time granularities, can take hours per incident and demands deep institutional knowledge of the system topology.

This post shows how AWS DevOps Agent, combined with a custom Model Context Protocol (MCP) server for Elasticsearch and native Datadog integration, automates end-to-end root cause analysis. When a Datadog alert fires, AWS DevOps Agent automatically initiates an investigation, correlates signals across all observability backends, and delivers root cause findings in minutes, without manual intervention.

In this post, we walk through the architecture, configuration steps, and a real-world scenario demonstrating how AWS DevOps Agent dramatically reduces mean time to identify (MTTI) for distributed system failures. DevOps engineers, site reliability engineers (SREs), and operations leaders managing containerized workloads will learn how to implement alert-triggered automated investigations that eliminate manual correlation and accelerate root cause identification in their own environments.

Challenges in correlating telemetry signals at scale

At scale, correlating telemetry signals across distributed systems is a key challenge. A platform processing billions of communications for regulated industries must track every message through its full lifecycle — ingestion, transformation, policy evaluation, archival, and retrieval — across dozens of production clusters, thousands of worker nodes, and terabytes of daily telemetry spread across multiple observability backends. A single message ID can generate log entries across multiple indices, correlated metrics in monitoring systems, and change events in audit trails. When a message goes missing or processing stalls, the operations team must pinpoint which cluster processed it, which log store holds the evidence, whether a recent deployment preceded the failure, and whether the issue is isolated or systemic — all while context-switching across tools with different query languages and data schemas. Before AWS DevOps Agent, this process routinely took hours per incident, and longer for complex multi-service failures.

The core difficulty is not the volume of data. It is the correlation of signals across heterogeneous systems that use different identifiers, different time granularities, and different data schemas. A message ID in Elasticsearch logs must be correlated to:

  • A trace ID in application performance monitoring (APM) systems
  • A pod name and namespace in Kubernetes event logs
  • A container image tag in Amazon Elastic Container Registry (ECR) push events
  • Metric anomalies (error rate spikes, pod restarts, CPU/memory deviations) in Datadog
  • Deployment events captured in AWS CloudTrail logs

Manual correlation requires engineers to maintain mental models of these relationships while executing queries across multiple systems. It is error-prone, non-repeatable, and heavily dependent on institutional knowledge. When the engineer with the deepest system familiarity is unavailable, resolution times increases.

Prerequisites

Complete the following prerequisites before configuring the integrations:

  1. The AWS Command Line Interface (AWS CLI) version 2. For installation instructions, see installing or updating to the latest version of the AWS CLI.
  2. Helm – the Kubernetes package manager used to deploy the sample application.
  3. Kubectl – the Kubernetes command-line tool used to deploy Filebeat and manage cluster resources.
  4. An EKS cluster with Control plane logs enabled.
  5. AWS DevOps Agent Agentspace. For installation instructions, refer to Creating an Agent Space.
  6. Elasticsearch cluster deployed and accessible (EC2-hosted, Amazon OpenSearch Service, or self-managed). Filebeat configured as a DaemonSet to collect pod logs and forward to Elasticsearch.
  7. Datadog account with API key and application key. To create, see API & Applications key.

Solution Architecture

The solution presented in this post combines three integrated components to deliver automated end-to-end message ID traceability:

  • AWS DevOps Agent as the intelligent investigation orchestrator
  • A custom ELK MCP Server providing structured access to Elasticsearch log data
  • Native Datadog integration for metrics, events, and alert-triggered investigations

Together, these components form an autonomous investigation pipeline that activates when an alert fires, correlates signals across all observability sources, builds a topological understanding of the affected services, and delivers a structured root cause analysis, without manual intervention.

In our implementation, application pods are instrumented to emit custom metrics to Datadog, including per-message-ID processing status, trace ID labels, and endpoint-level error counters. This instrumentation provides AWS DevOps Agent with the ability to correlate a specific message ID from an alert payload to its corresponding trace ID in application performance data, a correlation that previously required manual cross-referencing.

Webhook-Based Alert Triggering

A critical aspect of the architecture is the automated triggering of investigations when alerts fire. Rather than requiring manual investigation initiation, the solution configures Datadog alerting webhooks to invoke AWS DevOps Agent directly.

When a Datadog monitor enters an alert state, it fires a webhook to the AWS DevOps Agent endpoint, including:

  • The message ID associated with the processing failure
  • The trace ID for APM correlation
  • The alert timestamp and alert condition
  • The triggering monitor name and severity

AWS DevOps Agent authenticates the webhook using a bearer token and immediately initiates an investigation in the configured Agent Space. The Datadog webhook payload serves as the investigation’s initial context, seeding the agent with the specific identifiers it needs to perform targeted queries rather than broad searches across the full data volume.

Architecture Diagram

Architecture diagram showing the automated root cause analysis pipeline. A Datadog alert fires a webhook to AWS DevOps Agent, which queries three sources: an ELK MCP Server connected to Elasticsearch for log data, native Datadog integration for metrics, and AWS CloudTrail for deployment events. Results flow back to the Agent Space for correlated root cause analysis.
Figure 1: Automated root cause analysis pipeline — from Datadog alert to AWS DevOps Agent investigation across Elasticsearch, Datadog, and AWS CloudTrail

Implementation Walkthrough

The following walkthrough describes the end-to-end setup required to replicate the message ID traceability solution in your own environment.

Step 1: Configure EKS Cluster Access for AWS DevOps Agent

AWS DevOps Agent requires an access entry in each EKS cluster it will investigate. This enables the agent to describe Kubernetes objects, retrieve pod logs, and access cluster events.

  1. In the AWS DevOps Agent console, navigate to your Agent Space and select the Capabilities tab.
  2. Under the Cloud section, select the primary source and choose Edit. Note the Role Name shown in the Role Name field, this is the IAM role that requires EKS access.
  3. In the Amazon EKS console, select each cluster and open the Access tab.
  4. Under IAM Access Entries, choose Create to add a new access entry.
  5. Set the IAM Principal ARN to the Agent Space role noted in step 2.
  6. Under Access Policies, select AmazonAIOpsAssistantPolicy with Cluster scope. Choose Add Policy, then Next.
  7. Review and create the access entry.

At scale: For environments with 50+ clusters, use the AWS CLI, Terraform, or a GitOps pipeline to automate access entry creation across all clusters. The following CLI command creates an access entry for a single cluster:

aws eks create-access-entry --cluster-name <CLUSTER_NAME> --principal-arn <AGENTSPACE_ROLE_ARN> --region <REGION>
aws eks associate-access-policy --cluster-name <CLUSTER_NAME> --principal-arn <AGENTSPACE_ROLE_ARN> --policy-arn arn:aws:eks::aws:cluster-access-policy/AmazonAIOpsAssistantPolicy --access-scope type=cluster --region <REGION>

Step 2: Configure Datadog Integration in AWS DevOps Agent

AWS DevOps Agent includes native Datadog integration. Configuration requires your Datadog API credentials and designates the integration as a data source in the Agent Space. In the AWS DevOps Agent console, navigate to Integrations and choose Add Integration and follow the steps.

After configuration, AWS DevOps Agent can query Datadog metrics, monitors, and events during investigations. Custom application metrics (message throughput, error rates, processing status per message ID) are automatically accessible once the integration is active.

Step 3: Deploy and Configure the Custom ELK MCP Server

The Elasticsearch MCP server bridges AWS DevOps Agent to your self-managed Elasticsearch deployment. The MCP server is deployed as a publicly accessible endpoint with TLS authentication, enabling AWS DevOps Agent to call Elasticsearch APIs securely without requiring direct network access to the actual Elasticsearch/Kibana instances.

Note: For basic Elasticsearch integration, the official Elasticsearch MCP server provides a ready-to-use option. For this use case, we built a custom Python MCP server using FastMCP to expose investigation-specific tools — trace ID correlation, time-window log retrieval, and latency analysis — tailored to our message traceability workflow.

The custom server implementation provides the thirteen tools required for log search, index discovery, and aggregation.

Table: ELK MCP Server tools access by AWS DevOps Agent during investigations
MCP Tool Description
search_logs Search by Lucene query string, time range, and log level
get_error_summary Top recurring errors within a time window
get_recent_logs Fetch most recent log entries from an index
get_logs_by_service Filter logs by service or application name
count_logs_by_level Breakdown of log counts by level (ERROR, WARN, INFO, DEBUG)
get_slow_requests Find requests exceeding a latency threshold
get_logs_around_time Fetch logs within ±N minutes of a specific timestamp
search_by_trace_id Find all logs for a specific trace, request, or correlation ID
get_unique_errors Get distinct error messages in a time window
list_indices List all available Elasticsearch indices with doc counts and health
get_index_stats Get size, document count, and health of a specific index
get_logs_by_host Filter logs by the hostname that sent them via Filebeat
get_logs_by_file Filter logs by the source log file path
  1. Launch an Ubuntu instance (t4g.medium or larger) with a security group allowing inbound TCP 443 from AWS DevOps Agent service endpoints.Note: The single-instance deployment described here is intended for demonstration purposes and does not provide high availability. For production workloads requiring managed infrastructure, automatic scaling, and built-in resilience for your MCP servers, consider using Amazon Bedrock AgentCore Runtime.
  2. Install dependencies and obtain a TLS certificate:
    sudo apt update &&sudo apt install -y python3 python3-venv certbot
    python3 -m venv ~/elk-mcp-venv
    source ~/elk-mcp-venv/bin/activate
    pip install mcp elasticsearch uvicorn starlette
    sudo certbot certonly --standalone -d elk-mcp.yourcompany.com
  3. Create the MCP server. The key architectural decisions are: FastMCP for the Streamable HTTP transport, Starlette middleware for API key authentication, and environment variables for configuration: 
    import json, os
from mcp.server.fastmcp import FastMCP
from elasticsearch import Elasticsearch
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.responses import JSONResponse

ES_HOST = os.environ.get("ES_HOST", "http://localhost:9200")
API_KEY = os.environ.get("MCP_API_KEY", "YOUR_API_KEY")
es = Elasticsearch(hosts=[ES_HOST])
mcp = FastMCP("elk-logs", host="elk-mcp.yourcompany.com")

# API key authentication middleware
class APIKeyMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        if request.headers.get("x-api-key") != API_KEY:
            return JSONResponse({"error": "Unauthorized"}, status_code=401)
        return await call_next(request)

# Example: trace ID correlation tool
@mcp.tool
def search_by_trace_id(trace_id: str, index: str, size: int = 100) -> str:
    """Find all logs for a specific request or trace ID."""
    result = es.search(index=index, body={
        "query": {"multi_match": {
            "query": trace_id,
            "fields": ["trace_id", "request_id", "correlation_id", "traceId"]
        }},
        "sort": [{"@timestamp": "asc"}],
        "size": size
    })
    return json.dumps([h["_source"] for h in result["hits"]["hits"]], indent=2)

# ... additional tools follow the same pattern

app = mcp.streamable_http_app()
app.add_middleware(APIKeyMiddleware)

if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=443,
        ssl_keyfile="/etc/letsencrypt/live/elk-mcp.yourcompany.com/privkey.pem",
        ssl_certfile="/etc/letsencrypt/live/elk-mcp.yourcompany.com/fullchain.pem")
  4. Start the server: 
    sudo -E ES_HOST=http://<ELASTICSEARCH_IP>:9200 MCP_API_KEY=<YOUR_API_KEY> nohup ~/elk-mcp-venv/bin/python ~/elk_mcp_server.py > ~/mcp.log 2>&1 &
  5. Register in AWS DevOps Agent:
    1. In the AWS DevOps Agent console, navigate to Integrations -> Add MCP Integration.
    2. Enter the endpoint URL: https://elk-mcp.yourcompany.com:443/mcp
    3. Enter the API key for authentication.
    4. Verify the integration shows all available tools in the integration detail view.
    5. Add the MCP integration to your Agent Space under the Integrations tab.

Note: The MCP server must be publicly accessible over HTTPS. If your Elasticsearch cluster is in a private VPC, deploy the MCP server with network access to the cluster (e.g., in the same VPC or a peered VPC) while exposing only the MCP server endpoint publicly. Use security group rules to restrict access to AWS DevOps Agent’s known egress IP ranges where possible.

Step 4: Deploy the Application and Configure Filebeat on EKS

Filebeat runs as a Kubernetes DaemonSet on each EKS cluster, collecting pod logs and enriching them with Kubernetes metadata before forwarding to Elasticsearch. The following pipeline configuration ensures that message IDs and trace IDs are preserved as indexed fields, enabling efficient targeted queries during investigations.

  1. Clone the Repository and Build the Container Image: 
    # Clone the DevOps agent sample repository
git clone https://github.com/aws-samples/Amazon-prometheus-bedrock-agent-example.git

# Navigate to the smart demo directory
cd devops-agent/smart-demo-main/

# Authenticate Docker to your ECR registry
aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin <your-account-id>.dkr.ecr.us-west-2.amazonaws.com

# Build the container image
docker build -t sample-app .

# Tag the image for ECR
docker tag sample-app:latest <your-account-id>.dkr.ecr.us-west-2.amazonaws.com/smart-demp:latest

# Push the image to ECR
docker push <your-account-id>.dkr.ecr.us-west-2.amazonaws.com/smart-demp:latest
  2. Deploy the Application to EKS: 
    # Deploy the sample application to EKS using Helm
helm install sample-app ./helm --set image.repository=<your-account-id>.dkr.ecr.us-west-2.amazonaws.com/smart-demp --set image.tag=latest
  3. Deploy Filebeat as DaemonSet: 
    # Navigate to the Filebeat directory
cd filebeat

# Apply the Filebeat ConfigMap (autodiscovery, JSON parsing, K8s metadata enrichment, Logstash output)
kubectl apply -f filebeat-configmap.yaml

# Deploy Filebeat as a DaemonSet on every node in the cluster
kubectl apply -f filebeat-ds.yaml

Step 5: Configure Datadog Webhook for Automatic Investigation Triggering

We will automatically trigger AWS DevOps Agent investigations when Datadog alerts fire. This eliminates the human latency between alert detection and investigation initiation.

Retrieve the AWS DevOps Agent webhook URL and secret:

  1. In the AWS DevOps Agent console, navigate to your Agent Space and open the Capabilities tab.
  2. Under the Webhook section, choose Configure, then Generate webhook.
  3. Save the webhook URL and HMAC secret. These credentials are used to authenticate webhook requests from Datadog.

In Datadog, configure a webhook integration:

  1. Navigate to Integrations -> Webhooks and create a new webhook.
  2. Set the URL to the AWS DevOps Agent webhook endpoint.
  3. Add the Authorization header with the bearer token from step 3.
  4. Configure the payload to include the message ID, trace ID, and alert context: 
    {
"title": "Message Processing Failure - $EVENT_TITLE",
"description": "$EVENT_MSG",
"alert_id": "$ALERT_ID",
"alert_status": "$ALERT_STATUS",
"timestamp": "$TIMESTAMP",
"message_id": "$tags.message_id",
"trace_id": "$tags.trace_id",
"service": "$tags.service",
"cluster": "$tags.cluster_name",
"severity": "$ALERT_PRIORITY"
}
  5. Associate the webhook with the Datadog monitors that detect message processing failures by adding @webhook-webhook-name to the monitor notification message.

Step 6: Configure Agent Space Skills (Optional but Recommended)

AWS DevOps Agent Skills provide a Retrieval-Augmented Generation (RAG) knowledge base that gives the agent organization-specific context during investigations. Even a brief skills document (2-3 paragraphs) that identifies your application’s purpose, its key components, and its observability backends can reduce AWS DevOps Agent investigation time by helping the agent understand context before executing its first queries.

In our implementation, the skills document described the sample message-processing application, identified Elasticsearch as the logging backend, and identified Datadog as the metrics backend.

Real-World Investigation: Message Processing Failure Diagnosed in 6 Minutes

The following walkthrough and the scenario represent a common class of incident in distributed systems: a silent functional regression introduced through a new container image deployment that causes specific message types to fail processing without immediately obvious symptoms.

The Scenario

The production EKS cluster runs a message-processing application (sample-app) with four HTTP endpoints:

  • /health – Application health check
  • /metrics – Prometheus metrics endpoint
  • /process – Core message processing endpoint
  • /alert – Alert notification endpoint (newly introduced in recent deployment)

A new container image was pushed to Amazon ECR with a new /alert endpoint. However, the endpoint implementation was incomplete when called; it returned an HTTP 404 response and silently dropped the associated message. The Filebeat DaemonSet collected pod logs and sent them into Elasticsearch. Datadog captured application metrics with message ID and trace ID labels. A Datadog monitor detected the elevated error rate and fired.

Phase 1: Alert Fires and Investigation Initiates (T+0:00)

At 14:52:57 UTC, a Datadog monitor detected an elevated rate of failed message processing requests on the /alert endpoint. The monitor fired a webhook to the AWS DevOps Agent endpoint with the alert payload context.

Within 10 seconds, the Agent read its investigation skills (sample-app-incident and triaging-3p-monitoring-alerts) and began planning the investigation approach.

Screenshot of AWS DevOps Agent console showing an investigation automatically initiated via Datadog webhook. The alert payload context includes message ID, trace ID, and alert timestamp. Investigation skills sample-app-incident and triaging-3p-monitoring-alerts are loaded
Figure 2: AWS DevOps Agent investigation initiated automatically via Datadog webhook, showing alert payload context and investigation skills loaded

Phase 2: Cross-Source Signal Correlation (T+0:10 – T+2:30)

AWS DevOps Agent began its investigation by using the message ID from the alert payload as its primary search key. It decided its investigation strategy: extract the trace_id from Elasticsearch using the message_id, get Datadog monitor details, and search for errors around the alert time.

Elasticsearch Log Search
The agent invoked the ELK MCP server to list available indices and identify the relevant log store for the message-processor application. It then executed a targeted search and identified the relevant Elasticsearch index (logs-2026.03.25):

Screenshot of AWS DevOps Agent querying the ELK MCP Server to correlate a message ID to a trace ID across Elasticsearch indices. The agent identifies the relevant log index logs and retrieves matching log entries
Figure 3: AWS DevOps Agent querying the ELK MCP Server to correlate message ID to trace ID across Elasticsearch indices

AWS DevOps Agent surfaced the Message ID to Trace ID correlation without any explicit cross-referencing instruction. It recognized the relationship from the log structure. At T+1:41, the DevOps Agent launched three parallel tasks simultaneously, rather than investigating sequentially.

Screenshot showing the trace ID successfully resolved from the Elasticsearch log structure, triggering three parallel investigation tasks: Datadog metrics correlation, EKS topology analysis, and CloudTrail event correlation.Figure 4: Trace ID successfully resolved from log structure, triggering three parallel investigation tasks

Datadog Metrics Correlation
Using the trace ID extracted from the Elasticsearch logs, the agent queried Datadog for correlated metrics. It retrieved CPU and memory utilization, pod count, restart metrics and enhanced metrics for request latency and errors.

Screenshot of Datadog metrics retrieved by AWS DevOps Agent using the extracted trace ID, showing CPU utilization, memory usage, pod restart counts, and request latency metrics for the affected service.
Figure 5: Datadog metrics correlation showing CPU, memory, pod restarts, and request latency retrieved using the extracted trace ID

AWS EKS Topology & CloudTrail Event Correlation
AWS DevOps Agent queried AWS CloudTrail for deployment and configuration change events in the time window preceding the alert.

Screenshot of AWS CloudTrail event correlation identifying deployment and configuration change events in the time window preceding the alert, including ECR image push and EKS rolling deployment events
Figure 6: AWS CloudTrail event correlation identifying deployment and configuration changes in the alert time window

With the initial timeline established, AWS DevOps Agent examined all endpoints and their response patterns. This revealed that the application was fundamentally healthy, with three of four endpoints returned consistent 200 responses. However, it also revealed the anomaly was isolated to the /alert endpoint, which had never successfully served a request in its observable history.

Screenshot of AWS DevOps Agent endpoint analysis showing three healthy endpoints returning HTTP 200 responses (health, metrics, process) and the alert endpoint returning consistent HTTP 404 failures, isolating the anomaly to the newly deployed alert endpoint.
Figure 7: AWS DevOps Agent endpoint analysis revealing isolated 404 failures on the /alert endpoint while other endpoints remain healthy

Phase 3: Observation Streaming from Parallel Tasks (T+2:30 – T+3:54)

As the three tasks ran simultaneously, observations streamed in chronologically:

T+2:58 – Observation: Anomalous CPU Behavior Signals Pod Disruption

The first sign of trouble came from pod tr8pl. Its CPU usage dropped sharply. Around the same time, two unfamiliar pods (85bx4, 7h4bd) briefly appeared with minimal CPU. Shortly after, three new pods (hclf9, b7bp2, g6gkc) spun up. This pattern of old pods dying, short-lived intermediaries, and fresh containers starting up pointed strongly toward a rolling deployment in progress.

T+3:04 – Observation: ECR Image Push Traced as the Trigger

With the deployment pattern established, the next question was: what initiated it? CloudTrail provided the answer. At 14:57:19 UTC, a user vik**** (via role nht-admin) pushed a new container image to ECR repository sm***demp:latest (ECR image tag). The timeline now made sense:

  1. ECR image push at 14:57:19 UTC
  2. Rolling deployment (pods replaced)
  3. /alert endpoint 404 errors begin
  4. Datadog alert fires

Screenshot showing parallel task observations streaming into AWS DevOps Agent. Anomalous CPU behavior indicates pod disruption from a rolling deployment, and CloudTrail evidence links an ECR image push by user vik at 14:57:19 UTC to the deployment that introduced the failing endpoint.
Figure 8: Parallel task observations streaming into AWS DevOps Agent – anomalous CPU behavior indicating pod disruption and CloudTrail evidence linking the ECR image push to the rolling deployment

Phase 4: Findings Documented – Causal Chain Established (T+3:55 – T+5:08)

At T+3:55, the Agent documented its first formal Finding (elevated from Observation):

Screenshot of AWS DevOps Agent documenting its formal finding, establishing the causal chain: ECR image push triggered rolling deployment, new container image contained incomplete alert endpoint implementation, endpoint returns 404 and drops messages.
Figure 9: AWS DevOps Agent formal finding documenting the causal chain from ECR image push to alert endpoint failure

Phase 5: Root Cause Confirmation and Infrastructure Validation (T+5:10 – T+5:30)

AWS DevOps Agent validated infrastructure health. The complete absence of infrastructure issues, combined with the timeline evidence from CloudTrail and the historical 404 pattern on the /alert endpoint, allowed AWS DevOps Agent to deliver a high-confidence root cause identification.

At 14:58:12 UTC, exactly 5 minutes and 14 seconds (under 6 minutes) after the investigation began, AWS DevOps Agent delivered its root cause analysis:

Screenshot of the final root cause analysis delivered by AWS DevOps Agent at 14:58:12 UTC, 5 minutes and 14 seconds after investigation began. Root cause identified as an incomplete alert endpoint in the newly deployed container image that returns HTTP 404 and silently drops associated messages.
Figure 10: Final root cause analysis delivered by AWS DevOps Agent, identifying the incomplete alert endpoint in the newly deployed container image

Clean-up

Step 1: Delete the AWS DevOps Agent AgentSpace

  1. In the AWS DevOps Agent console, navigate to Agent Spaces.
  2. Select the Agent Space you created for this walkthrough.
  3. Remove all integrations (Datadog, ELK MCP Server) from the Agent Space by navigating to the Capabilities tab and choosing Remove for each.
  4. Choose Delete Agent Space from Actions dropdown and confirm the deletion.

Step 2: Terminate the Microservices and Delete the EKS Cluster

First, remove the application workloads and Filebeat DaemonSet deployed on the cluster:

# Uninstall the sample application Helm release
helm uninstall sample-app

# Delete the Filebeat DaemonSet and ConfigMap
kubectl delete -f filebeat/filebeat-ds.yaml
kubectl delete -f filebeat/filebeat-configmap.yaml

Once the workloads are removed, delete the EKS cluster.

Step 3: Terminate EC2 Instances

Terminate the EC2 instances hosting both the MCP server and the Elasticsearch cluster. You can do this from the AWS Management Console or the AWS CLI:

# Terminate the MCP server EC2 instance
aws ec2 terminate-instances --instance-ids <MCP_SERVER_INSTANCE_ID> --region <REGION>

# Terminate the Elasticsearch EC2 instances
aws ec2 terminate-instances --instance-ids <ELASTICSEARCH_INSTANCE_ID> --region <REGION>

Conclusion

Distributed systems have created a correlation problem that scales faster than the human capacity to solve it. As microservices architectures grow to span dozens of clusters, hundreds of services, and terabytes of daily telemetry, the manual investigation practices that worked at smaller scale become the primary obstacle in maintaining operational quality.

AWS DevOps Agent addresses this challenge at its root by automating the multi-source correlation that previously required experienced engineers working across multiple systems. The combination of native Datadog integration, custom ELK MCP server connectivity, and AWS CloudTrail access enables AWS DevOps Agent to build the complete picture of an incident: from the first metric anomaly, through the log evidence, to the deployment event that caused it. The scenario described in this post demonstrates that a message processing incident that previously consumed hours can be diagnosed to root cause in under six minutes automatically, without manual intervention, and with the full investigation documented for audit and learning purposes.

About the Authors

Bhuvan Jain

Bhuvan Jain

Bhuvan is a Senior Technical Account Manager at Amazon Web Services, supporting independent software vendor (ISV) customers. He is passionate about helping customers build Well-Architected solutions on AWS, with a focus on enterprise-scale networking. As a subject matter expert, Bhuvan offers guidance on designing network architectures that are highly available, resilient, and cost-effective. He holds a Master’s degree in Electrical and Computer Engineering from the University of Illinois at Chicago (UIC). In his free time, he enjoys playing basketball and volleyball, as well as watching movies and TV series.

Vikram Venkataraman

Vikram Venkataraman

Vikram Venkataraman is a Principal Specialist Solutions Architect at Amazon Web Services. He helps customers modernize, scale, and adopt best practices for containerized workloads on Amazon EKS. With the emergence of AI-powered automation, Vikram has been actively working with customers to leverage AWS AI/ML services to solve complex operational challenges, streamline monitoring workflows, and enhance incident response through intelligent automation. He designed and built the POC architecture described in this post and is a co-author of the EKS knowledge graphs blog.

A systematic approach to benchmarking SQL processing engines on AWS

Post Syndicated from Anubhav Awasthi original https://aws.amazon.com/blogs/big-data/a-systematic-approach-to-benchmarking-sql-processing-engines-on-aws/

Selecting the right SQL processing solution for large-scale data analytics is a critical decision for organizations. As data volumes grow exponentially, the technology landscape has evolved to offer diverse options for processing and analyzing this information efficiently. This post presents a systematic framework for evaluating and benchmarking SQL processing engines on AWS, using Apache JMeter to conduct practical performance testing at scale.

The AWS analytics ecosystem

AWS offers a rich portfolio of SQL processing solutions to meet various analytical needs:

  • Serverless query services – Amazon Athena is a serverless, interactive query service that uses standard SQL to analyze data in Amazon Simple Storage Service (Amazon S3), offering automatic scaling, parallel query execution, and pay-per-query pricing with no infrastructure management required
  • Data warehouse solutions – Amazon Redshift offers scalable, high-performance cloud data warehousing with serverless options, zero-ETL integrations, AI-powered query assistance, and seamless machine learning (ML) integration for modern analytics at scale
  • Managed open source engines – Amazon EMR supports Apache Spark SQL, Apache Trino (formerly PrestoSQL), and other distributed query frameworks
  • Self-managed options – You can deploy open source engines like Apache Spark, Apache Flink, and Trino on Amazon Elastic Kubernetes Service (Amazon EKS) for greater control
  • Partner solutions – You can access specialized big data analytics tools through AWS Marketplace

These options are further enhanced by modern open table formats such as Apache Iceberg, Delta Lake, and Apache Hudi, which bring crucial enterprise features like ACID (Atomicity, Consistency, Isolation, and Durability) transactions, schema evolution, and time travel capabilities to data lakes. These SQL processing solutions operate under the AWS Shared Responsibility Model. AWS manages the security of the underlying infrastructure and services, and customers are responsible for secure configuration, access management, and data protection within their testing environments. This division of responsibility remains important when evaluating and benchmarking different SQL engines. Proper security configuration and implementation by customers is essential for maintaining a secure analytics environment.

Evaluation challenges in SQL engine selection

The rich ecosystem of SQL processing options creates significant evaluation challenges. Each SQL engine employs unique architectural approaches and optimization strategies, making direct comparisons complex. Organizations embarking on this evaluation journey face several interconnected obstacles:

  • Creating environments that accurately reflect production scenarios
  • Developing test datasets that mirror real-world data characteristics and volumes
  • Replicating real-world query patterns and concurrency levels
  • Maintaining uniform testing conditions across different engine architectures
  • Controlling infrastructure expenses throughout the evaluation process

Performance considerations at petabyte scale

When evaluating solutions for petabyte-scale deployments, the complexity intensifies considerably. Several critical factors come into play:

  • Resource management – Distributed SQL engines require precise balancing of CPU, memory, and storage resources. Suboptimal resource allocation can lead to query failures and performance degradation, particularly as data volumes grow.
  • Data distribution patterns – How data is distributed across partitions or nodes significantly impacts query performance. Data skew can create processing bottlenecks, with some nodes handling disproportionate workloads while others remain underutilized.
  • Concurrency handling – High-concurrency environments demand sophisticated workload scheduling and resource isolation mechanisms. The ability to maintain consistent performance under varying concurrent loads becomes a critical differentiator between solutions.
  • Meaningful metrics – Performance evaluation at scale requires comprehensive metrics analysis:
    • Mean, median, and percentile response times (particularly p90 and p95)
    • Query throughput under varying concurrency levels
    • Scalability characteristics across diverse workload types
    • Resource utilization efficiency during peak loads

Limitations of traditional benchmarks

Although industry-standard benchmarks like TPC-DS and TPC-H provide valuable insights, our experience with multiple customer engagements has shown that tailored, workload-specific testing often reveals performance characteristics not captured by these standardized tests. This is especially true for complex, multi-tenant environments with diverse query patterns. Organizations that complement standard benchmarks with workload-specific testing typically experience shorter proof-of-concept cycles, optimized evaluation costs, and more efficient testing operations. This comprehensive approach helps reduce uncertainty in the final solution selection process.

Prerequisites

Before you dive into the evaluation process, make sure you have the following prerequisites:

  • An AWS account with appropriate permissions to create and manage Amazon Elastic Compute Cloud (Amazon EC2) instances and access the SQL engines you plan to benchmark.
  • Basic familiarity with AWS services, particularly Amazon EC2 and the SQL engines you intend to evaluate (such as Athena, Amazon Redshift, or Amazon EMR).
  • Experience with SQL and data analytics concepts.
  • Access to the SQL engines you choose to benchmark. This post assumes you’ve already set up the engines you want to test. For setup instructions, refer to the AWS documentation for each service.
  • A dataset suitable for your benchmarking needs. Dataset creation and loading are not covered in this post. Build petabyte-scale synthetic test data with Amazon EMR on EC2 provides prescriptive guidance to generate test datasets at scale. Make sure your test datasets are stored in S3 buckets with encryption enabled (using SSE-KMS or SSE-S3) and that all service connections use TLS for data in transit.

Benefits of Apache JMeter

As organizations scale their analytics workloads to petabyte levels, there is a growing need for a robust, structured approach to SQL query performance testing. Although many organizations develop custom testing frameworks or use various benchmarking tools, these approaches often lack standardization and can be difficult to replicate across different SQL engines. The complexity of modern data architectures, combined with the variety of available SQL processing solutions, demands a systematic evaluation methodology. Apache JMeter emerges as a powerful solution to address this challenge. Though traditionally known for web application testing, JMeter’s extensible architecture and robust feature set make it particularly well-suited for SQL performance testing at scale.JMeter offers several advantages for evaluating SQL engines:

  • Support for multiple protocols and connections
  • Ability to simulate complex concurrent workloads
  • Built-in performance metrics and reporting
  • Extensible architecture for custom testing scenarios
  • Integration capabilities with continuous integration and continuous delivery (CI/CD) pipelines

Through this proposed framework, which has been validated across multiple customer engagements at petabyte scale, we aim to help organizations make more informed decisions when selecting a SQL processing solution. Our experience working with customers to assess various AWS Analytics services and open source solutions has demonstrated that a systematic evaluation approach significantly reduces proof-of-concept cycles and optimizes resource investments. This framework has helped organizations effectively evaluate services like Athena, Amazon Redshift, and Amazon EMR, alongside open source solutions such as Trino on Amazon EKS, based on their specific workload profiles and performance requirements.With this methodology, organizations can accomplish the following:

  • Navigate the complex landscape of large-scale data processing technologies
  • Reduce proof-of-concept cycles from months to weeks
  • Minimize infrastructure costs during evaluation phases
  • Make data-driven decisions about technology selection
  • Better align technology choices with business requirements
  • Establish repeatable testing patterns for future evaluations

Testing methodology in practice

A successful SQL engine evaluation requires understanding and replicating real-world workload patterns. Our methodology, refined through numerous customer engagements, focuses on comprehensive testing across multiple dimensions while remaining adaptable to specific organizational needs.

Query pattern selection

We begin by selecting representative query patterns that mirror production workloads:

  • Aggregation queries that summarize large datasets using operations like SUM, AVG, and COUNT
  • Complex join operations that test the engine’s ability to combine data efficiently across multiple tables
  • String operations that evaluate text processing capabilities
  • Nested queries that assess the engine’s optimization capabilities for complex query structures

A carefully selected set of 8–10 queries typically provides sufficient coverage while keeping the evaluation manageable. These should reflect your actual workload characteristics and business requirements.

Data volume variations

Testing across different data volumes is important for understanding scalability characteristics. We structure our tests around varying data scan ranges:

  • Small-scale scans – Queries accessing 1–7 days of data (megabytes to gigabytes)
  • Large-scale scans – Queries spanning 14–30 days (terabytes to petabytes)

This approach evaluates both I/O efficiency with large datasets and metadata handling with smaller, frequent queries, helping understand how services like Amazon EMR, Amazon Redshift, or Athena optimize query execution across different access patterns.

Concurrency testing

Real-world analytics environments rarely process single queries in isolation. Our methodology incorporates the following features:

  • Progressive concurrency testing starting at lower levels (typically 16, 32, 64, and 128 parallel queries), though these numbers can be adjusted based on your test infrastructure capacity and specific requirements. We recommend starting with smaller concurrency levels and gradually scaling up to understand performance characteristics
  • Varied query complexity and frequency (referred to as query weights) to simulate realistic workload distributions. This means some queries are run more often or are more resource-intensive than others, mimicking real-world usage patterns.
  • Mixed query patterns running simultaneously to test resource management.
  • Consistent execution across different date ranges to evaluate scaling behavior.

This approach is particularly important when evaluating managed services like the workload management capabilities of Amazon Redshift or the resource allocation strategies of Amazon EMR.

Query weight distribution

Production environments typically see varying frequencies of different query types. Our framework incorporates weighted query distribution to simulate real-world scenarios more accurately. In a typical distribution, frequent lightweight queries might represent 60% of the workload, complex analytical queries might comprise 30%, and resource-intensive data processing operations might make up the remaining 10%.This weighted approach makes sure performance testing reflects actual usage patterns rather than artificial benchmarking scenarios. The exact distribution should mirror your organization’s specific workload patterns.

Sequential vs. concurrent testing

Our methodology implements two distinct testing phases:

  • Sequential testing – Establishes baseline performance metrics:
    • Runs each query type independently across different date ranges
    • Runs multiple iterations to provide consistency and identify variability
    • Helps understand individual query performance characteristics
  • Concurrent testing – Simulates real-world multi-user scenarios:
    • Implements weighted query distributions
    • Tests different concurrency levels to identify scaling limitations
    • Evaluates resource management capabilities of different engines

JMeter efficiently implements both testing phases while maintaining consistent test conditions across SQL engines. Its ability to handle various JDBC connections makes it particularly suitable for testing AWS analytics services.Through this structured approach, organizations can gather comprehensive performance data reflecting their specific use cases, enabling informed SQL engine selection decisions while maintaining core principles of systematic evaluation and realistic workload simulation.

Test plans

To evaluate SQL engines’ performance under varying workloads, we designed two test scenarios: sequential and concurrent execution plans. Each scenario was executed across different data volumes by adjusting the query date range filters to cover 1, 7, 14, and 30 days. These variations simulate typical analytical workloads with progressively increasing data sizes.For sequential runs, each test was treated as a distinct batch, grouping all queries (Query 1 to Query 9) under the same date range—each query will scan data for 1, 7, 14, and 30 days with appropriate date filtering in the query’s where predicate. We used JMeter to capture average query response times for each batch. This configuration was run three times, and the final metrics reflect the average response time across these iterations to ensure reliability and account for environmental variance.Although three iterations provide initial insights, if you observe significant variations in results (typically more than 10% deviation between runs), consider expanding to 10 or more iterations. This additional sampling helps establish statistical significance, identify true performance patterns, and distinguish outliers (beyond three standard deviations) from normal variations. Document any consistent anomalies, because they may indicate important performance or security considerations for your specific environment.The following table shows the sample test plans template for the sequential test plan run.

 

Dataset Time Range Run Query Weights
Query 1 Query 2 Query 3 Query 4 Query 5 Query 6 Query 7 Query 8 Query 9
1 day Run 1
Run 2
Run 3
Avg
7 days Run 1
Run 2
Run 3
Avg
14 days Run 1
Run 2
Run 3
Avg
30 days Run 1
Run 2
Run 3
Avg

For the concurrent test plan, we introduced a probabilistic weighted distribution to the queries (Query 1 to Query 9), simulating a more realistic production-like environment where query frequency varies based on business relevance and usage patterns. This added a layer of complexity to better reflect how the SQL engine would perform under real-world concurrent access patterns.The following table shows the sample test plans template for the concurrent test plan run.

Dataset Time Range Concurrent Runs Query Weights
Query 1 Query 2 Query 3 Query 4 Query 5 Query 6 Query 7 Query 8 Query 9
1 days 8 11% 11% 11% 11% 11% 11% 11% 11% 11%
16 10% 5% 24% 5% 5% 5% 24% 14% 10%
32 8% 3% 24% 5% 5% 5% 24% 16% 8%
64 7% 3% 24% 6% 4% 6% 26% 16% 9%
128 1% 4% 19% 8% 5% 7% 14% 20% 22%
*7 days 8 11% 11% 11% 11% 11% 11% 11% 11% 11%
16 10% 5% 24% 5% 5% 5% 24% 14% 10%
32 8% 3% 24% 5% 5% 5% 24% 16% 8%
64 7% 3% 24% 6% 4% 6% 26% 16% 9%
**128 1% 4% 19% 8% 5% 7% 14% 20% 22%
14 days 8 11% 11% 11% 11% 11% 11% 11% 11% 11%
16 10% 5% 24% 5% 5% 5% 24% 14% 10%
32 8% 3% 24% 5% 5% 5% 24% 16% 8%
64 7% 3% 24% 6% 4% 6% 26% 16% 9%
128 1% 4% 19% 8% 5% 7% 14% 20% 22%
30 days 8 11% 11% 11% 11% 11% 11% 11% 11% 11%
16 10% 5% 24% 5% 5% 5% 24% 14% 10%
32 8% 3% 24% 5% 5% 5% 24% 16% 8%
64 7% 3% 24% 6% 4% 6% 26% 16% 9%
128 1% 4% 19% 8% 5% 7% 14% 20% 22%

For example, for configuration of *7 days concurrent run with **128 concurrency, the proposed configuration distributes Query 1 to Query 9 with appropriate weighted submissions such that Query 9 is executed the greatest number of times in the overall 128 executions submitted across all 9 queries for this run.

JMeter setup

To begin, you must set up JMeter on a machine that can handle the desired test load. An EC2 instance is a flexible and cost-effective option. Choose an instance type with sufficient vCPUs to support your maximum planned concurrency. For example, a c6i.4xlarge or higher is typically suitable for moderate to high throughput testing scenarios. For the operating system, you might choose Amazon Linux, which is optimized for AWS. For production-grade testing environments, deploy the JMeter EC2 instance in a private subnet of a virtual private cloud (VPC) with appropriate security groups that allow only required connections. This network isolation helps maintain security while executing performance tests. Consider using Amazon Virtual Private Cloud (Amazon VPC) endpoints for secure access to AWS services.

After the instance is provisioned, install Java (Java 17 LTS or Java 21 LTS) and download the latest version of JMeter. Be sure to configure the system with appropriate JVM options to allocate sufficient heap memory for large-scale test executions. Refer to Getting Started to learn more.

# Install Java
sudo yum update -y # For Amazon Linux
sudo yum install java-17-amazon-corretto -y

# Download JMeter and place the appropriate jdbc driver for the engine of your selection under lib folder
wget https://downloads.apache.org//jmeter/binaries/apache-jmeter-5.6.3.tgz
tar -xvzf apache-jmeter-5.6.3.tgz
cd apache-jmeter-5.6.3/lib

# Launch JMeter in GUI mode (if using a GUI-capable setup) or use CLI for remote testing
./bin/jmeter

JMeter concepts

Before you create test plans in JMeter, it’s important to understand a few foundational concepts that influence how your test plan behaves—such as thread groups, user-defined variables, and JDBC connection. These components enable the simulation of real-world query loads, including concurrency and pacing.

Test plans

The test plan is the top-level container for a JMeter test. It defines the overall testing strategy, including the queries to execute, their parameters, and the concurrent user behavior. These plans are represented as jmx files that can then be used for CLI-based execution. JMeter supports both GUI and CLI modes. It is highly recommended that you use the JMeter GUI primarily for creating test plans as jmx, and use the CLI for large load tests. You can also run thread groups consecutively for sequential execution. The default behavior is to run all thread groups in parallel suited for concurrent execution. Refer to Building a Test Plan to learn more about options available with test plans.

User-defined variables

User-defined variables are global parameters that you can reuse throughout the test plan. They are helpful for defining database credentials, server URLs, or query parameters. For example:DB_URL=jdbc:trino://trino-cluster.example.com:8889?SSL=true #Enable SSL/TLS

You can configure authentication (user name and password) through your organization’s approved methods, such as AWS Secrets Manager (see Move hardcoded secrets to AWS Secrets Manager) AWS Identity and Access Management (IAM) roles, or other secure credential management systems.

Thread groups

A thread group represents a group of virtual users (threads) executing test actions. Each thread simulates a single user sending requests to the SQL engine. This can be used to simulate concurrent runs. For example, in the preceding template, Query 3 has 19% weightage across 128 runs. This means .19*128=25 total runs, so we set the thread group to 25.

JDBC connection configuration

JDBC connection configuration sets up the database connection for the test. It specifies the database URL, driver, and credentials required for executing SQL queries. Key fields to configure are database URL and JDBC driver class. The following table summarizes the different configuration settings.

 

SQL Engine JDBC Driver JDBC Driver Class
Trino on EMR trino-jdbc-<trino_version>-amzn-0.jar io.trino.jdbc.TrinoDriver
Athena Athena JDBC 3.x driver com.amazon.athena.jdbc.AthenaDriver
Amazon Redshift Amazon Redshift JDBC driver com.amazon.redshift.jdbc.Driver
Trino on EKS Trino JDBC driver io.trino.jdbc.TrinoDriver

JDBC requests

The JDBC request executes SQL queries against the database using the configuration defined in the JDBC connection configuration.

For example, following command runs the JMeter in CLI mode:

# Run benchmarks in CLI mode 
./jmeter -n -t <path_to>.jmx -l <local path for log>.log -e -o <local path for>/output/

The output folder will contain an HTML report with different statistics. The following screenshot illustrates 128 concurrent runs.

Monitoring and logging

For comprehensive visibility and audit requirements, enable AWS CloudTrail logging, VPC Flow Logs, and service-specific logs (like Amazon S3 access logs). These logs can be centralized in Amazon CloudWatch Logs for monitoring and analysis. This provides proper audit trails while evaluating different SQL engines and helps track access patterns and potential security events.

Post-test steps

After running your JMeter tests, proceed with the following steps:

  1. Review the HTML report’s key metrics, including response times, throughput, and error rates across different query types and concurrency levels.
  2. Run identical test plans across your candidate SQL engines for direct performance comparison.
  3. Refine your test plans based on initial findings, focusing on areas where performance differences are significant.
  4. Factor in the cost implications alongside performance metrics to make a balanced decision.

These steps can help you systematically evaluate and select the most suitable SQL engine for your analytics workloads.

Resources

In the preceding steps, we walked through a UI-based setup for JMeter along with test plans. We have created a few sample JMeter test plans for both sequential and concurrent runs along with sample test reports. You can modify the plans to fit your needs.

  1. JMeter sample report
  2. JMeter test plan for sequential run
  3. JMeter test plan for concurrent run

Clean up

After you complete your benchmarking process, clean up the resources to avoid unnecessary costs:

  1. Stop or delete the EC2 instances used for running JMeter.
  2. Depending on which SQL engines you used for testing, clean up active resources.
  3. Review your AWS Management Console to confirm no active resources remain.
  4. If you created test datasets in Amazon S3 or other storage services specifically for this benchmarking, consider deleting them if they’re no longer needed.
  5. Although JMeter test plans and results don’t incur AWS costs, organize or delete local files as needed for your record-keeping.

Summary

Selecting the right SQL processing solution for large-scale analytics demands a systematic, data-driven approach. Our JMeter framework can help organizations effectively evaluate different SQL engines by simulating real-world workload patterns across various query types, data volumes, and concurrency levels. This methodology reduces proof-of-concept cycles and provides insights beyond traditional benchmarks, helping you assess managed AWS services like Athena and Amazon Redshift and open source solutions on Amazon EKS.

About the authors

Anubhav Awasthi

Anubhav Awasthi

Anubhav is a Senior Big Data Specialist Solutions Architect at Amazon Web Services (AWS). He collaborates with customers to provide expert architectural guidance for implementing and optimizing analytics solutions using Amazon EMR, Amazon Athena, AWS Glue, and AWS Lake Formation.

Gagan Brahmi

Gagan Brahmi

Gagan is a Specialist Senior Solutions Architect at Amazon Web Services (AWS), focused on Data Analytics and AI/ML. With over 20 years in information technology, he partners with customers to solve complex AI/ML challenges by leveraging data and AI/ML platforms. Gagan helps customers architect scalable, high-performance solutions that utilize distributed data processing, real-time streaming technologies, and AI/ML services to drive business transformation through artificial intelligence and data-driven insights. When not designing cloud-native data and AI solutions, Gagan enjoys exploring new places with his family.

Jayaprakash Boreddy

Jayaprakash Boreddy

Jayaprakash is a Senior Solutions Architect at AWS. He works with ISV customers in designing and building highly scalable, flexible and resilient applications on AWS Cloud.

Sahil Thapar

Sahil Thapar

Sahil is a Principal Solutions Architect. He works with ISV customers to help them build highly available, scalable, and resilient applications on the AWS Cloud.

Build petabyte-scale synthetic test data with Amazon EMR on EC2

Post Syndicated from Anubhav Awasthi original https://aws.amazon.com/blogs/big-data/build-petabyte-scale-synthetic-test-data-with-amazon-emr-on-ec2/

As you scale your data systems, you face a challenge: how to test thoroughly without putting customer data at risk. Using production data for testing can expose sensitive customer information to unauthorized access or breaches. For customers in regulated industries like finance and healthcare, this risk isn’t only a concern. It’s unacceptable. A data breach during testing could compromise their privacy, damage their trust, and expose organizations to significant compliance penalties. Synthetic test data solves this problem by generating artificial datasets that replicate the structure and patterns of real data without containing any actual customer information. This approach means you can test performance, validate data pipelines, and develop new features while ensuring that customer data remains protected and compliance requirements are met.

As data volumes grow from terabytes to petabytes, the architecture for generating synthetic data must evolve to meet increasing demands for scale, performance, and data quality. In this post, we show how you can build a scalable synthetic data generation solution using Amazon EMR, Apache Spark, and the Faker library.

The challenge of synthetic data generation

Traditional benchmark datasets like TPC-DS provide standardized schemas and predetermined data volumes for consistent testing environments across different systems. However, they fall short in meeting real-world testing requirements. These benchmarks don’t capture industry-specific patterns or the complex relationships found in actual production data. Their rigid schemas and simplified distributions fail to reflect business requirements, and scaling them while maintaining data consistency proves difficult. Perhaps most critically, generating massive datasets with traditional approaches requires specialized architectures to avoid proportional increases in compute costs and time.

Requirements for production-grade synthetic data

Effective workload validation demands synthetic data that mirrors production distributions while maintaining referential integrity across related tables and entities. The generation process must scale horizontally to accommodate growing data volumes while delivering deterministic results. Given identical input parameters, the system should produce the same dataset across multiple runs, supporting consistent testing cycles and comparative analysis.

Beyond technical requirements, synthetic data addresses compliance needs by minimizing exposure of personally identifiable information (PII) and protected health information (PHI) in non-production environments. This approach satisfies GDPR, HIPAA, and CCPA requirements while supporting secure cross-border data transfer, regular stress testing without compromising sensitive information, and providing an audit-friendly alternative to data masking that preserves analytical properties.

Solution overview

Architecting a synthetic data generation system that scales from terabytes to petabytes requires balancing several competing demands: the system must scale horizontally while maintaining data quality, generate large volumes efficiently, manage compute and storage resources cost-effectively, and support various schemas and output formats.

Our architecture addresses these challenges through four core components. Apache Spark on Amazon EMR provides the distributed computing framework necessary for large-scale generation. The Faker library offers synthetic data generation functions that integrate with Spark. Amazon Simple Storage Service (Amazon S3) with Apache Iceberg serves as the storage layer. We chose Iceberg for its schema and partition evolution capabilities without data rewrites, atomic transactions for consistency, precise time travel features for reproducible testing, and optimized performance at extreme scale. Amazon EMR handles dynamic resource allocation and cluster management.

The following diagram illustrates the solution architecture.

Solution architecture showing Amazon EMR generating synthetic data with Apache Spark and Faker, storing results in Amazon S3 with Apache Iceberg

Synthetic data generation at scale with Amazon EMR

Amazon EMR emerges as a particularly powerful solution for this use case, offering several advantages that directly address our requirements. It facilitates scaling of compute resources through instance fleets and Spot Instances, which can reduce costs by up to 90% compared to On-Demand pricing. The service provides built-in performance optimization for Spark applications with real-time monitoring through Amazon CloudWatch integration.

The managed infrastructure reduces operational overhead by handling the underlying Spark ecosystem and cluster lifecycle, while still providing control over scaling policies, instance types, and configurations. Integration with Amazon S3, AWS Glue, and Amazon Athena facilitates end-to-end data generation and testing workflows. Support for multiple programming languages and notebooks provides flexibility in implementing generation logic tailored to specific testing scenarios.

The synthetic data generation process follows a systematic approach designed for efficiency and scalability, as illustrated in the following diagram.

Synthetic data generation workflow showing the systematic process from configuration through data generation to storage

Although synthetic data generation isn’t a sensitive workload, it’s important to maintain robust security throughout the data generation process. Amazon EMR provides security features that align with organizational compliance requirements.

For comprehensive security guidance specific to Amazon EMR deployments, refer to Security in Amazon EMR. The solution follows the AWS Shared Responsibility Model, where AWS manages the security of the cloud infrastructure, and customers maintain responsibility for data security, access management, and compliance controls in the cloud. Specifically for synthetic data generation workloads, AWS manages the security of the underlying Amazon EMR infrastructure, network, and service operations, and customers implement appropriate security controls for their data generation pipelines. Consider the following key areas:

  • Data protection – Enable encryption at rest and in transit using Amazon EMR security configurations, including Amazon S3 encryption and TLS certificates for inter-node communication.
  • Network security – Deploy Amazon EMR clusters in private subnets with security groups following least privilege, and enable the Amazon EMR block public access feature.
  • Access control – Implement AWS Identity and Access Management (IAM) roles with least privilege for Amazon EMR service roles, Amazon Elastic Compute Cloud (Amazon EC2) instance profiles, and runtime roles to isolate job access. Fine-grained table-level and column-level permissions can be controlled using AWS Lake Formation. Additional authentication options are available using Kerberos and LDAP.

Optimize Faker for petabyte-scale data generation

When generating synthetic data at petabyte scale, using Faker’s implementations can quickly lead to performance bottlenecks. To overcome these limitations, adopt a combination of different optimization approaches instead of the default setup. Some of the approaches we adopted in this scenario are discussed in this section.

Faker instance pooling

The following code creates multiple Faker instances to avoid contention when generating data in parallel:

NUM_FAKER_INSTANCES = 10
faker_pool = [Faker() for _ in range(NUM_FAKER_INSTANCES)]

Consistent seed management

The following code provides reproducible data generation across distributed executors:

for faker in faker_pool:
    faker.seed_instance(42)  # For reproducibility
    random.seed(42)

Random access to Faker pool

The following code distributes load across multiple Faker instances to reduce contention:

faker = faker_pool[random.randint(0, NUM_FAKER_INSTANCES-1)]

Broadcast variables for reference data

The following code efficiently distributes reference data to all executors:

tenant_ids_broadcast = spark.sparkContext.broadcast(tenant_ids)
protocols_bc = spark.sparkContext.broadcast(protocols)

Batch generation of synthetic data

The following code generates fake data in batches rather than one-by-one:

return spark.range(1, num_endpoints + 1)
    .withColumn("hostname", random_hostname_udf())

ThreadPoolExecutor for parallel processing

The following code uses Python’s threading for parallel operations within executors:

def parallel_write_with_sync(dataframe_configs, max_workers=3):
    with ThreadPoolExecutor(max_workers=max_workers) as executor:
        # Parallel processing

Optimize Amazon EMR and Spark

When processing massive datasets with Spark on Amazon EMR, carefully tuning configurations can substantially enhance performance beyond the standard settings. In this section, we discuss ways to optimize the execution environment, so you can efficiently handle petabyte-scale workloads with synthetic data generation. By strategically using Spark’s advanced features and configuring Amazon EMR for your specific use case, you can improve throughput, reduce processing time, and maximize resource utilization.

Arrow configuration

The following code enables Apache Arrow for efficient data transfer between Python and JVM. The default value is false.

.config("spark.sql.execution.arrow.pyspark.enabled", "true")

Enable this configuration when your PySpark application frequently converts data between Python and JVM, especially for large DataFrames or when using Pandas operations. Keep this setting disabled for pure Spark SQL workloads or when memory is constrained.

This optimization is most effective in the following scenarios:

  • When processing large-scale datasets that require frequent conversion between Python and JVM.
  • In a PySpark application where large DataFrame operations and Pandas integration are needed.
  • With data science workloads that combine Python UDFs with Spark SQL operations.

Consider the following trade-offs:

  • Arrow maintains in-memory columnar format, resulting in increased memory consumption.
  • Not all data types are fully supported in older versions of Spark.
  • It might introduce overhead for very small datasets where conversion costs outweigh the benefits.

Adaptive query execution

The following code allows Spark to dynamically optimize query execution plans. The default value is true in Spark 3.2 and later, and false in earlier versions.

.config("spark.sql.adaptive.enabled", "true")

This optimization is generally recommended to keep enabled for most workloads. Consider disabling only when you have highly optimized, predictable queries where the adaptive overhead isn’t beneficial, or when troubleshooting query performance issues.

This optimization is most effective in the following scenarios:

  • Complex join operations with unknown or skewed data distributions.
  • Multi-stage queries where initial plans might be suboptimal.
  • When processing data with changing characteristics over time.

Consider the following trade-offs:

  • You may experience additional overhead during the query planning phase.
  • You might occasionally choose suboptimal plans for certain edge cases.

Parallelism configuration

The following code sets appropriate parallelism for distributed data processing based on the volume of data you’re generating. The default value for spark.default.parallelism is the total number of cores on all executor nodes or 2, whichever larger. The default value for spark.sql.shuffle.partitions is 200.

.config("spark.default.parallelism", 1000)
.config("spark.sql.shuffle.partitions", 1000)

Adjust this configuration when the default of 200 shuffle partitions creates too many small tasks (increase data volume) or too few large tasks (decrease for smaller datasets). Generally, aim for partition sizes of 100–200 MB. Modify default.parallelism when your RDD operations need different parallelism than the CPU-based default.

This optimization is most effective in the following scenarios:

  • When generating consistent volumes of synthetic data across multiple runs.
  • When you have predictable resource requirements.
  • When you need to precisely control executor utilization.

Consider the following trade-offs:

  • Static configuration might not adapt well to varying data volumes.
  • Too many partitions can lead to task scheduling overhead.
  • Too few partitions might cause memory pressure on executors.

Memory management

The following code optimizes memory allocation for execution and storage. The default value for spark.memory.fraction is 0.6, and for spark.memory.storageFraction is 0.5.

.config("spark.memory.fraction", 0.8)
.config("spark.memory.storageFraction", 0.3)

Increase memory.fraction from 0.6 to 0.8 when your workload is memory-intensive and you’re not using the JVM heap for other purposes. Adjust storageFraction based on your caching vs. execution memory needs. Decrease to 0.3 if you do minimal caching but have complex computations, and increase to 0.7 or higher for cache-heavy workloads.

This optimization is most effective in the following scenarios:

  • Workloads that are memory-intensive and need fine-grained control.
  • Workloads that balance between execution memory and cached data.
  • During synthetic data generation that has many interdependent fields.

Consider the following trade-offs:

  • Incorrect memory configuration can lead to frequent spills to disk or out-of-memory (OOM) errors.
  • You might need to change the configuration to suit different workload characteristics.
  • The settings must be monitored and tuned for optimal performance.

Limited Python UDF usage

The following code uses Spark’s built-in functions where possible instead of Python user-defined functions (UDFs). No additional configuration is needed. This is a coding practice.

.withColumn("risk_score", F.round(F.rand() * 9 + 1, 2).cast(DecimalType(3, 2)))

We recommend using Spark functions over Python UDFs when the same functionality can be achieved. Use Python UDFs only when complex business logic can’t be expressed using Spark’s built-in functions, or when integrating with specialized Python libraries.

This optimization is most effective in the following scenarios:

  • Simple transformations that can be performed using Spark functions.
  • High-throughput workloads where serialization overhead needs to be minimized.

Consider the following trade-offs:

  • This approach is less flexible compared to customer Python-based transformations or functions.
  • You might need to use complex expressions to accomplish certain data patterns.
  • There is a potential learning curve to familiarize yourself with Spark functions.

DataFrame caching

The following code caches frequently used DataFrames to avoid regenerating data. The default behavior doesn’t use caching. DataFrames are recomputed on each action.

endpoints_df = generate_endpoints().cache()

Use this optimization to cache DataFrames that are accessed multiple times in your application. Monitor memory usage and use MEMORY_AND_DISK storage level for large DataFrames. Uncache DataFrames when they’re no longer needed to free memory.

This optimization is most effective in the following scenarios:

  • When reusing reference data across multiple operations (can result in performance gains).
  • For workloads where the same data is processed on multiple occasions.

Consider the following trade-offs:

  • Too much caching might lead to memory process.
  • Planning is required to manage cache in environments where memory is scarce.

Optimal partitioning

By default, Spark determines partitioning based on input data and previous operations. The following code makes sure data is properly distributed across executors:

.repartition(20)

Use repartition() when you need to increase partitions for better parallelism or support even data distribution. Use coalesce() when reducing partitions to avoid small files. Generally, target 100–200 MB per partition for optimal performance.

This optimization is most effective in the following scenarios:

  • When controlling data distribution and avoiding data skew is very important.
  • Before executing an expensive operation that will benefit from balanced data distribution.
  • When optimizing downstream consumption use cases.

Consider the following trade-offs:

  • This option is more expensive than coalesce(). For large datasets, repartition() can lead to large shuffle.
  • The approach requires trial and experimentation to determine the optimal partition count.
  • There is no “one-size-fits-all” setting. Different applications or operations might gain performance with different partitioning.

Partition-aware writing

By default, data is written without partitioning. The following code organizes data for efficient storage and retrieval:

{"df": network_events_df, "name": "network_events", "partition_cols": ["tenant_id"]}

Partition data when you have predictable query patterns that filter on specific columns. Choose partition columns that are frequently used in WHERE clauses and have reasonable cardinality (avoid too many small partitions or too few large ones).

This optimization offers the following benefits:

  • Allows for highly parallel write operation across multiple executors.
  • Organizes the data that is close to real-world production data.
  • Allows for partition pruning when querying the data.

Consider the following trade-offs:

  • Excess partitioning or too fine-grained partitioning might result in small files.
  • It might result in data skew because of hot partitions.
  • You might encounter storage and metadata overhead because of excessive partitions.

Best practices

Through our journey from terabytes to petabytes, we’ve identified several best practices:

  • Begin with a modest dataset and incrementally scale, allowing for identification of bottlenecks at each stage.
  • Implement robust data validation checks to confirm synthetic data maintains expected properties at scale.
  • Regularly review and adjust Amazon EMR configurations, using Spot Instances and right-sizing clusters.
  • Develop parameterized job scripts that can adjust data volume, complexity, and cluster resources dynamically.
  • Design your synthetic data schema and generation logic to quickly accommodate new fields or changing distributions over time.

Conclusion

Our journey from terabytes to petabytes of synthetic data generation demonstrates how Amazon EMR, combined with Spark and Faker, can effectively address large-scale testing needs. The architecture we explored in this post scales to meet demanding data generation requirements while maintaining data quality and cost-efficiency.

We showed how starting with a solid foundation at terabyte scale, then gradually expanding through Amazon EMR managed services and Spot Instances, helps organizations build robust synthetic data pipelines. The combination of efficient data generation techniques, proper validation, and continuous monitoring provides reliable results at scale.

To begin implementing your own synthetic data generation system, start small, test thoroughly, and scale incrementally. For implementation guidance, refer to Generate production-grade synthetic data at petabyte-scale using Apache Spark and Faker on Amazon EMR.

About the authors

Anubhav Awasthi

Anubhav Awasthi

Anubhav is a Senior Big Data Specialist Solutions Architect at Amazon Web Services (AWS). He collaborates with customers to provide expert architectural guidance for implementing and optimizing analytics solutions using Amazon EMR, Amazon Athena, AWS Glue, and AWS Lake Formation.

Gagan Brahmi

Gagan Brahmi

Gagan is a Specialist Senior Solutions Architect at Amazon Web Services (AWS), focused on Data Analytics and AI/ML. With over 20 years in information technology, he partners with customers to solve complex AI/ML challenges by leveraging data and AI/ML platforms. Gagan helps customers architect scalable, high-performance solutions that utilize distributed data processing, real-time streaming technologies, and AI/ML services to drive business transformation through artificial intelligence and data-driven insights. When not designing cloud-native data and AI solutions, Gagan enjoys exploring new places with his family.

Jayaprakash Boreddy

Jayaprakash Boreddy

Jayaprakash is a Senior Solutions Architect at AWS. He works with ISV customers in designing and building highly scalable, flexible and resilient applications on AWS Cloud.

Sahil Thapar

Sahil Thapar

Sahil is a Principal Solutions Architect. He works with ISV customers to help them build highly available, scalable, and resilient applications on the AWS Cloud.

How ALS GeoAnalytics LITHOLENS ™ revolutionizes core logging through machine learning with Amazon EKS

Post Syndicated from Saransh Burman original https://aws.amazon.com/blogs/architecture/how-als-geoanalytics-litholens-revolutionizes-core-logging-through-machine-learning-with-amazon-eks/

In the mining industry, accurate geological analysis is required for improving mine design and development. Traditionally, this involved labor-intensive and time-consuming on-site inspections of drill core samples, often conducted in remote and challenging environments. ALS GeoAnalytics has streamlined this process through its LITHOLENS ™ platform, a machine learning (ML)-powered system that uses deep learning and machine vision to automate core logging. LITHOLENS ™ significantly enhances data consistency, operational efficiency, and scalability while significantly reducing logging-related costs and lowering greenhouse gas emissions to support sustainable mineral extraction.

This post explores how ALS GeoAnalytics successfully deployed LITHOLENS ™ with Amazon Elastic Kubernetes Service (Amazon EKS) to scale model training and inference while minimizing cost.

The challenge

Development of a new mine involves the creation of a 3D map of the ore body, known as a geological or resource model. This model drives all future design decisions and creating it requires drilling thousands of holes throughout the ore body to examine the structure and composition of the samples extracted. This process is subject to numerous challenges that affect both active and historical drilling campaigns. Challenges such as:

  • Remote site access requiring geologists to travel long distances to visually inspect physical core boxes
  • Subjective interpretations led to inconsistencies, with different experts often producing varying geological logs
  • Underutilized historical imagery from past campaigns lacked standardized tools for meaningful analysis
  • Lost or degraded physical samples made it difficult to revisit legacy data or validate past interpretations
  • Limited transparency in logging and decision-making processes hindered collaboration and accountability
  • Scheduling bottlenecks arose from reliance on a small pool of qualified experts
  • Non-standardized data collection methods prevented effective scaling and cross-project comparison

These limitations not only delayed project timelines but also restricted the ability to generate reliable, high-resolution geological insights—ultimately impeding the speed and effectiveness of exploration strategies.

Machine learning at geological scale

ALS GeoAnalytics developed a comprehensive suite of machine learning and computer vision models to automate geological and geotechnical logging, transforming raw core imagery and data into actionable insights.

A machine learning pipeline formed the foundation for high-resolution visual analysis. It begins with the Color Extraction module, which scans each image to identify unique pixel colors and store the results in Amazon Simple Storage Service (Amazon S3). This data is fed into the Color Clustering module, where users can specify clustering parameters and choose from algorithms such as K-Means, which assigns pixels to clusters based on proximity to centroids, or the Gaussian Mixture Model (GMM), which uses probabilistic distributions to capture more complex variance structures within the color data. These methods effectively reduced image complexity and helped highlight mineralogical variation.

To quantify color composition along the core, the Percentage Report module was introduced. It segmented each image into user-defined sections (for example, 20 cm intervals) and calculated the proportional distribution of each color cluster, enabling spatial analysis of lithological patterns.

On the deep learning front, the team developed and deployed an advanced suite of models tailored for geological and geotechnical analysis. A highlight of this work was the development of RoQE Net, a state-of-the-art neural network designed for geotechnical parameter extraction. RoQE Net demonstrated exceptional accuracy in computing Rock Quality Designation (RQD) and extracting alpha angles, key metrics for assessing core integrity and rock mass quality. In parallel, VeinNet and CobbleNet were engineered to identify and map complex geological features such as veins, cobbles, and lithological structures with high precision. These models were benchmarked against industry standards and consistently outperformed traditional methods in terms of accuracy, reliability, and scalability.Together, these machine learning and deep learning components form the backbone of the LITHOLENS ™ platform—delivering automated, scalable, and highly accurate geological intelligence that accelerates decision-making and enhances the efficiency of exploration and resource modeling workflows.

Solution architecture

ALS GeoAnalytics built LITHOLENS ™ on AWS using a hybrid architecture that combines containerized workloads with serverless components. The system uses Amazon EKS for compute-intensive machine learning tasks, AWS Lambda for API operations, Amazon S3 for data storage, and Amazon Relational Database Service (Amazon RDS) for structured data management.

Figure 1: Architecture Diagram

LITHOLENS ™ uses a unified API model to drive next-generation rock and mineral data analysis. This unified API created is a unified application programming interface that combines multiple services, data streams, and analytic capabilities into a single, powerful access point. Unlike traditional APIs—which might deliver basic, one-dimensional data—you can use the unified API to connect, analyze, and automate complex workflows across departments, vendors, and a wide variety of data sources all at once. With the unified REST API, users can submit geological analysis jobs, monitor progress, and retrieve results through a single interface. This API combines multiple services and data streams into one access point, so users can automate complex workflows across departments and data sources.

Architecture flow:

  1. Request Intake – Jobs are submitted through Amazon API Gateway with a payload specifying job parameters and EKS configuration.
  2. Job Orchestration – The API backend, running on AWS Lambda, provisions EKS containers with the appropriate configuration. User data scripts bootstrap each instance with required setup and execution commands.
  3. Execution and Data Flow
    • Input data is retrieved from Amazon S3.
    • Computation is performed on EKS pods using G6 instances.
    • Logs and intermediate results are continuously tracked.
    • Results are stored in S3 or persisted into RDS through dedicated API calls.
  4. Resource Management – Upon job completion, EKS containers instances automatically shut down, reducing costs.

Architecting for scale and efficiency

To handle variable workloads efficiently, ALS GeoAnalytics implemented a hybrid architecture that’s designed for both performance and cost. The system uses Amazon EKS for compute-intensive ML tasks while using AWS Lambda for lightweight API operations and job orchestration.

Key architectural decisions:

  • Amazon EKS for ML Workloads – Deep learning model training and inference require sustained compute power with GPU acceleration. EKS provides the container orchestration needed to manage these workloads across G6 instances, with automatic scaling based on job queue depth.
  • Lambda for API Gateway – Job submission, status checking, and result retrieval are handled through serverless functions. This removes the overhead of maintaining always-on API servers for sporadic client requests, reducing costs during low-usage periods.
  • Pre-configured AMIs – Custom Amazon Machine Images contain all required dependencies and model artifacts, reducing container startup time from several minutes to under 30 seconds. This approach improves job throughput and reduces compute costs by minimizing idle time.
  • Automated Resource Management -–EKS clusters scale down to zero when no jobs are queued, so compute resources are only consumed during active processing. Combined with S3 for data persistence and RDS for metadata, this creates a cost-effective architecture that scales with actual usage.

This design addresses the geological industry’s unpredictable workload patterns while maintaining the performance needed for complex computer vision tasks.

Business impact and results

LITHOLENS ™ has seen success with 10 different mining companies on over 40 active projects, with substantially accelerated project completion and a standard analysis process used across all projects. This new approach has made mineral detection and classification more accurate while reducing the need for experts to visit sites. Teams can now trace how analysis decisions are made, grade minerals more consistently, and plan projects and assign resources more effectively. Real-time monitoring and reporting give managers up-to-the-minute information on how projects are progressing.

Conclusion

The massive scalability of Amazon EKS has allowed ALS GeoAnalytics to fundamentally transform how core logging and analysis is conducted. AWS suite of services enables LITHOLENS ™ to efficiently implement computer vision and machine learning, bringing new operational capabilities to our customers and opening business opportunities throughout the mining industry. The success of LITHOLENS ™ demonstrates how cloud computing and AI can help modernize a long-standing industry like mining, creating value through improved operational efficiency, accuracy, and scalability. ALS GeoAnalytics continues to evolve its platform on AWS, using cloud computing to push the boundaries of what’s possible, and looking to grow LITHOLENS ™ in to promising applications in oil and gas, civil engineering, and even space exploration.

About the authors

How Synthesia optimizes generative AI video inference on Amazon EC2 G7e instances

Post Syndicated from Moises Hernandez original https://aws.amazon.com/blogs/architecture/how-synthesia-optimizes-generative-ai-video-inference-on-amazon-ec2-g7e-instances/

Synthesia, an enterprise-focused AI video platform, has transformed content creation, helping everyone to create video content without cameras or microphones. To achieve this, Synthesia allows its users to create video avatars that synthesize the likeness and voice of real people. Synthesia achieves this through a series of in-house developed models based on various architectures, including latent diffusion video generation models.
Customers like Synthesia often choose to host their models on Amazon Elastic Compute Cloud (Amazon EC2) instances because of the flexibility and control over the underlying hardware that the service provides, and among them they find the G7e instance family to be a cost-efficient option to serve GPU-memory intensive generative AI video models, giving customers access to NVIDIA RTX PRO 6000 Blackwell GPUs, with 96GB of GPU memory.

When generating videos using AI models with a Variational Auto Encoder (VAE) Decoder in the architecture, customers often find the GPU utilization is bottlenecked by the saving rate of the video frames to a file held on storage. This causes GPU stalls and reduces average GPU kernel utilization, which is the percentage of time the GPU is actively executing compute kernels rather than idle.

This post introduces a video decoding optimization technique that we have ideated in collaboration with Synthesia Research Engineering team, which we call Asynchronous Frame Generation Pipeline. Adopting this technique allows you to overlap GPU compute, device-to-host (D2H) data transfer, and host-side post-processing. In this post, we apply this technique to the VAE decoder of a Wan video generation model as an example, where our benchmarks on G7e show increased GPU kernel utilization from 82% to 99.9%, in turn leading to an 8.2% decrease in latency (and increase in throughput) for video decoding. We expect this technique to benefit any customer with a chunked video generation pipeline that transfers frames to host memory.

You can find an end-to-end example implementation of the Asynchronous Generation Pipeline applied to the Hugging Face Diffusers format of the Wan 2.2 14B Model in the associated GitHub repository.

In the following section, you will see how latent diffusion models generate video and why the traditional sequential decoding approach can lead to inefficient GPU utilization.

Understanding the sequential decoding bottleneck

Latent diffusion video generation models have emerged as powerful tools for generating temporally coherent video sequences. To reduce compute and memory requirements, these models perform the diffusion process in a compressed latent space of a variational auto-encoder (VAE), as depicted in the below figure, which has generally a much lower dimensionality than the originating video pixel space.

Fig. 1 High-level architecture of a VAE model. The depicted frames are taken from the Wan 2.2 repository sample video used in the example implementation in the next sections.

At inference time, a latent representation is initialized with noise and iteratively denoised via a Diffusion Process. To make sure that the output follows the text prompt, every denoising step is conditioned on the text input if present. After the final denoising step, the Diffusion Process yields a generated video that is still represented in the latent space of the VAE. The last step therefore consists in decoding the latent video back into a human-readable pixel video using the decoder part of the VAE.

Processing the whole latent or pixel video at once is usually too resource-intensive, even on large GPUs. It is therefore common to split the video along the temporal dimension and to decode video frames one latent frame at a time, resulting in a chunk of, for example, 4 consecutive pixel frames, as shown in Fig 2.

Fig.2 Decoding one latent results in a chunk of 4 time-consecutive pixel frames which are transferred from the GPU to the host.

Once a chunk has been decoded and processed on the GPU, the corresponding pixel frames must be transferred back to host (CPU) memory with a D2H transfer, so they can be written to a file or further processed. If you wait to transfer the video frames once the entire video has been decoded, the full decoded video must fit in GPU memory before being transferred, making this strategy hard to scale for arbitrarily long videos. A more memory-efficient alternative is to transfer the decoded frames every time a chunk is decoded, so that the GPU memory footprint of the decoder scales with the size of the chunk instead of the full video.

Traditionally, a newly generated set of frames in a chunk N is passed from GPU memory to CPU RAM synchronously, and it is committed to storage before the CPU can launch the CUDA kernels (referred to as kernels from now on) that process the N+1 chunk. This leads to systematic GPU stalls between chunks, as the copy from device to host prevents the GPU from immediately starting work on the next chunk, reducing overall hardware utilization, and in turn increasing processing time. We will refer to this synchronous process as Sequential Frame Generation Pipeline, which is depicted in Fig. 3.


Fig.3 Schematic representation of the Sequential Frame Generation Pipeline. Launching the kernels processing Chunk N+1 needs to wait for the full copy and storage of the frames in Chunk N.

In the following section, you will learn how to overcome this bottleneck using an asynchronous pipeline that overlaps GPU computation with data transfers and host-side processing.

Asynchronous Frame Generation Pipeline

To minimize GPU stalling, and in turn increase GPU utilization, you need to modify the Sequential Frame Generation Pipeline introduced in the previous section, so that all host-side CPU work (such as appending decoded frames to a file) runs in parallel with a stream of uninterrupted device-side kernels. This section shows you how to achieve this using as an example the implementation of an Asynchronous Wan VAE Decoder, which is implemented with PyTorch and you can find in the associated GitHub repository.

By default, PyTorch schedules work on a single (default) CUDA stream per device which executes the operations in issue order unless additional streams are explicitly created. To decouple compute and D2H copies, the implementation uses two CUDA streams so that the GPU is allowed to overlap compute kernels, which are enqueued on the default stream, and D2H copies on a dedicated copy stream. Compute kernels are enqueued on the default stream, which from now on can be referred to as Compute Stream, and D2H copies on a dedicated copy stream, namely the Copy Stream. In Fig. 4 you can find the full setup on a GPU accelerated instance (G7e in our example) depicted.


Fig. 4 High level diagram of the key components in our implementation of the Asynchronous Frame Generation Pipeline.

On the host side, kernel submissions are interleaved with D2H transfers and post-transfer processing and, to avoid host-side blocking calls and maximize GPU utilization the Asynchronous Frame Generation Pipeline introduces two mechanics:

  • A dedicated Worker CPU thread responsible for reading chunks from Host Memory (RAM), and writing them to file, leaving the main Python thread to focus on launching kernels and scheduling D2H transfers.
  • Two in-memory Buffers on the GPU Memory (VRAM), and on the Host Memory (RAM), and page-lock the required Host Memory buffers to make sure D2H copies are performed fully asynchronously.

Using a double-buffer strategy makes sure that for adjacent chunks the compute, D2H transfer, and host processing can overlap safely as they operate on distinct memory buffers.


Fig. 5 Schematic representation of the interplay between Events, Streams, Buffers, and Worker component in the Asynchronous Frame Generation Pipeline implementation. Events are depicted as left and right conditions, expressing what the process WAITs for, and ISSUEs when it has completed. The size of the boxes representing operations has been scaled to fit the text, and therefore not representative of the actual processing time.

Since buffers can be accessed concurrently by different components, the implementation introduces a synchronization mechanism to avoid different kernels and threads corrupting data held in the buffer. This can be achieved using CUDA Events which are used as a barrier that clears if it can answer closed questions such as: Has decoding of chunk N completed? You can find the interplay between Streams, Threads and Events depicted in Fig. 5.

G7e benchmark results

To quantify the impact of the Asynchronous Frame Generation pipeline, you can benchmark the decoding of a single 41 latent-frames test video for both the Synchronous and Asynchronous pipeline on a g7e.2xlarge EC2 instance using this sample notebook. To have a common baseline, the benchmarks are based on the unoptimized Hugging Face Diffusers Wan 2.2 14B model, and, in both cases, the benchmarking loop performs an initial full decoding cycle to let CUDA and PyTorch initialize and allocate memory pools and cache. After warmup, the benchmarking session carries out 10 consecutive full video decoding cycles. We have run an end-to-end run as an example and present the results in Table 1.

Metric Synchronous (time s /video) Asynchronous (time s / video)
min 21.98 20.16
mean 21.99 20.17
P99 22.01 20.20

Table. 1 Benchmark results for 10 consecutive decoding runs for the Synchronous and Asynchronous pipelines.

The example results show a speed gain of up to 8.2% on this benchmark, decreasing the Real Time Factor of the decoder from 3.21 to 2.95. Considering the g7e.2xlarge pricing as of writing this post ($3.36 per GPU on-demand in the Ohio region), this leads to an average saving of $896 for decoding 1,000 hours of video on a single GPU. This is a theoretical saving, calculated assuming that the underlying model operates at full computational efficiency without bottlenecks.

To have a qualitative understanding of how this performance boost is achieved, you can profile both the Synchronous and Asynchronous pipelines using this sample notebook. The results for an example run are reported in Fig. 6 and Fig. 7, showing the absence of GPU stalls in the Asynchronous pipeline as opposed to the Synchronous pipeline.


Fig. 6 Profile of the Synchronous pipeline single thread and stream. As the main CPU thread is writing Chunk N frames to disk, the GPU stream stalls, waiting for the CPU to launch the kernels needed to process Chunk N+1


Fig.7 In the Asynchronous pipeline, the Compute Stream is not interrupted by the copying and writing of the frames to disk. The Main Thread is free to launch the processing kernels. This figure is displayed with a different zoom level than Fig. 6 to highlight the qualitative differences.

By focusing on the temporal axis for two consecutive chunks in the steady state, you can quantify the GPU kernel utilization increase, which, in our example run was 82% for the Synchronous case, increasing to 99.9% for the Asynchronous case.

Conclusion

In this post, we demonstrated how the Asynchronous Frame Generation Pipeline can improve GPU utilization when decoding latent videos on Amazon EC2 G7e instances. By decoupling GPU compute from device-to-host transfers and host-side I/O using dual CUDA streams, pinned memory buffers, and a dedicated worker thread, we eliminated the GPU stalls inherent in traditional synchronous decoding pipelines.

The sample benchmarks on the Wan 2.2 14B VAE decoder show an 8.2% reduction in decoding latency, translating to approximately $896 in savings per 1,000 hours of decoded video on a single GPU when considering the g7e.2xlarge pricing as of writing this post ($3.36 per GPU on-demand in the Ohio region). These gains come without any changes to model weights or inference quality, purely from better hardware utilization. We expect the kernel utilization gain to be even more impactful on optimized and compiled models, which make more efficient use of the GPU.

The techniques presented here are not specific to the Wan architecture, nor to the specific GPU utilized. Any chunked video generation pipeline that transfers frames to host memory can benefit from this approach. We encourage you to explore the associated sample implementation to experiment with how the Asynchronous Frame Generation Pipeline could be integrated into your own video generation workloads.

To get started with G7e instances for your generative AI video workloads, visit the Amazon EC2 G7e instance page.

About the authors

Sharing Capacity Blocks for ML Across Your AWS Organization

Post Syndicated from Tyler Klimas original https://aws.amazon.com/blogs/compute/sharing-capacity-blocks-for-ml-across-your-aws-organization/

When your data science team reserves GPU instances for a two-week training job but completes it in four days, that capacity has the potential to sit unused while your computer vision team waits another week to start their project. Now you can eliminate this GPU waste and scheduling conflict by sharing Capacity Blocks for ML across your AWS Organization. This scheduling mismatch between teams creates bottlenecks that delay product launches, increase infrastructure costs, and slow your ability to deliver machine learning (ML) powered features to customers. With cross-account sharing for Amazon Elastic Compute Cloud (Amazon EC2) Capacity Blocks for ML, you can now distribute reserved graphics processing unit (GPU) capacity across teams based on actual demand rather than rigid scheduling predictions. This means your computer vision team can use the capacity as soon as the data science team is done.

In this post, we’ll show you how to configure cross-account sharing for Capacity Blocks for ML, set up monitoring for your shared resources, and optimize instance utilization through alerting. By increasing the utilization rates and reducing over-provisioning, you improve your resource efficiency and cost optimization for your organization.

You can reduce idle resources in your ML team’s account by sharing capacity with other teams waiting for GPUs. Additionally, you can maintain Capacity Blocks for ML centrally. This lets you control which teams have access to the capacity and helps you reduce waste and bottlenecks in your organization. Before starting into the tutorial, let’s review how Capacity Blocks for ML and AWS RAM work together.

Overview

Capacity Blocks for ML let you reserve GPU-based accelerated compute instances ahead of time for short duration ML workloads. When you launch instances in Capacity Blocks for ML, Amazon EC2 automatically places the instances in Amazon EC2 UltraClusters, giving you low-latency, petabit scale networking. UltraClusters provide the high performance networking your training workloads require.

You see exactly when GPU capacity is available and schedule your Capacity Blocks for ML to start when it makes sense for your project. You pay upfront for the entire reservation period. This makes Capacity Blocks for ML useful when you need GPUs for days to months. It provides predictable capacity without long-term commitments.

When you purchase Capacity Blocks for ML, you can share it with other accounts in your AWS Organization using AWS Resource Access Manager (AWS RAM). With AWS RAM, you can share AWS resources across accounts within your organization. When you share with other accounts, those accounts become consumer accounts that can launch instances using your capacity. As the owner account, you pay the upfront reservation cost and retain ownership. If you’re launching instances from a consumer account, you are responsible for additional costs such as operating system licensing charges. Capacity Blocks can be shared to multiple accounts simultaneously, with the entire Capacity Block reservation being shared on a first come, first served basis.

Overview of AWS Organizations showing an owner account sharing to two consumer accounts using an AWS RAM resource share.

Figure 1: Capacity Block sharing using Resource Access Manager.

With the share feature, you benefit from flexible GPU capacity management when your priorities shift, or teams finish work at different times. Now, when your data science team completes experimentation early, your other teams can use that capacity for production training. If priorities shift mid-quarter, you can move capacity where it’s needed most.

In this tutorial, you’ll share a Capacity Block for ML across accounts and then create an alarm to monitor utilization when it drops below a threshold. Before you start, complete the following prerequisites.

Prerequisites

To share Capacity Blocks for ML, you must first find and purchase a Capacity Block. Only standard Capacity Blocks for ML can be shared using AWS RAM. UltraServer Capacity Blocks are not eligible for sharing.

You can share Capacity Blocks only within your AWS Organization. Verify the owner of the Capacity Blocks as well as the consumer(s) are within the same organization. For guidance, see Creating and configuring an organization.

Before sharing Capacity Blocks, you must configure resource sharing with AWS Organizations. Only the management account with the following required AWS Identity and Access Management (IAM) permissions can enable resource sharing within an Organization:

ram:EnableSharingWithAwsOrganization

iam:CreateServiceLinkedRole

organizations:EnableAWSServiceAccess

organizations:DescribeOrganization

Using the AWS Management Console of the management account:

  1. Navigate to the AWS RAM console.
  2. In the left navigation pane, choose Settings.
  3. Select Enable sharing with AWS Organizations.

Enable sharing with AWS Organizations in Settings of Resource Access Manager.

Figure 2: Enable sharing with AWS Organizations in AWS RAM.

Using the AWS Command Line Interface (CLI):

  1. Run this command to give AWS RAM trusted access to your organization’s account structure: 
        aws organizations enable-aws-service-access --service-principal ram.amazonaws.com
  2. Turn on resource sharing within your organization so accounts and OUs can access shared resources without manual acceptance: 
        aws ram enable-sharing-with-aws-organization

After you turn on sharing in your organization, you need the following IAM permissions to create resource shares:

ram:CreateResourceShare

ram:AssociateResourceShare

ram:GetResourceShares

Now that you’ve completed the prerequisites, you’ll learn how to share the Capacity Blocks for ML to other accounts of your organization.

Tutorial

You’ll complete this sharing process in four steps:

  1. Create a resource share.
  2. Attach Capacity Block to the resource share.
  3. Verify the share in your consumer account.
  4. Monitor the resource share.

Verify Capacity Reservation (console)

  1. In your Capacity Block owner’s account, navigate to the Amazon EC2 console.
  2. In the left navigation pane, choose Capacity Reservations.
  3. Confirm your Capacity Blocks for ML is in Active or Scheduled state.
  4. If you have a Resource share already configured, choose Actions, Share and select your Resource share.

Figure 3: EC2 Capacity Reservation

Share Capacity Blocks for ML (console)

You now will create a Resource Share and associate the following resources.

  1. Navigate to the AWS RAM console in your Capacity Block owner’s account.
  2. In the left navigation pane, choose Resource shares.
  3. Choose Create resource share.

Create Resource share in AWS RAM Console

Figure 4: Create Resource share in AWS RAM

  1. Enter a name for your resource share.
  2. Under Select resource type, choose Capacity Reservations.
  3. Select your Capacity Block from the list.
  4. Under Principals, specify the accounts, organizational units, or organization to share with.Select principals to share resources in AWS RAM.Figure 5: Select principals to share resources with
  5. Choose Create resource share.

Share Capacity Blocks for ML (AWS CLI)

Replace the placeholder values in the following CLI commands below with your actual values:

  • arn:aws:ec2:us-east-2:123456789012:capacity-reservation/cr-1234abcd56EXAMPLE → Your Capacity Reservation ARN
  • 111122223333 → The AWS account ID of the principal you’re sharing with
  • arn:aws:ram:us-east-2:123456789012:resource-share/7ab63972-b505-7e2a-420d-6f5d3EXAMPLE → Your RAM resource share ARN

 

  1. Create resource share with Capacity Block and principals:
    aws ram create-resource-share \
         --name capacity-block-share \
         --resource-arns arn:aws:ec2:us-east-2:123456789012:capacity-reservation/cr-1234abcd56EXAMPLE \ 
         --principals 111122223333
  1. To add a Capacity Block to existing resource share:
    aws ram associate-resource-share \
         --resource-share-arn arn:aws:ram:us-east-2:123456789012:resource-share/7ab63972-b505-7e2a-420d-6f5d3EXAMPLE \
         --resource-arns arn:aws:ec2:us-east-2:123456789012:capacity-reservation/cr-1234abcd56EXAMPLE

Access and Launch shared Capacity Blocks (console)

After you add the Capacity Block to a resource share, your consumer accounts automatically gain access when you share the Capacity Block within the same AWS Organization.

  1. Navigate to the AWS RAM console in your consumer account.
  2. In the left navigation pane, choose Shared with me, Resource shares. Verify the Resource share is Active.Within your consumer account, verify resource share.Figure 6: In your consumer account, verify the resource share
  3. Navigate to the Amazon EC2 console. In the left navigation pane, choose Capacity Reservations.
  4. Confirm the shared Capacity Block appears and is in Active or Scheduled state. Because sharing is asynchronous, the Capacity Block may take a few moments to appear even after the resource share shows Active.
  5. Navigate to the Amazon EC2 console and choose Launch instance.
  6. Configure your instance as required (AMI, instance type, key pair, etc.).
  7. Under Advanced details, for Purchasing option, choose Capacity Blocks.
  8. For Capacity reservation, choose Specify Capacity Reservation.
  9. For Capacity reservation targeted ID, select or enter your Capacity Block reservation ID.
  10. Launch the instance.

Access shared Capacity Blocks (AWS CLI)

Replace the placeholder values in the following CLI commands below with your actual values:

  • ami-0abcdef1234567890 → Your AMI ID
  • cr-0c54f6734d944345a → Your Capacity Reservation ID
  1. List resource shares in your consumer account:
    aws ram get-resource-shares --resource-owner OTHER-ACCOUNTS
  1. Verify that capacity reservation is available:
    aws ec2 describe-capacity-reservations
  1. Launch EC2 instance from Capacity Block:
    aws ec2 run-instances \
         --image-id ami-0abcdef1234567890 \
         --count 1 \
         --instance-type p5.48xlarge \
         --key-name my-key-pair \
         --subnet-id subnet-0abcdef1234567890 \
         --instance-market-options MarketType='capacity-block' \
         --capacity-reservation-specification CapacityReservationTarget={CapacityReservationId=cr-0c54f6734d944345a}

Monitor usage (console)

You can create Amazon CloudWatch alarms to proactively identify low utilization of your Capacity Block. This helps you to improve the usage of your capacity reservation. This section shows you how to create an Amazon Simple Notification Service (Amazon SNS) email notification when the number of running instances drops below a certain threshold.

In addition to monitoring usage, AWS CloudTrail logs capture API events related to your Capacity Block, including the CapacityReservationId. As the owner, you can see which accounts are consuming instances and when.

Step 1: Create an SNS Topic for Notifications

  1. Open the Amazon SNS console.
  2. In the left navigation pane, choose Topics.
  3. Choose Create topic.
  4. For Type, select Standard.

Create SNS Topic for CloudWatch alarm.

Figure 7: Create SNS Topic

  1. For Name, enter capacity-block-alerts.
  2. Choose Create topic.

Step 2: Create an SNS Subscription:

  1. In the left navigation pane, choose Create subscription.Create SNS SubscriptionFigure 8: Create SNS Subscription
  2. For Protocol, choose Email.
  3. For Endpoint, enter your email address.
  4. Choose Create subscription.

Step 3: Create the CloudWatch Alarm

  1. Navigate to the Amazon CloudWatch console.
  2. In the left navigation pane, choose Alarms, All alarms.
  3. Choose Create alarm.
  4. Choose Select metric.
  5. Choose EC2 Capacity Reservations.
  6. Choose By Capacity Reservation.
  7. Find your Capacity Block ID (e.g., cr-12345678abcdef).
  8. Select the checkbox next to InstanceUtilization.
  9. Choose Select metric.

Step 4: Configure the Metric

  1. Under Metric:
  2. For Statistic: Select Average.
  3. For Period: Select 5 minutes.
  4. Under Conditions choose Threshold type: Select Static.
  5. Whenever InstanceUtilization is…: Select Lower than…: Enter 20 (This metric is measured in percentage).
  6. Choose Next.

Step 5: Configure Actions

  1. Under Notifications:
  2. Alarm state trigger: Select In alarm.
  3. Select an SNS topic: Choose Select an existing SNS topic.
  4. Send a notification to…: Select capacity-block-alerts.Configure CloudWatch AlarmFigure 9: Configure CloudWatch Alarm
  5. Choose Next.

Step 6: Name and Create Alarm

  1. For Alarm name, enter: CapacityBlock-LowUtilization-cr-123456789abcdef.
  2. For Alarm description, enter: Alert when Capacity Block utilization drops below 20%.
  3. Choose Next.
  4. Review your configuration and choose Create alarm.

Monitor usage (AWS CLI)

Replace the placeholder values in the following CLI commands below with your actual values:

  • 123456789012 → Your 12-digit AWS account number
  • cr-0c54f6734d944345a → Your Capacity Reservation ID
  • 7ab63972-b505-7e2a-420d-6f5d3EXAMPLE → Your RAM resource share ID
  • [email protected] → Your email address for notifications

 

  1. Create the SNS topic:
    aws sns create-topic \
        --name capacity-block-alerts
  1. Using the TopicArn from the output, subscribe your email:
    aws sns subscribe \
        --topic-arn arn:aws:sns:us-east-2:123456789012:capacity-block-alerts \
        --protocol email \
        --notification-endpoint [email protected]
  1. Create the full CloudWatch alarm:
    aws cloudwatch put-metric-alarm \
        --alarm-name "CapacityBlock-LowUtilization-cr-1234EXAMPLE" \
        --alarm-description "Alert when Capacity Block utilization drops below 20%" \
        --namespace "AWS/EC2CapacityReservations" \
        --metric-name "InstanceUtilization" \
        --dimensions Name=CapacityReservationId,Value=cr-0c54f6734d944345a \
        --statistic Average \
        --period 300 \
        --evaluation-periods 1 \
        --threshold 20 \
        --comparison-operator LessThanThreshold \
        --alarm-actions arn:aws:sns:us-east-2:123456789012:capacity-block-alerts

Clean up (console)

As the owner of the Capacity Block, you retain the ability to modify the resource share. However, owners cannot modify instances that consumers launch into Capacity Blocks they have shared. This section outlines how to clean up your previous work.

Using the AWS Management Console:

Stop sharing the Capacity Block

  1. Navigate to AWS RAM console.
  2. In the left navigation, choose Shared by me, Resource shares.
  3. Select your resource share.
  4. Choose Modify.
  5. Remove the Capacity Block from the resource share or delete the entire resource share.

Delete the CloudWatch Alarm

  1. Navigate to the Amazon CloudWatch console.
  2. In the left navigation, choose Alarms, All alarms.
  3. Select the alarm you created.
  4. Choose Actions, Delete.
  5. Confirm deletion.

Delete the SNS Topic and Subscription

  1. Navigate to the Amazon SNS console.
  2. In the left navigation, choose Subscriptions.
  3. Select the subscription and choose Delete.
  4. In the left navigation, choose Topics.
  5. Select capacity-block-alerts and choose Delete.
  6. Confirm deletion.

Clean up (AWS CLI)

Replace the placeholder values in the following CLI commands below with your actual values:

  • 123456789012 → Your 12-digit AWS account number
  • 7ab63972-b505-7e2a-420d-6f5d3EXAMPLE → Your RAM resource share ID
  • cr-0c54f6734d944345a → Your Capacity Reservation ID
  • a1b2c3d4-5678-90ab-cdef-EXAMPLE → Your SNS subscription ID
  1. Remove the Capacity Block from the resource share 
        aws ram disassociate-resource-share \
        --resource-share-arn arn:aws:ram:us-east-2:123456789012:resource-share/7ab63972-b505-7e2a-420d-6f5d3EXAMPLE \
        --resource-arns arn:aws:ec2:us-east-2:123456789012:capacity-reservation/cr-0c54f6734d944345a
  2. Delete the resource share 
        aws ram delete-resource-share \
        --resource-share-arn arn:aws:ram:us-east-2:123456789012:resource-share/7ab63972-b505-7e2a-420d-6f5d3EXAMPLE
  3. Delete the CloudWatch Alarm 
        aws cloudwatch delete-alarms \
         --alarm-names "CapacityBlock-LowUtilization-cr-123456789"
  4. Delete the SNS Topic and Subscription
    1. List subscriptions to get the subscription ARN 
          aws sns list-subscriptions-by-topic \
         --topic-arn arn:aws:sns:us-east-2:123456789012:capacity-block-alerts
    2. Delete the subscription 
          aws sns unsubscribe \
         --subscription-arn arn:aws:sns:us-east-2:123456789012:capacity-block-alerts:a1b2c3d4-5678-90ab-cdef-EXAMPLE
    3. Delete the Topic 
          aws sns delete-topic \
         --topic-arn arn:aws:sns:us-east-2:123456789012:capacity-block-alerts

Conclusion

In this post, we showed you how to share Capacity Blocks for ML across your AWS Organization using AWS RAM. We covered configuring the AWS RAM integration with Organizations, creating resource shares, and accessing shared Capacity Blocks for ML from consumer accounts. Finally, we showed you how to monitor and alert on low instance utilization.

By sharing Capacity Blocks across your organization, you can reduce idle GPU capacity, eliminate scheduling bottlenecks between teams, and maximize the return on your reserved compute investment. To take this further, consider building dashboards in Amazon CloudWatch to track utilization trends across multiple Capacity Blocks.

You can get started by purchasing Capacity Blocks for ML and sharing it across your organization today. For more details on other resources you can share with AWS RAM, visit the Shareable AWS resources in the user guide. If you have questions, contact your AWS account team or leave a comment below.

Building Self-Extending CLI Tools with Strands Agent

Post Syndicated from Ragib Ahsan original https://aws.amazon.com/blogs/devops/building-self-extending-cli-tools-with-aws-strands/

I. Introduction

Engineering teams build internal command-line interface (CLI) tools because repetitive operational tasks such as generating reports, auditing infrastructure, and checking service health are faster and more reliable when automated behind a consistent interface. A well-built CLI replaces ad-hoc scripts with structured commands, standardized error handling, and composable workflows that any team member can run. However, building these tools follows a predictable development lifecycle. The developer sets up a package, writes commands, handles errors, and ships it, then spends the next six months as its sole maintainer. Meanwhile, requests for new commands, custom report formats, and one-off integrations pile up as other teams across the organization discover the tool is useful for their workflows too. Frameworks like Click and Typer reduce the friction, but every new command still needs to be written, tested, and deployed manually.

Tools that generate their own capabilities on demand offer a different approach. Instead of writing each command manually, users can describe what is needed in natural language, and the tool writes the code, loads it, and makes it available at runtime without requiring a restart or redeployment. This is called meta-tooling, a repeatable pattern for giving applications the ability to create their own tools dynamically. For teams that maintain growing collections of internal utilities, this eliminates the bottleneck of having a single developer write every new feature.

In this post, we will walk through one implementation of this pattern, a CLI generator called CLI Creator. CLI Creator combines three technologies into a mechanism that organizations can adapt for their own use cases:

  • Amazon Bedrock, a fully managed service for building generative AI applications with foundation models, with Anthropic’s Claude Opus 4.6 for AI-powered code generation.
  • Strands Agents SDK, an open-source Python framework for building AI agents with tool use, for dynamic tool creation, loading, and execution at runtime.
  • Model Context Protocol (MCP), an open standard for connecting AI applications to external data sources and tools, for automatically discovering API servers that give generated tools additional knowledge.

The result is a development workflow where new CLI capabilities go from request to working command in minutes instead of days, without manual coding. By the end of this post, a single natural language prompt will have produced a complete, installable CLI. That CLI can extend itself with new tools, refine them iteratively, and discover relevant MCP servers through an interactive selection workflow.

II. Solution Overview

The Challenge

As an example, consider a platform engineering team that produces weekly operations reports for leadership. Every Monday morning, stakeholders expect a summary of their AWS footprint, including which Amazon DynamoDB tables are running hot, which Amazon Simple Storage Service (Amazon S3) buckets are growing fastest, and who made significant infrastructure changes last week. The AWS CLI can list tables and buckets, but it cannot produce these reports.

Each report is a multi-step workflow that involves calling several APIs, joining the data, computing derived metrics like estimated monthly cost or growth rate, and formatting the output for a specific audience. The team ends up writing Python scripts for each report, and every new report request means another script by a developer.These are each their own small project, often requiring a hundred lines of Python to pull multiple APIs, compute derived metrics, and format output before you even think about error handling. Requirements shift weekly, so each change means modifying source code, testing, and redeploying. The tooling never converges; the team ends up with a folder of disconnected scripts, each with its own argument parsing, error handling, and output formatting. Any team that builds small, purpose-built utilities faces the same friction, and operations reporting is the example we use to illustrate the meta-tooling pattern.

The Solution

Prerequisites

To follow along with this post, you will need:

  • Python 3.12 or later
  • An AWS account with Amazon Bedrock access enabled for Anthropic Claude models in us-west-2
  • AWS credentials configured locally (via `aws configure` or environment variables)
  • Git installed (for tool version tracking)

The source code is available on GitHub. Installation instructions are in the repository README.

Walkthrough

Instead of writing report scripts manually, organizations describe what they need in natural language.Terminal screenshot of a CLI Creator tool generating an AWS operations reporting CLI called "aws-ops-reporter." The tool analyzes requirements, detects API keywords (DynamoDB, S3, CloudTrail), and displays 18 available MCP servers. The user selects servers 1, 9, and 18 (AWS DynamoDB, AWS S3, AWS CloudTrail). A planned CLI structure shows four commands: dynamo-capacity, unused-s3, audit-cloudtrail, and cost-summary. A confirmation prompt reads "Proceed with generation? [Y/n].

The system then does the following:

  1. Claude Opus 4.6 on Amazon Bedrock analyzes the description and extracts a structured list of commands, arguments, and options.
  2. MCP servers are discovered automatically, wherein the system detects keywords like “DynamoDB”, “S3”, and “CloudTrail” in the description, searches the MCP registry for relevant API servers, and presents an interactive selection prompt for choosing which servers to include.
  3. Once the user confirms, the system generates complete Python code for each command. These are not stubs or placeholders that users may typically see within generated code, but working implementations with validated AWS SDK for Python (Boto3) calls, error handling, and type hints.
  4. Finally, the output is packaged as an installable Python project with a pyproject.toml file and entry points configured.

Most importantly, the generated CLI includes a tool command group that enables self-extension at runtime. After installation, users can ask the CLI to create entirely new reporting tools and iteratively refine them without touching source code. This is the repeatable part of the pattern because any generated tool inherits the ability to extend itself. This mechanism is built into every generated CLI, so each one is immediately capable of growing beyond its original scope.

III. Technical Implementation

Strands Agents SDK Integration

The Strands Agents SDK is the backbone of the meta-tooling pattern. It provides three features that make self-extending tools possible, and these features are not specific to CLI generation. Any Python application can use them to dynamically create and manage capabilities at runtime.

The @tool Decorator

When a user asks a generated CLI to create a new tool, Claude Opus 4.6 on Amazon Bedrock produces Python code that uses the Strands @tool decorator. This decorator registers the function with Strands’ tool system, making it immediately discoverable and executable:

from strands import tool

@tool 
def list_s3_buckets_with_costs() -> List[Dict[str, Any]]:

The @tool decorator registers the function’s signature, type hints, and docstring as a tool specification that the Strands Agent can reason about and invoke.

Runtime Tool Loading

The Strands Agents SDK includes a tool loading system that can discover and import @tool-decorated functions from Python files at runtime. Tools do not need to be registered at application startup. They can be created, saved to a directory, and made available to the agent dynamically.In our implementation, generated tools are saved as standalone Python files in a directory called `tools/`. Each time a CLI command runs, the application scans this directory, loads any @tool-decorated functions it finds, and adds them to the agent’s tool collection without requiring a restart.The self-extending pattern works because of this scan-on-invocation approach. A user can create a tool, execute it, decide it needs changes, update it, and execute again without any rebuild or reinstall step since each CLI invocation discovers and loads whatever tools exist on disk.

Agent Orchestration with BedrockModel

The Strands Agent class ties everything together. It connects to Amazon Bedrock via BedrockModel and manages a collection of tools:

from strands import Agent
from strands.models import BedrockModel

agent = Agent(
    model=BedrockModel(
        model_id=""
    ),
    tools=[shell_tool, editor_tool] + loaded_tools,
    system_prompt="You are a tool creation assistant..."
)

When the agent receives a tool creation request, it calls Amazon Bedrock to generate the implementation and saves it as a Python file in the tools/ directory. The next CLI command automatically discovers and loads the new tool.

Amazon Bedrock Integration

CLI Creator connects to Anthropic’s Claude through Amazon Bedrock’s cross-region inference profile. Amazon Bedrock serves two distinct roles in the system.

Role 1: CLI Requirements Analysis with Structured Output

When you run cli-creator create, the first step is analyzing the natural language description and extracting a structured specification. Instead of parsing raw text from the model, we use the Strands Agents SDK’s structured output feature with Pydantic models to guarantee the response conforms to our schema:

from pydantic import BaseModel, Field
from strands import Agent
from strands.models import BedrockModel

class CommandSpec(BaseModel):
    name: str = Field(description="Command name in kebab-case")
    description: str = Field(description="What this command does")
    arguments: Optional[List[str]] = Field(default_factory=list)
    options: Optional[List[CommandOption]] = Field(default_factory=list)

class CLIRequirements(BaseModel):
    cli_name: str = Field(description="CLI name in kebab-case")
    description: str = Field(description="One-line description")
    commands: List[CommandSpec] = Field(description="Commands to generate")
    dependencies: List[str] = Field(default_factory=list)

# Create agent and invoke with structured output
agent = Agent(
    model=BedrockModel(model_id="us.anthropic.claude-opus-4-6-v1"),
    system_prompt="You are an expert CLI designer..."
)

result = agent(
    f"Analyze this CLI description: {description}",
    structured_output_model=CLIRequirements
)

# Access the validated Pydantic model — no JSON parsing needed
requirements: CLIRequirements = result.structured_output

By passing the structured_output_model, the Strands Agent constrains the model’s response to match the Pydantic schema. The result is a validated Python object where if the model’s first attempt does not conform to the schema, Strands automatically sends the validation errors back to the model and retries, producing a correct response without manual intervention. This approach eliminates malformed JSON, missing fields, wrong types, and hallucinated structure.

Role 2: Complete Command Generation with AI Functions

The second Amazon Bedrock role is generating complete command implementations. Direct integration of AI agents in code generation workflows is often avoided because of the model’s non-deterministic nature. There is no guarantee that generated code will compile, follow the expected structure, or avoid common pitfalls like empty error handlers. Strands AI Functions addresses this through runtime post-condition checking. AI Functions is a Python library for building reliable AI-powered applications through a new abstraction of functions that behave like standard Python functions but are evaluated by reasoning AI Agents. You decorate a function with @ai_function, write its prompt as a docstring with curly-brace placeholders, and attach post-conditions that the output must satisfy. If any post-condition fails, AI Functions automatically initiates a self-correcting loop, sending the specific error back to the model and retrying until all conditions pass or the maximum attempts are reached.

We use AI Functions to build a self-correcting code generation pipeline. Each generated command must pass three post-conditions before it is accepted:

from ai_functions import ai_function, PostConditionResult

def check_syntax(response: str) -> PostConditionResult:
    try:
        compile(response, '<generated>', 'exec')
        return PostConditionResult(passed=True)
    except SyntaxError as e:
        return PostConditionResult(
            passed=False,
            message=f"Python syntax error on line {e.lineno}: {e.msg}. Fix: {e.text}"
        )

def check_has_decorator(response: str) -> PostConditionResult:
    if '@cli.command' in response:
        return PostConditionResult(passed=True)
    return PostConditionResult(
        passed=False,
        message="Missing @cli.command() decorator."
    )

@ai_function(
    post_conditions=[check_syntax, check_has_decorator, check_no_empty_try],
    max_attempts=3
)
def generate_click_command(command_name: str, description: str, ...) -> str:
    """
    Generate a complete Click CLI command function in Python.

    Use @cli.command() decorator. Include needed imports using 'from X import Y' style.
    Always use 'import click' and reference as click.echo(), click.style().

    Command: {command_name}
    Description: {description}
    """

The @ai_function decorator turns the function’s docstring into a prompt template. Curly-brace placeholders like {command_name} are filled from the function arguments at call time. Each post-condition receives the model’s response and returns a PostConditionResult. When a condition fails, AI Functions sends the error message back to the model and retries automatically, up to max_attempts. The model sees the specific failure (“syntax error on line 42”, “missing @cli.command decorator”, “empty try/except block detected”) and corrects it on the next attempt.

The prompt embedded in the docstring still enforces coding conventions (use import click rather than from click import, use from X import Y for all other imports) to prevent import conflicts. Post-conditions catch what the prompt misses, making the pipeline significantly more reliable than prompt engineering alone.

MCP Server Discovery and Integration

The Model Context Protocol adds automatic discovery of external API knowledge to the pattern. When your tool description mentions AWS services, the system searches for MCP servers that can provide domain-specific tooling. Generated tools can tap into live, structured API knowledge beyond what Amazon Bedrock knows at generation time.

How Discovery Works

The system uses Amazon Bedrock to extract API keywords dynamically. The api_keywords field is part of the same CLIRequirements Pydantic model used for structured output, so keyword detection happens in the same call that extracts commands and dependencies at zero additional cost:

class CLIRequirements(BaseModel):
    cli_name: str = ...
    commands: List[CommandSpec] = ...
    dependencies: List[str] = ...
    api_keywords: List[str] = Field(
        default_factory=list,
        description="API/service keywords to search for MCP servers"
    )

When the model returns keywords like ["dynamodb", "s3", "cloudtrail"], the system uses a Strands Agent with the http_request tool from Strands Agents Tools to search the MCP registry for each keyword. Results are merged and deduplicated.

Interactive MCP Selection

After discovering relevant MCPs, the system presents them to the user for selection:

Terminal screenshot showing an AI-powered MCP server discovery process. The tool analyzes requirements, detects API keywords (DynamoDB, S3, CloudTrail), searches for relevant servers, and displays 18 available MCP servers in a numbered list. The user selects servers 1, 9, and 18 at a prompt, confirming the inclusion of AWS DynamoDB, AWS S3, and AWS CloudTrail.

Selected MCPs are configured in the generated CLI’s .mcp.json file, and a bridge module is copied to the output project. This bridge connects to MCP servers at runtime, extracts their tool metadata, and converts them into Strands @tool functions that the Agent can invoke.

Terminal screenshot showing CLI code generation in progress. The user confirms generation with "y." Command 1/5 (dynamo-capacity) succeeds with 9,181 characters generated. Command 2/5 (unused-buckets) fails validation due to a missing @cli.command() decorator, retries (attempt 1/3), and ultimately succeeds with 8,834 characters. Command 3/5 (audit-changes) begins generating.

After MCP selection, CLI Creator generates each command sequentially using AI Functions. Here, the unused-buckets command initially fails the check_has_decorator post-condition for missing the @cli.command decorator, and AI Functions automatically retries generation with the error fed back to the model, producing valid code on the second attempt. All commands go through this process before having an installable CLI.

The Meta-Tooling Workflow: Create, Update, Revert

The most distinctive feature of the pattern is the iterative tool refinement workflow. This is where meta-tooling becomes practical, and it is the part most easily adapted to domains beyond CLI generation.

Step 1: Install and verify the generated CLI

Terminal screenshot showing successful CLI tool generation. A green checkmark with green text confirms "CLI tool generated successfully!" A yellow warning notes "External dependencies detected!" followed by installation instructions: navigate to the generated/aws-ops-audit directory, install dependencies with pip, and verify with the --help command.

After generation completes, install the CLI and verify it works:

Terminal screenshot showing the help output for the "aws-ops-audit" CLI tool, run inside a Python virtual environment (test-venv). The tool is described as "An AWS operations reporter that checks DynamoDB capacity, lists unused S3 buckets, and audits CloudTrail changes." Two options (--version, --help) and six commands (audit-trail, check-capacity, configure, full-report, tool, unused-buckets) are listed.

Each command is fully implemented. Here is unused-buckets pulling live S3 data:

Terminal screenshot showing the output of the "aws-ops-audit unused-buckets" command run in a Python virtual environment. An "Unused S3 Buckets Report" lists four S3 buckets with metadata including name, region, object count, size, last modified date, creation date, and reason for being flagged — either "no activity since [date]" or "empty bucket."

After installation, the CLI is ready to use. Each subcommand supports --help for detailed parameter information.

Step 2: Create a new reporting tool at runtime

Consider a scenario where leadership requests a new report that was not part of the original CLI, such as a summary of all Amazon S3 buckets with their sizes, sorted by cost impact. Instead of modifying source code, use the built-in tool create command:

Terminal screenshot showing the output of the "aws-ops-audit unused-buckets" command run in a Python virtual environment. An "Unused S3 Buckets Report" lists four S3 buckets with metadata including name, region, object count, size, last modified date, creation date, and reason for being flagged — either "no activity since [date]" or "empty bucket."

Amazon Bedrock generates a complete Strands tool, saves it to `tools/`, and commits it to git. The next CLI command automatically discovers and loads the new tool from disk, so you can execute it right away:

Terminal screenshot showing the output of "aws-ops-audit tool execute list_s3_buckets_by_cost_impact." A summary section displays aggregate statistics for 26 S3 buckets totaling 3.99 GB and $1.10/year in estimated costs. Below, a detailed data table ranks all 26 buckets by cost impact, showing columns for bucket name, cost rank, cost tier (MINIMAL or EMPTY), creation date, estimated monthly/annual costs, object count, region, and size metrics.

Output is automatically formatted based on data type, so lists of dictionaries render as tables, single dictionaries display as key-value pairs, and everything else falls back to JSON.

Step 3: Update and review changes

Suppose the initial output needs adjustment. Leadership wants the report to exclude buckets with an object count of zero. The user describes this change in natural language using the tool update command.

Terminal screenshot showing the update of the "list_s3_buckets_by_cost_impact" tool using the aws-ops-audit CLI with the instruction to "exclude buckets with zero objects." The tool is updated via Bedrock AI analysis, versioned with git, and a git diff displays changes to the function's docstring — adding exclusion behavior documentation and new return fields (total_buckets_scanned, excluded_empty_buckets).

CLI Creator commits the current version to git before overwriting, then generates a new version. The tool diff command shows exactly what changed. Now execute the updated tool to see the improvements:

Terminal screenshot showing the updated output of "aws-ops-audit tool execute list_s3_buckets_by_cost_impact" after the zero-object exclusion update. Summary statistics now show 22 active buckets out of 26 scanned, with 4 empty buckets excluded. The data table lists only 22 rows, all classified as MINIMAL — confirming the empty bucket exclusion is working correctly.

The same update workflow applies regardless of what the tool does, whether it is an Amazon S3 cost report, an Amazon DynamoDB capacity analyzer, or a Salesforce data exporter.

Step 4: Revert if needed

If the update didn’t work as expected, tool revert restores the previous version from git:

Terminal screenshot showing the revert of the "list_s3_buckets_by_cost_impact" tool using the aws-ops-audit CLI. A confirmation prompt asks "Revert 'list_s3_buckets_by_cost_impact' to previous backup? [Y/n]:" and the user enters "y." A green checkmark confirms "Tool reverted!" followed by a note: "Restored from git history."

The git log shows the full history of create, update, and revert operations, all tracked automatically.

Terminal screenshot showing a two-line git log output. The most recent commit (ef4a65e, HEAD → main) reverts the list_s3_buckets_by_cost_impact tool. The previous commit (3640bde) updated the same tool to "exclude buckets with zero objects and add a total."

Under the hood, tool create, tool update, and tool revert are convenience wrappers around git. Each operation commits to the repository, so the version history is standard git and works with any existing workflow. The tool diff and tool revert commands exist so that someone iterating conversationally can see changes and undo them without switching context to git commands, but git log, git diff, and git revert work just as well. Git-based versioning and one-command reverts make it safe to experiment.

Step 5: Output formats

Reports often need to be consumed in different ways. The --format flag lets you control how output is rendered:

Terminal screenshot showing a two-line git log output. The most recent commit (ef4a65e, HEAD → main) reverts the list_s3_buckets_by_cost_impact tool. The previous commit (3640bde) updated the same tool to "exclude buckets with zero objects and add a total."

The formatter attempts to use the Rich library for colored tables when available and falls back to an ASCII table implementation when it is not installed. Here is a new AWS Lambda tool stored in `tools/`, rendering as a table by default:

Terminal screenshot showing the creation and immediate execution of a new "list_lambda_functions_by_code_size" tool using the aws-ops-audit CLI. The tool is created from a natural language description, then executed to produce a Lambda functions report for us-east-1 showing 11 functions with a total code size of 206.31 MB, displayed in a table sorted by code size descending.

IV. Conclusion

The meta-tooling pattern demonstrated here combines Amazon Bedrock for code generation, the Strands Agents SDK for runtime tool management, and Model Context Protocol for external API discovery into a system where CLIs extend themselves through natural language. The implementation has clear limitations today. Generated code still requires human review before production use; post-conditions catch structural errors but cannot verify business logic correctness, and the MCP ecosystem is young enough that server coverage is uneven across domains.

V. Next Steps

CLI tools are a natural starting point because they have a well-defined structure and fast feedback loops, but the same mechanism applies to any software that could benefit from generating and refining small, composable units of functionality at runtime. Infrastructure-as-code modules, data pipeline transformations, API integration adapters, and compliance policy checks are all domains where the creation pattern is repetitive, and the validation criteria are expressible as post-conditions. To explore the pattern:

– Start with Amazon Bedrock for foundation model access.

– Use the Strands Agents SDK for tool orchestration.

– Browse the MCP ecosystem at mcpservers.org.

– Fork the CLI Creator source code on GitHub.

About the authors

Ragib Ahsan

Ahsan is an AI Acceleration Architect at Amazon Web Services (AWS), where he helps organizations build and implement AI/ML solutions. Specializing in computer vision and industrial manufacturing, he works with AWS partners and engineering teams to create practical applications using cloud technologies.

Modernizing Excel VBA to Python at Scale with AWS Transform custom

Post Syndicated from Somnath Chatterjee original https://aws.amazon.com/blogs/devops/modernizing-excel-vba-to-python-at-scale-with-aws-transform-custom/

Learn how AWS Transform custom can help migrate Excel VBA applications to modern Python code while overcoming context window limitations, preserving functional equivalence, and enabling cloud-native deployment—turning weeks of manual rewriting into hours of AI-guided transformation.

Introduction

Many organizations maintain dozens of Excel VBA applications built over decades, containing business-critical logic trapped in workbooks—budget planning tools, demand planning, inventory management, financial modeling, and engineering calculations. Manual migration typically costs thousands of dollars per workbook and takes weeks, while traditional AI tools fail on large codebases that exceed context windows.

This post demonstrates how you can migrate VBA to Python using AWS Transform custom, addressing three key challenges: processing large codebases through intelligent chunking, converting legacy code to maintainable Python while preserving functionality, and validating equivalence through automated testing. You can reuse the transformation across similar projects or apply it to entire portfolios.

With AWS Transform custom, you can accelerate migration timelines, eliminate transcription errors, and scale from single applications to enterprise portfolios.

Solution Overview

The following diagram illustrates how AWS Transform custom migrates VBA source code to Python output through a four-step process powered by an AI agentic system.

Architecture diagram showing how AWS Transform custom migrates VBA source code to Python output through a four-step process powered by an AI agentic system

Figure1:AWS Transform custom VBA to Python migration architecture

AWS Transform custom provides an interactive workflow where you describe your migration requirements. The system interprets your intent and iteratively refines the transformation definition until it meets your specifications. As the system processes your code, it improves the quality of each subsequent run. Once you finalize a transformation, you can publish it to a registry so your team can reuse it across multiple projects without starting from scratch.

Migration Approach

The migration follows a three-phase process that takes you from defining your transformation through execution and validation and finally scaling across your portfolio.

In the first phase, transformation definition creation, you start an interactive session with AWS Transform custom to describe your migration requirements. You can reference your VBA code, API documentation, and target framework guides as context. The system uses these inputs to automatically generate transformation rules and patterns, which you can iteratively refine through build validation until the definition accurately captures your migration logic.

In the second phase, execution and validation, AWS Transform custom applies the transformation to your codebase. It intelligently chunks large codebases into logical modules and processes them in dependency-aware order, so cross-module references remain intact. Throughout execution, the system continuously validates builds and tests, automatically detecting and correcting errors as they arise.

In the third phase, scale and reuse, you publish your finalized transformation to the registry, making it available for your team to apply across similar projects. You can run campaign-based bulk executions across multiple repositories, extract knowledge items for continuous improvement, and integrate the transformation into your CI/CD pipelines.

Key capabilities

AWS Transform custom addresses the core challenges you face when migrating large VBA codebases to Python.

For context window management, the system automatically segments your codebase into logical modules, tracks cross-module dependencies, preserves interface contracts across chunks, and maintains state throughout the transformation. This means you can process codebases that far exceed standard AI context window limits without losing coherence between modules.

For intelligent code restructuring, AWS Transform custom recognizes VBA idioms and maps them to Python equivalents. It refactors procedural code into object-oriented designs, replaces Windows-specific APIs with cross-platform libraries, and applies proper encapsulation and separation of concerns. The result is a clean, maintainable Python that follows modern coding standards.

For functional equivalence preservation, the system generates automated tests based on the original VBA behavior and runs regression testing after each transformation step. It benchmarks performance metrics such as timing and resource usage and validates edge cases to confirm that your transformed Python code produces the same results as the original VBA application.

Prerequisites

Before you begin, you need an AWS account with the appropriate permissions to use AWS Transform custom. Start by configuring authentication and setting up the AWS CLI for your environment.

To create an IAM user for AWS Transform, follow the step-by-step instructions for creating an IAM user and managing IAM policies in the IAM User Guide.

Setup using AWS CLI:

# Create policy
cat > transform-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["transform-custom:*"],
      "Resource": "*"
    }
  ]
}
EOF

aws iam create-policy \
  --policy-name AWSTransformCustomPolicy \
  --policy-document file://transform-policy.json

# Attach to your IAM user (replace with your username and account ID)
aws iam attach-user-policy \
  --user-name YOUR_USERNAME \
  --policy-arn arn:aws:iam::YOUR_ACCOUNT_ID:policy/AWSTransformCustomPolicy

Local Environment Setup

  • Operating System: Linux, macOS, or WSL (Windows Subsystem for Linux)
  • Node.js: Version 20 or higher (required for AWS Transform CLI installation via npm)
  • Git: Required for all target repositories
  • Internet Access: Required for AWS Transform service communication

AWS Transform CLI Installation

Follow the official AWS Transform Custom Getting Started Guide for complete installation and setup instructions, including:

  • Platform requirements (Linux, macOS, or WSL)
  • Installation script usage
  • Authentication configuration
  • Network requirements and firewall rules

Source Code Repository

  • VBA application source code (Excel workbook with macros or exported .bas files)
  • Git repository initialized in source directory
  • Build/test commands defined (if applicable)

Target Environment Setup (for Python migration)

  • Python 3.8 or higher installed
  • Virtual environment tool (venv or virtualenv)
  • Target framework dependencies (e.g., pygame for game applications, tkinter/PyQt6 for GUI, pandas for data processing, openpyxl for spreadsheet operations)

Walkthrough: VBA to Python Migration

Animated demonstration of the VBA to Python migration workflow using AWS Transform custom

Figure2:AWS Transform custom CLI executing a VBA to Python transformation

Step 1: Prepare Your VBA Application

Initialize Git Repository

AWS Transform Custom requires your code to be in a Git repository. If your code isn’t already in Git:

cd your-vba-project
git init
git add .
git commit -m "Initial VBA code"

Step 2: Create Transformation Definition

Start AWS Transform Interactive Session

Provide Migration Context

Screenshot showing the AWS Transform custom interactive session interface for providing migration context

Figure3:AWS Transform custom interactive session with migration context

When you start the interactive session, provide a natural language description of your migration goal, such as “Migrate VBA application to Python.” Specify the target framework you want to use, for example, pygame for GUI applications or pandas for data processing. You should also include document references to give the system the context it needs — this includes your VBA code files, Python framework documentation, and any API migration guides relevant to your project.

Example conversation:

User: I want to migrate a VBA application to Python.
AWS Transform: I’ll help you create a transformation definition. Let me analyze the VBA patterns and generate Python equivalents…

Define Scope and Entry Criteria

Before creating your transformation, review your existing codebase to understand its structure, patterns, and dependencies. Identify the code type and technology stack you are working with, such as an Excel VBA game that uses Windows API calls. Check the transformation registry for existing similar transformations to avoid duplicating work. From there, define a clear objective and scope for your transformation, and document the entry criteria that determine what code qualifies. Outline detailed implementation steps with specific technical mappings, establish validation and exit criteria for measuring success, and name your transformation appropriately so your team can discover and reuse it.

Iterative Refinement

After you provide your migration context, AWS Transform custom generates an initial transformation definition.You can review this definition at: ~/.aws/atx/custom/<session-id>/artifacts/tp-staging/transformation_definition.md.

Examine the generated rules and provide feedback on patterns, edge cases, or missing scenarios. AWS Transform custom incorporates your feedback and regenerates the definition, allowing you to iterate until the transformation accurately captures your migration logic.

Key Patterns to Address

During the transformation definition process, you should account for the common patterns that differ between VBA and Python. AWS Transform custom maps VBA ColorIndex values to standard RGB tuples and converts Range objects to Python data structures such as lists and arrays. Do While loops in VBA translate to event-driven loops in Python. The transformation replaces Windows API calls with cross-platform libraries that provide portable alternatives. VBA UserForms map to Python GUI frameworks like tkinter or PyQt6, where form controls become Python widgets. VBA Property Get/Let constructs convert to Python @property and @setter decorators. Finally, VBA’s 1-based array indexing must be adjusted to Python’s 0-based indexing throughout your codebase.

Step 3: Execute Transformation

Apply Transformation to Codebase

# Execute transformation interactively
atx custom def exec \
  --transformation-name "VBA-to-Python-Migration" \
  --code-repository-path "./your-vba-project" \
  --build-command "python3 -m py_compile **/*.py"

# Execute transformation non-interactively
atx custom def exec \
  --transformation-name "VBA-to-Python-Migration" \
  --code-repository-path "./your-vba-project" \
  --build-command "python3 -m py_compile **/*.py" \
  --trust-all-tools \
  --non-interactive

What Happens During Execution

When you run the transformation, AWS Transform custom begins with agent planning. It analyzes your codebase structure, identifies all VBA components and their dependencies, and generates a step-by-step transformation plan with logical ordering — for example, data models first, then business logic, then UI. Each step in the plan has a clear scope and validation criteria.

Next, the system performs automatic code chunking. It analyzes your codebase size and complexity, then segments the code into logical modules such as constants, game logic, rendering, and controller. These chunks are processed in dependency order based on the plan, and the system manages the context window by focusing on one module at a time.

Throughout execution, AWS Transform custom tracks dependencies by mapping relationships between VBA subroutines, verifying that dependent code references remain valid, and maintaining interface contracts across modules.

The transformation proceeds incrementally, converting one module at a time. After each module is transformed, the system validates the build. If a failure occurs, it automatically rolls back and retries the transformation for that module before moving on.

Sample Output:

Analyzing codebase structure...
Identified 4 logical modules: constants, core_logic, rendering, main_controller
Transforming module 1/4: constants
 - Converting VBA Enums to Python IntEnum
 - Mapping ColorIndex values to RGB tuples
 - Generating color constants
 ✓ Build validation passed
Transforming module 2/4: core_logic
 - Converting collision detection subroutines
 - Refactoring global variables to class attributes
 - Translating VBA arrays to Python lists
 ✓ Build validation passed
Transforming module 3/4: rendering
 - Replacing Range object manipulation with pygame rendering
 - Converting cell-based drawing to pixel-based graphics
 - Implementing screen update logic
 ✓ Build validation passed
Transforming module 4/4: main_controller
 - Converting Do While loop to pygame event loop
 - Replacing GetAsyncKeyState with pygame.event.get()
 - Implementing game state management
 ✓ Build validation passed
Transformation complete! Generated 5 Python modules.

Review Transformed Code

After the transformation completes, check the generated Python files to verify the output. You can review the transformation logs at ~/.aws/atx/custom/<conversation-id>/logs/ to understand the decisions the system made during each step. Examine the build validation results to confirm that the transformed code compiles and passes all checks.

Step 4: Validate Functional Equivalence

Automated Test Generation

AWS Transform generates validation tests based on original VBA behavior:

test_validation.py – Verifies core logic:

# Example generated test (conceptual)
def test_collision_detection():
 """Verify collision logic matches VBA behavior"""
 # Test cases extracted from VBA code analysis

def test_scoring_calculation():
 """Ensure scoring algorithm is preserved"""

def test_state_transitions():
 """Validate game state changes"""

test_performance.py – Benchmarks non-functional requirements:

def test_frame_rate():
 """Verify rendering meets 60 FPS target"""

def test_response_time():
 """Ensure input handling latency matches VBA"""

Run Validation Suite

# Execute all tests
python3 -m pytest tests/

# Run with coverage
python3 -m pytest --cov=. tests/

Manual Validation Checklist

  • Application launches without errors
  • UI renders correctly (layout, colors, sizing)
  • User interactions work as expected (keyboard, mouse)
  • Core functionality produces correct results
  • Performance meets requirements (no lag, smooth rendering)
  • Edge cases handled properly (boundary conditions, invalid inputs)

Step 5: Refine and Iterate

As you refine your transformation, you may encounter a few common issues. Here is how you can address them and provide feedback to AWS Transform custom.

Timing and performance differences can occur when the Python application runs faster or slower than the VBA original. To fix this, adjust timing constants and frame rate limiters. For example, if you notice the game runs too fast because VBA used Sleep(500) for piece drops, you can provide that feedback. AWS Transform custom corrects this by replacing time.sleep() with pygame.time.Clock.tick(60).

Color rendering mismatches happen when colors in the Python output don’t match the VBA version. This is typically caused by incorrect RGB mappings for VBA ColorIndex values. If you notice that ColorIndex 3 should be pure red (255,0,0) but is showing as dark red, provide that feedback. AWS Transform custom updates the COLOR_MAP dictionary with accurate RGB values.

Collision detection bugs may appear when pieces move through walls or other pieces. This requires refining boundary checking and collision logic. For instance, if rotation near walls allows pieces to go out of bounds, you can report this issue. AWS Transform custom adds boundary validation before rotation commits to resolve it.

Continual learning in action

Each time you provide a correction, AWS Transform custom captures it as a knowledge item. Future transformations automatically incorporate these fixes, so the same issues don’t recur. With each execution, the system improves quality for similar migrations across your portfolio.

Step 6: Publish and Scale

Save Transformation as Draft (for testing)

atx custom def save-draft \
  --name "VBA-to-Python-Migration" \
  --source-directory "~/.aws/atx/custom/<session-id>/artifacts/tp-staging"

Publish to Registry (for team-wide use)

atx custom def publish \
  --name "VBA-to-Python-Migration" \
  --description "Migrate Excel VBA applications to Python with pygame rendering" \
  --source-directory "~/.aws/atx/custom/<session-id>/artifacts/tp-staging"

Apply to Multiple Projects

# Apply to each project individually
atx custom def exec -n "VBA-to-Python-Migration" -p ./project1 -c "python3 -m py_compile **/*.py" --trust-all-tools --non-interactive
atx custom def exec -n "VBA-to-Python-Migration" -p ./project2 -c "python3 -m py_compile **/*.py" --trust-all-tools --non-interactive

CI/CD Integration

# Example GitHub Actions workflow
name: VBA Modernization
on:
  push:
    branches: [main]
jobs:
  transform:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Run VBA to Python transformation
        run: |
          atx custom def exec \
            --transformation-name "VBA-to-Python-Migration" \
            --code-repository-path "." \
            --build-command "python3 -m py_compile **/*.py" \
            --non-interactive \
            --trust-all-tools

Benefits

By using AWS Transform custom, you can reduce tech debt and accelerate enterprise modernization at scale. The service supports diverse transformation use cases beyond VBA to Python, adapting to the specific needs of your codebase and target platform. With continual learning and improvement, each transformation builds on the corrections and knowledge items from previous runs, delivering higher quality results over time. Once you define a transformation, you can apply it everywhere — across similar projects, teams, and entire application portfolios — without recreating the migration logic from scratch.

Architecture Evolution

Before migration, your legacy VBA application is constrained to a Windows-only execution environment, tightly coupled to the Excel runtime. The code follows a procedural style with a global state, and testing and deployment are manual processes.

After migration with AWS Transform custom, your application follows a modular architecture with clean separation of concerns across constants, logic, rendering, and controller layers. The code uses object-oriented design with proper encapsulation and runs cross-platform on Windows, macOS, and Linux. The transformed application is cloud-ready, with a structure that supports containerization through Dockerfile generation, serverless adaptation with decoupled logic and rendering, and standard Python packaging through requirements.txt and setup.py. It also integrates with modern DevOps workflows, including automated testing with pytest, CI/CD pipeline compatibility, and version control through Git.

You can deploy the transformed application in several ways: containerized with Docker and Kubernetes for scalable web applications, serverless with AWS Lambda for event-driven processing, as a standalone desktop executable with PyInstaller, or as a web application with FastAPI or Flask wrappers for browser access.

The following screenshot shows the transformed Tetris application running as a Python pygame application.

Screenshot of the transformed Tetris application running as a Python pygame application

Figure4:Transformed Tetris game running as a Python pygame application

Transformation definition creation

When creating your transformation definition, provide comprehensive context by including your VBA code, target framework documentation, and migration guides. Focus on documenting application behavior by describing what the application does functionally, not just its code structure. Identify platform-specific dependencies such as Windows APIs, Excel features, and ActiveX controls that require special handling during migration. Define clear success criteria including performance targets, functional requirements, and test coverage expectations. Be explicit about construct mappings in your transformation definition — the more specific you are, the more consistent the output.

Execution strategy

Start with a pilot project by testing your transformation on a smaller, representative application before applying it broadly. Validate incrementally by reviewing each module transformation before proceeding to the next. Capture corrections as feedback by documenting issues and fixes so the system can incorporate them through continual learning. Iterate on edge cases to refine the transformation for corner cases and error conditions that may not surface during initial runs.

Quality assurance

Generate comprehensive automated test suites from the original VBA behavior to verify functional equivalence. Run performance benchmarking to confirm that non-functional requirements match the original application. Conduct manual validation through user acceptance testing to verify UI and UX consistency. After any refinements, run the full test suite for regression prevention to ensure that fixes in one area haven’t introduced issues elsewhere.

Scaling across your portfolio

When you are ready to scale, categorize your applications by similarity — for example, data processing, UI-heavy, or calculation-focused workbooks. Create transformation variants with customized definitions for each category. Use campaigns for bulk execution to process multiple applications in parallel. Monitor and aggregate results across your portfolio to track success rates, identify common issues, and measure time savings.

Cleanup

Archive Conversation Logs

# Conversation logs are in ~/.aws/atx/custom/<conversation-id>/
# Archive for future reference before cleanup
tar -czf vba-migration-logs.tar.gz ~/.aws/atx/custom/<conversation-id>/

Remove Temporary Files

# Remove conversation data and transformation session artifacts (kept for 30 days automatically)
rm -rf ~/.aws/atx/custom/<conversation-id>

# Remove draft transformations (if not needed)
atx custom def delete --name "VBA-to-Python-Migration-Draft"

Manage Published Transformations

# List your transformations
atx custom def list

# Delete transformations no longer needed
atx custom def delete --name "VBA-to-Python-Migration-Old-Version"

# Update transformation tags for organization
atx custom def tag --transformation-name "VBA-to-Python-Migration" \
  --tags "team:platform-engineering,language:python,status:production"

Knowledge Items Management

# List knowledge items for transformation
atx custom def list-ki --transformation-name "VBA-to-Python-Migration"

# Review and approve valuable knowledge items
atx custom def get-ki --transformation-name "VBA-to-Python-Migration" --id <ki-id>

# Enable approved knowledge items
atx custom def update-ki-status \
  --transformation-name "VBA-to-Python-Migration" \
  --id <ki-id> \
  --status ENABLED

# Delete low-quality knowledge items
atx custom def delete-ki --transformation-name "VBA-to-Python-Migration" --id <ki-id>

Conclusion

In this post, you learned how AWS Transform custom can help you migrate Excel VBA applications to Python at scale. With AWS Transform custom, you can reduce weeks of manual migration work to hours. Intelligent chunking handles large codebases that exceed standard AI context window limits, while continuous validation and automated testing preserve functional equivalence throughout the process. Because transformations are reusable, and the system learns from each execution, you gain compound efficiency with every subsequent migration. The transformed applications are cloud-ready, supporting containerization, serverless deployment, and modern DevOps workflows.

AWS Transform custom is a strong fit when you are modernizing a portfolio of dozens or hundreds of similar legacy applications, when functional equivalence must be guaranteed for quality-critical migrations, or when you have repeatable migration patterns across multiple projects.

Next Steps

The transformation demonstrated in this post — VBA to Python migration with context window management, automated restructuring, and functional validation — is a template you can apply to other modernization challenges, such as mainframe COBOL to Java, Progress ABL to Spring Boot, or .NET Framework to .NET Core. The principles remain consistent: intelligent code analysis, dependency-aware processing, continuous validation, and transformation reuse.

Call to Action

Get Started with AWS Transform Custom

Documentation and Resources:

Explore AWS-Managed Transformations: Access pre-built transformations for common migration patterns:

  • AWS SDK Java v1 to v2: AWS/java-aws-sdk-v1-to-v2
  • Python Version Upgrades: AWS/python-version-upgrade
  • Node.js Version Upgrades: AWS/nodejs-version-upgrade
  • Java Version Upgrades: AWS/java-version-upgrade

Estimate Your Migration: Use the AWS Pricing Calculator to estimate costs for your specific modernization project.

Ankit Srivastava

is a Strategic Technical Account Manager at Amazon Web Services (AWS), where he serves as a trusted advisor to global enterprise customers. With over 15 years of experience in cloud architecture, DevOps and distributed systems, Ankit helps organizations navigate cloud transformation, architecture modernization, and harness the power of Generative AI on AWS.

Somnath Chatterjee

is an accomplished Senior Technical Account Manager at Amazon Web Services (AWS), Somnath is dedicated to guiding customers in crafting and implementing their cloud solutions on AWS. He collaborates strategically with customers to help them run cost-optimized and resilient workloads in the cloud. Beyond his primary role, Somnath holds specialization in the compute, SAP and Developer Experience technical field community. With over 14 years of experience in the information technology industry, he excels in cloud architecture and helps customers achieve their desired outcomes on AWS.